Engineering

The newline rules that break JavaScript obfuscators

In almost all of JavaScript, a line break is whitespace. In five specific places it is not: it is a statement terminator, and it changes what the code means. The specification calls these restricted productions. They are the single most reliable way we have found to break a tool that rewrites JavaScript — including, on three separate occasions, our own.

The complete list

There are exactly five. A line terminator immediately after any of these keywords or before any of these operators ends the statement:

return    // `return \n x`   is  `return;` then `x;`   -> returns undefined
throw     // `throw  \n x`   is a SyntaxError (throw needs an operand)
yield     // `yield  \n x`   is  `yield;` then `x;`    -> yields undefined
++ --     // `x \n ++y`      is  `x;` then `++y;`      -> postfix never applies
break     // `break  \n lbl` is a plain break, the label is a separate statement
continue  // same as break

The return one is famous, because it bites humans too — it is why the “opening brace on the same line” convention exists. The other four are not famous, and that is precisely why tools get them wrong.

Why this is hard for a rewriter specifically

A tool that transforms JavaScript has to answer “where does this expression end?” thousands of times. In nearly every case the answer ignores whitespace, which makes ignoring whitespace the natural implementation — and it is right everywhere except these five places.

Worse, the mistake is invisible in the output. Consider:

function* g() {
  var x = yield
  1;
}

The correct reading is var x = yield; followed by an expression statement 1;. The generator yields undefined. A tool that treats the newline as whitespace reads yield 1 and produces a generator that yields 1. Both versions parse. Both run. One returns the wrong value, forever, silently. No test that checks “does the build succeed” will ever see it.

The ++ case is the same shape and more common in real code:

let a = 1, b = 1;
let r = a
++b
// r is 1, b is 2.  NOT `a++` followed by `b`.

This appears in real minified and generated code more often than you would guess, because the semicolon-free style makes it easy to write by accident, and the code still works — until something rewrites it.

The second-order trap: your tool has to read its own output

This one cost us a full round of debugging and is worth sharing, because it is not obvious until it happens.

Getting the parse right is only half the job. Once a parser correctly understands yield\n1 as a bare yield, the writer has to emit a bare yield — and a writer that emits it on one line produces yield ;. If the parser was only ever taught “a newline ends a yield” and not “a semicolon, brace, bracket, comma or end-of-file also ends a yield”, then the tool cannot re-read its own output. It fails with an unexpected-token error the moment anything re-processes the file.

The general rule that falls out of this: the terminator set has to be the spec's, not the one case you found. Otherwise you have moved the bug rather than fixed it. A round-trip check — parse the output with your own parser — catches this class immediately, and it caught ours.

Test your own toolchain in ten minutes

You do not need to trust anybody's claims here. Put this in a file, run it before and after your minifier, bundler, transpiler or obfuscator, and compare the two outputs:

function r1() {
  return
  42;
}

function r2() {
  let a = 1, b = 1;
  let x = a
  ++b
  return [x, a, b];
}

function* g1() { var x = yield
  1; return x; }

function lbl() {
  let hits = 0;
  outer: for (const i of [0, 1]) {
    for (const j of [0, 1]) { if (j) continue outer; hits++; }
  }
  return hits;
}

console.log(JSON.stringify([r1(), r2(), [...g1()], lbl()]));

// Correct output, verified against Node:
// [null,[1,1,2],[null],2]
// (the nulls are undefined -- that is how JSON.stringify renders it in an array)

Run it with Node directly, then run the processed version, then diff the two lines of output. If they differ, your pipeline has a restricted-production bug, and you now have a minimal reproduction to send the vendor.

One more, which fails in the opposite direction

Add this separately, because it is a SyntaxError rather than a wrong value:

function* g2() { yield
  * [7, 8]; }

The grammar is yield [no LineTerminator here] * AssignmentExpression, so the line break makes this a bare yield followed by a statement starting with * — which is not valid JavaScript. Node rejects it, and so should your tool.

The failure mode to watch for is a tool that accepts it, reads it as a delegation, and emits working code. That is not leniency being helpful; it means the tool checks for the * before it checks for the line break, and the same ordering bug will silently mis-read the valid cases above. The check has to come after.

Why we are writing this down

We found three restricted-production defects in our own engine in a single week — ++/-- across a line break, yield across a line break, and the re-parse problem above. All three were pre-existing. All three produced wrong values rather than errors. We found them by enumerating the family rather than waiting for a report, which is the only method we know that works on this class.

The reason to publish it is that the same family will exist in other tools, and the test above is vendor-neutral. If you run it against something we make and it fails, we want the report. If you run it against something else and it fails, you have saved yourself a very confusing afternoon.

Related reading