Two conformance gaps: \0 followed by a digit is accepted, and underscores are silently dropped from numbers

[mrvonkalus]

Summary

pyjson5 2.0.1 (current release) accepts two classes of input that the JSON5 spec forbids and that both reference implementations — the JS json5 package (2.2.3) and the pure-Python json5 package (dpranke, 0.15.0) — reject.

Found by differential fuzzing: the same 3,000 seeded inputs to pyjson5 and the JS reference, comparing accept/reject and parsed values. Both classes reproduce deterministically.

Class A: \0 followed by a digit

The spec (following ECMAScript 5.1) allows \0 only when not followed by a decimal digit — the rule exists to keep legacy octal escapes out. pyjson5's _get_escape_sequence handles \\1–\\9 correctly (they raise), but the b'0' branch returns NUL unconditionally, with no lookahead:

elif c0 == b'0':
    return 0x0000          # src/_decoder.pyx — missing: lookahead ∉ DecimalDigit

import pyjson5
pyjson5.loads('"\\01"')    # '\x001'  — should raise (JS json5 and dpranke both reject)
pyjson5.loads('"\\012"')   # '\x0012' — should raise
pyjson5.loads('"\\00"')    # '\x000'  — should raise
pyjson5.loads('"\\0"')     # '\x00'   — correct, legal

Class B: underscores silently dropped from numbers

JSON5 has no numeric separators (that's ES2021, not ES5.1). pyjson5's number scanners consume and discard _ in three places (integer/hex, fraction, and exponent loops in src/_decoder.pyx, each with an elif c0 == b'_' skip), so:

pyjson5.loads('1_0')       # 10        — should raise
pyjson5.loads('1__0')      # 10        — should raise
pyjson5.loads('1_')        # 1         — should raise
pyjson5.loads('0x_1')      # 1         — should raise
pyjson5.loads('.5_0')      # 0.5       — should raise
pyjson5.loads('1e1_0')     # 10000000000.0 — should raise

I checked the README, CHANGELOG, and issue tracker — neither behavior is documented or reported, and no test asserts the underscore behavior. If numeric separators are an intentional extension, a doc line would close this; as-is it looks accidental, especially 1_ → 1 and .5_0 → 0.5.

Why these survived

The official json5-tests suite (which this repo runs via third-party/json5-tests) has no test for \0-followed-by-digit and none for underscore separators — the strings/ directory has five escape tests, none covering the \0 lookahead. Both gaps live exactly where the conformance suite is blind. Adding cases there would protect every implementation that runs the suite, not just this one.

Impact

Modest but real: cross-parser config confusion. A polyglot stack that validates JSON5 config with a strict parser and consumes it with pyjson5 (or the reverse) reads different documents from the same bytes.

Disclosure

Found by AI agents (differential fuzzing harness) working under my direction; minimization, root-causing, and this report were reviewed and run by me. Happy to share the harness or turn either class into a PR with tests if you'd like.