Skip to content

fix(parse): a NUL byte inside a file is a character, not its end - #32

Merged
lejmr merged 2 commits into
mainfrom
fix/nul-byte
Oct 25, 2026
Merged

lejmr merged 2 commits into
mainfrom
fix/nul-byte

Conversation

@lejmr

@lejmr lejmr commented Oct 25, 2026

Copy link
Copy Markdown
Owner

A NUL byte inside a file ended it. In a comment that happened without an error:

a = 1 # note<NUL>
b = 2

read as {"a":1}, every line below the NUL gone. A file saved as UTF-16 without a byte order mark has a NUL after every ASCII character, so # settings then port = 8080 read as {}. Found by fuzzing (F016); F011 and F013, a NUL in unquoted and triple-quoted text failing with UnexpectedToken, have the same cause.

The spec

  • comments.1 (L125): "Anything between // or # and the next newline is considered a comment and ignored", a NUL included.
  • unquoted-strings.15 (L280): "unquoted strings have no restriction on control characters, other than the ones listed as forbidden characters", and NUL is not one of them.
  • multi-line-strings.1 (L291): "all Unicode characters until a closing """ sequence are used unmodified".
  • unquoted-strings.14 (L278): quoted JSON strings may not contain control characters, so a NUL there stays an error.

Changes

  • src/Tokenizer.zig: the tokenizer reads a [:0]const u8 and took every 0 byte for the sentinel past the input. Now only the NUL at input.len ends it, checked the way the .eof state already did; one before it is a character in every state. A table test pins all five places it can stand.
  • Suite, 482 cases: comments/015 (ignored in a comment, the file goes on), unquoted-strings/031 (kept), 032 (an error in a quoted string), multi-line-strings/010 (kept). typesafe/config does all four as the spec says.
  • Oracle protocol: the adapter holds a case in a shell variable, which drops a NUL, so typesafe/config and pyhocon never saw one. suite.encode now sends it as \0 and Oracle.java decodes it; pyhocon's unicode_escape already does.
  • CHANGELOG has a Fixed line. It also drops a sentence fix(resolve): a self-reference with nothing below reads the environment #31 left in twice.

A file whose last byte is a stray NUL is now an error (typesafe/config: "Key '' may not be followed by token: end of file") where it read as if the NUL were not there.

Scores

spec Java mode
zig-hocon 482/482 423/482
typesafe/config 1.4.9 423/482 482/482
pyhocon 0.3.63 340/482 311/482

zig build test and zig build test-python (100 % coverage) are green.

lejmr and others added 2 commits October 25, 2026 19:36
Four cases: a NUL in a comment is ignored and the file goes on, one in
unquoted and in triple-quoted text is kept, one in a quoted string is an
error, as any control character. typesafe/config does all four.

The oracle adapter holds a case in a shell variable, which drops a NUL, so
typesafe/config and pyhocon never saw one: the protocol now carries it as \0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD
The tokenizer took every 0 byte for the sentinel past the input. In a
comment that ended the file without an error: `a = 1 # note<NUL>` dropped
every line below, and a UTF-16 file without a byte order mark read as {}.
Only the NUL at input.len ends the input now; one before it is ignored in a
comment, kept in unquoted and triple-quoted text, and rejected in a quoted
string by the parser, as any control character (comments.1,
unquoted-strings.14-15, multi-line-strings.1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD
@lejmr
lejmr merged commit c07b466 into main Oct 25, 2026
42 checks passed
@lejmr
lejmr deleted the fix/nul-byte branch October 26, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant