Repository navigation
fix(parse): a NUL byte inside a file is a character, not its end - #32
Merged
Merged
Conversation
Four cases: a NUL in a comment is ignored and the file goes on, one in unquoted and in triple-quoted text is kept, one in a quoted string is an error, as any control character. typesafe/config does all four. The oracle adapter holds a case in a shell variable, which drops a NUL, so typesafe/config and pyhocon never saw one: the protocol now carries it as \0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD
The tokenizer took every 0 byte for the sentinel past the input. In a
comment that ended the file without an error: `a = 1 # note<NUL>` dropped
every line below, and a UTF-16 file without a byte order mark read as {}.
Only the NUL at input.len ends the input now; one before it is ignored in a
comment, kept in unquoted and triple-quoted text, and rejected in a quoted
string by the parser, as any control character (comments.1,
unquoted-strings.14-15, multi-line-strings.1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A NUL byte inside a file ended it. In a comment that happened without an error:
read as
{"a":1}, every line below the NUL gone. A file saved as UTF-16 without a byte order mark has a NUL after every ASCII character, so# settingsthenport = 8080read as{}. Found by fuzzing (F016); F011 and F013, a NUL in unquoted and triple-quoted text failing withUnexpectedToken, have the same cause.The spec
//or#and the next newline is considered a comment and ignored", a NUL included."""sequence are used unmodified".Changes
src/Tokenizer.zig: the tokenizer reads a[:0]const u8and took every 0 byte for the sentinel past the input. Now only the NUL atinput.lenends it, checked the way the.eofstate already did; one before it is a character in every state. A table test pins all five places it can stand.comments/015(ignored in a comment, the file goes on),unquoted-strings/031(kept),032(an error in a quoted string),multi-line-strings/010(kept). typesafe/config does all four as the spec says.suite.encodenow sends it as\0andOracle.javadecodes it; pyhocon'sunicode_escapealready does.A file whose last byte is a stray NUL is now an error (typesafe/config: "Key '' may not be followed by token: end of file") where it read as if the NUL were not there.
Scores
zig build testandzig build test-python(100 % coverage) are green.