Repository navigation
Reject Unicode whitespace between tokens in the python backend - #202
nabeel-dev21 wants to merge 2 commits into
Conversation
The python backend's Lexer skips everything between LEXEME_RE matches, and the regex ended in \S, so any character Python's re module treats as whitespace separated tokens. That includes U+001C-U+001F, U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F and U+3000, none of which are JSON whitespace, so a document like [1,<U+00A0>2] parsed with the python backend while the yajl backends and the json module reject it. Only skip the characters yajl's lexer skips (space, \t, \n, \v, \f and \r). Anything else is now a lexeme of its own, which parse_value rejects.
Reviewer's GuideImplementationRestrict the Python lexer to skipping the whitespace characters the yajl backends skip, while documenting why vertical tab and form feed remain accepted.
DocumentationDocument the Python backend’s rejection of Unicode whitespace and its continued acceptance of vertical tab and form feed.
TestsAdd cross-backend coverage verifying non-JSON whitespace cannot separate tokens.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="CHANGELOG.md" line_range="32-34" />
<code_context>
later surfaced as an unrelated `SystemError` or crashed the interpreter.
* The `yajl2_c` backend no longer leaks the event value when an error is raised
while `parse()`, `items()` or `kvitems()` process it.
+* The `python` backend now rejects Unicode whitespace such as `U+00A0`
+ or `U+2028` between tokens, as the `yajl` backends already do.
+ It used to skip anything Python's `re` module considers whitespace.
## [3.6.0]
</code_context>
<issue_to_address>
**Whitespace rejection overstated**
When a caller relies on the changelog's general statement for input containing vertical tab or form feed, the backend still skips U+000B and U+000C because `LEXEME_RE` excludes `\v` and `\f`, so readers can understand the changelog to promise rejection of whitespace that the backend accepts.
Qualify the changelog entry to specify that it rejects Unicode whitespace beyond the whitespace the yajl backends accept.
</issue_to_address>Sourcery assessment
Approved.
|
Thanks @nabeel-dev21 for another nice contribution. The memleak tests are tripping for some reason, but they have been consistently working fine for years now, I'd need to investigate what's going on, but at least I cannot reproduce it locally with the exact same version of Python :(. The last scheduled run on
That's unfortunate, because they indeed are not considered whitespace by the RFC. And the built-in |
Let me take that back -- I managed to reproduce it, hopefully I'll be able to understand what's going on now. |
|
Added a comment above LEXEME_RE saying \v and \f aren't JSON whitespace but stay because yajl skips them and system or dynamically loaded yajl builds can't be fixed from ijson. The CHANGELOG entry now says the same, so it no longer reads as if every non-JSON whitespace character is rejected. On the memleak failures, this change only touches the python backend's regex, so I'd be surprised if it's the cause, but I'm happy to rebase once you've tracked it down. |
python backend accepts Unicode whitespace between tokens
Lexerskips everything betweenLEXEME_REmatches and the regex ended in\S, so any character Python'sretreats as whitespace worked as a token separator. That covers 23 BMP code points that aren't JSON whitespace (U+001C-U+001F, U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000), and documents using them parse with the python backend while the yajl backends andjsonreject them:The lexer now skips only what yajl's lexer skips.
\vand\faren't JSON whitespace either, but every backend accepts them, so they stay. Feeding each BMP code point in after[1,, after{"a":and before], the python and yajl2_c backends disagreed on 69 documents before this change and on none after. This is separate from #193, which was about number lexemes inparse_value; this one is the lexer's separator set.The new
test_non_json_whitespacefails for the python backend on master and passes with the fix. The full suite and doctests pass on CPython 3.11 and 3.13,--memleaks-onlyon 3.13, and the python-backend tests on PyPy 3.11.Summary by Sourcery
Align Python backend token separation with YAJL by rejecting Unicode whitespace that JSON does not permit.
Bug Fixes:
Tests:
Chores: