Skip to content

Reject Unicode whitespace between tokens in the python backend - #202

Open
nabeel-dev21 wants to merge 2 commits into
ICRAR:masterfrom
nabeel-dev21:python-lexer-whitespace
Open

nabeel-dev21 wants to merge 2 commits into
ICRAR:masterfrom
nabeel-dev21:python-lexer-whitespace

Conversation

@nabeel-dev21

@nabeel-dev21 nabeel-dev21 commented Nov 14, 2026 •

Copy link
Copy Markdown

python backend accepts Unicode whitespace between tokens

Lexer skips everything between LEXEME_RE matches and the regex ended in \S, so any character Python's re treats as whitespace worked as a token separator. That covers 23 BMP code points that aren't JSON whitespace (U+001C-U+001F, U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000), and documents using them parse with the python backend while the yajl backends and json reject them:

doc = '{"a":\N{NO-BREAK SPACE}[1,\N{LINE SEPARATOR}2]}'.encode()
list(ijson.get_backend('python').items(io.BytesIO(doc), ''))   # [{'a': [1, 2]}]
list(ijson.get_backend('yajl2_c').items(io.BytesIO(doc), ''))  # IncompleteJSONError
json.loads(doc)                                                # JSONDecodeError

The lexer now skips only what yajl's lexer skips. \v and \f aren't JSON whitespace either, but every backend accepts them, so they stay. Feeding each BMP code point in after [1,, after {"a": and before ], the python and yajl2_c backends disagreed on 69 documents before this change and on none after. This is separate from #193, which was about number lexemes in parse_value; this one is the lexer's separator set.

The new test_non_json_whitespace fails for the python backend on master and passes with the fix. The full suite and doctests pass on CPython 3.11 and 3.13, --memleaks-only on 3.13, and the python-backend tests on PyPy 3.11.

Summary by Sourcery

Align Python backend token separation with YAJL by rejecting Unicode whitespace that JSON does not permit.

Bug Fixes:

  • Make the Python backend reject non-JSON Unicode whitespace used between tokens, matching the behavior of the YAJL backends.

Tests:

  • Add coverage verifying that representative non-JSON whitespace characters cannot separate JSON tokens across backends.

Chores:

  • Document the Python backend's whitespace compatibility behavior in the changelog.

The python backend's Lexer skips everything between LEXEME_RE matches,
and the regex ended in \S, so any character Python's re module treats
as whitespace separated tokens. That includes U+001C-U+001F, U+0085,
U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F and
U+3000, none of which are JSON whitespace, so a document like
[1,<U+00A0>2] parsed with the python backend while the yajl backends
and the json module reject it.

Only skip the characters yajl's lexer skips (space, \t, \n, \v, \f and
\r). Anything else is now a lexeme of its own, which parse_value
rejects.
@sourcery-ai

sourcery-ai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Implementation

Restrict the Python lexer to skipping the whitespace characters the yajl backends skip, while documenting why vertical tab and form feed remain accepted.

  • src/ijson/backends/python.py L11-15

Documentation

Document the Python backend’s rejection of Unicode whitespace and its continued acceptance of vertical tab and form feed.

  • CHANGELOG.md L32-36

Tests

Add cross-backend coverage verifying non-JSON whitespace cannot separate tokens.

  • tests/test_basic_parse.py L153-159

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="CHANGELOG.md" line_range="32-34" />
<code_context>
   later surfaced as an unrelated `SystemError` or crashed the interpreter.
 * The `yajl2_c` backend no longer leaks the event value when an error is raised
   while `parse()`, `items()` or `kvitems()` process it.
+* The `python` backend now rejects Unicode whitespace such as `U+00A0`
+  or `U+2028` between tokens, as the `yajl` backends already do.
+  It used to skip anything Python's `re` module considers whitespace.

 ## [3.6.0]
</code_context>
<issue_to_address>
**Whitespace rejection overstated**

When a caller relies on the changelog's general statement for input containing vertical tab or form feed, the backend still skips U+000B and U+000C because `LEXEME_RE` excludes `\v` and `\f`, so readers can understand the changelog to promise rejection of whitespace that the backend accepts.

Qualify the changelog entry to specify that it rejects Unicode whitespace beyond the whitespace the yajl backends accept.
</issue_to_address>

Sourcery assessment

Approved.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread CHANGELOG.md Resolved
@rtobar

rtobar commented Nov 15, 2026

Copy link
Copy Markdown

Thanks @nabeel-dev21 for another nice contribution.

The memleak tests are tripping for some reason, but they have been consistently working fine for years now, I'd need to investigate what's going on, but at least I cannot reproduce it locally with the exact same version of Python :(. The last scheduled run on master saw no issues either, adding to the confusion.

The lexer now skips only what yajl's lexer skips. \v and \f aren't JSON whitespace either, but every backend accepts them, so they stay

That's unfortunate, because they indeed are not considered whitespace by the RFC. And the built-in json module also rejects them. This is a bug in yajl, and while I could fix it in my fork (which is what I embed when building the binary wheels for all platforms), this won't solve the problems where users link against their own yajl installation (like in the fast-build-and-test.yaml action) or when using the backends that dynamically load the yajl shared library. So yes, I'd agree it's best to leave these in. Could you add a small comment to this effect just above the new regular expression, so readers know why these two characters were left in? The entry in the CHANGELOG could also have that clarification added. Thanks!

@rtobar

rtobar commented Nov 15, 2026

Copy link
Copy Markdown

The memleak tests are tripping for some reason, but they have been consistently working fine for years now, I'd need to investigate what's going on, but at least I cannot reproduce it locally with the exact same version of Python :(. The last scheduled run on master saw no issues either, adding to the confusion.

Let me take that back -- I managed to reproduce it, hopefully I'll be able to understand what's going on now.

@nabeel-dev21

Copy link
Copy Markdown
Author

Added a comment above LEXEME_RE saying \v and \f aren't JSON whitespace but stay because yajl skips them and system or dynamically loaded yajl builds can't be fixed from ijson. The CHANGELOG entry now says the same, so it no longer reads as if every non-JSON whitespace character is rejected. On the memleak failures, this change only touches the python backend's regex, so I'd be surprised if it's the cause, but I'm happy to rebase once you've tracked it down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants