Skip to content

Propagate items/kvitems builder errors in yajl2_c parse_basecoro - #199

Merged
rtobar merged 1 commit into
ICRAR:masterfrom
nabeel-dev21:parse-basecoro-propagate-errors
Nov 13, 2026
Merged

rtobar merged 1 commit into
ICRAR:masterfrom
nabeel-dev21:parse-basecoro-propagate-errors

Conversation

@nabeel-dev21

@nabeel-dev21 nabeel-dev21 commented Nov 3, 2026 •

Copy link
Copy Markdown

yajl2_c drops errors raised while items()/kvitems() build an object

When items()/kvitems() are chained in C, parse_basecoro_send_impl() calls items_basecoro_send_impl()/kvitems_basecoro_send_impl() directly and ignores their return value. A map_type that raises (here one rejecting duplicate keys) therefore leaves its exception set while yajl keeps parsing. The next map_type() call fails with SystemError, the builder stays active with a NULL value, and the following end_map segfaults in builder_value():

class NoDuplicateKeysDict(dict):
    def __setitem__(self, key, value):
        if key in self:
            raise ValueError(f"duplicate key {key!r}")
        super().__setitem__(key, value)

doc = io.BytesIO(b'[{"a": 1, "a": 2}, {}]')
list(ijson.items(doc, 'item', map_type=NoDuplicateKeysDict))  # SIGSEGV

kvitems() and both async variants crash the same way. The other backends raise the ValueError, and so does yajl2_c's *_coro path because it goes through the generic CORO_SEND; the #188 thread assumed yajl2_c propagated downstream errors everywhere, but that only holds for that generic path. Once the error propagates, add_event_and_value() returns before releasing the event value, leaking about 100 KB per failed parse with a 100 KB string, so it now releases the value first. The new tests segfault on master and pass with the fix on all four backends; the full suite passes on CPython 3.13 and 3.15, along with doctests and --memleaks-only on 3.13.

Summary by Sourcery

Fix yajl2_c error handling for items() and kvitems() so downstream exceptions propagate safely without crashes or leaked event values.

Bug Fixes:

  • Propagate exceptions raised by custom map types through the yajl2_c items() and kvitems() parsing paths instead of continuing with a pending error that could cause SystemError or interpreter crashes.
  • Release parsed event values when downstream processing fails to prevent memory leaks.

Tests:

  • Add coverage for map-type errors in items() and kvitems() across supported backends.

@sourcery-ai

sourcery-ai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Implementation

Propagate failures from the items and kvitems builders, and release the event value before returning on the error path.

  • src/ijson/backends/ext/_yajl2/basic_parse_basecoro.c L29-31
  • src/ijson/backends/ext/_yajl2/parse_basecoro.c L104-107

Documentation

Document the propagated builder errors and the event-value leak fix in the changelog.

  • CHANGELOG.md L27-31

Tests

Add a map type that rejects duplicate keys and verify its errors propagate from items and kvitems.

  • tests/test_base.py L282-288
  • tests/test_items.py L8, L49-54
  • tests/test_kvitems.py L3, L50-55

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="src/ijson/backends/ext/_yajl2/parse_basecoro.c" line_range="104" />
<code_context>

 	if (KVItemsBasecoro_Check(gen->target_send)) {
-		kvitems_basecoro_send_impl(gen->target_send, prefix, event, value);
+		N_N(kvitems_basecoro_send_impl(gen->target_send, prefix, event, value));
 	}
 	else if (ItemsBasecoro_Check(gen->target_send)) {
</code_context>
<issue_to_address>
**Resumed iteration uses failed builder**

When a caller catches a `map_type` exception and calls `next()` again on the same file-backed items() or kvitems() iterator, `reading_generator_next()` re-raises the saved exception without closing the iterator, so a later `next()` feeds more input to the same parser and builder. The builder remains active after `builder_event()` fails; a resumed `end_map` can therefore emit a partial object or dereference a NULL value, leaving callers with corrupted output or a crash.

Make a builder/parser error permanently terminate the reader, or reset all affected builder and parser state before allowing iteration to resume.

Also at `src/ijson/backends/ext/_yajl2/parse_basecoro.c:107`.
</issue_to_address>

### Comment 2
<location path="src/ijson/backends/ext/_yajl2/parse_basecoro.c" line_range="104" />
<code_context>

 	if (KVItemsBasecoro_Check(gen->target_send)) {
-		kvitems_basecoro_send_impl(gen->target_send, prefix, event, value);
+		N_N(kvitems_basecoro_send_impl(gen->target_send, prefix, event, value));
 	}
 	else if (ItemsBasecoro_Check(gen->target_send)) {
</code_context>
<issue_to_address>
**Async parse errors retain input buffers**

When an async `items()` or `kvitems()` builder raises, the propagated error makes `async_reading_generator_next()` return before releasing its `Py_buffer` and decrefing the input buffer, so each failed parse retains the buffer and its export.

Release the `Py_buffer` and decref the input buffer on the error path before returning.

Also at `src/ijson/backends/ext/_yajl2/parse_basecoro.c:107`, `src/ijson/backends/ext/_yajl2/basic_parse_basecoro.c:31`.
</issue_to_address>

### Comment 3
<location path="src/ijson/backends/ext/_yajl2/parse_basecoro.c" line_range="104" />
<code_context>

 	if (KVItemsBasecoro_Check(gen->target_send)) {
-		kvitems_basecoro_send_impl(gen->target_send, prefix, event, value);
+		N_N(kvitems_basecoro_send_impl(gen->target_send, prefix, event, value));
 	}
 	else if (ItemsBasecoro_Check(gen->target_send)) {
</code_context>
<issue_to_address>
**Failed emissions retain built results**

When a downstream callback raises while an `items()` or `kvitems()` result is being delivered, the `N_N` checks propagate the exception, but `items_basecoro_send_impl()` and `kvitems_basecoro_send_impl()` return before decrefing their local built-object or tuple references, retaining each failed emission’s result.

Decref the built object or tuple before returning from the send implementations when `CORO_SEND` fails.

Also at `src/ijson/backends/ext/_yajl2/parse_basecoro.c:107`.
</issue_to_address>

Sourcery assessment

Approval pending. 3 findings to address first.

Blocking findings: src/ijson/backends/ext/_yajl2/parse_basecoro.c:104, src/ijson/backends/ext/_yajl2/parse_basecoro.c:104, src/ijson/backends/ext/_yajl2/parse_basecoro.c:104


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

@rtobar rtobar left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again @nabeel-dev21 for this, it looks like an easy fix 👍. I've left a couple of comments, after which I'm happy to merge.

Comment thread CHANGELOG.md Resolved
@nabeel-dev21
nabeel-dev21 force-pushed the parse-basecoro-propagate-errors branch from 9003227 to d4f0136 Compare November 6, 2026 13:59
@rtobar

rtobar commented Nov 10, 2026

Copy link
Copy Markdown

As a coincidence, the macos-14 runners have been phased out from GitHub, so when I tried to run them in this PR the macOS jobs didn't kick in. I bumped the runners on master. @nabeel-dev21 could you rebase once again on top of the latest master? I'll then run the workflows and merge 👍

When items()/kvitems() are chained in C, parse_basecoro_send_impl()
calls items_basecoro_send_impl()/kvitems_basecoro_send_impl() directly
and ignored their return value. An error while building an object, such
as a map_type whose __setitem__ rejects a duplicate key, was dropped:
the yajl callback reported success and parsing went on with the
exception still set. The next Python call (e.g. the next map_type())
then failed with SystemError, which left the builder active with a NULL
value, and the following end_map crashed in builder_value().

Return NULL when either call fails so the callback cancels the parse
and the original exception propagates. add_event_and_value() now also
releases the event value on that error path, which it used to leak.
@rtobar
rtobar force-pushed the parse-basecoro-propagate-errors branch from d4f0136 to ddc73f7 Compare November 12, 2026 15:16
@rtobar

rtobar commented Nov 12, 2026

Copy link
Copy Markdown

@nabeel-dev21 I took the liberty of rebasing your changes on top of the latest master to get the new macOS runners. We also just got RISCV workers to produce riscv64 binary wheels, so those will also kick off eventually. Once CI is green I'll merge 👍

@rtobar
rtobar merged commit b5dfaa0 into ICRAR:master Nov 13, 2026
55 checks passed
@nabeel-dev21

Copy link
Copy Markdown
Author

Thanks for sorting out the rebase and merging this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants