Repository navigation
Propagate items/kvitems builder errors in yajl2_c parse_basecoro - #199
Conversation
Reviewer's GuideImplementationPropagate failures from the items and kvitems builders, and release the event value before returning on the error path.
DocumentationDocument the propagated builder errors and the event-value leak fix in the changelog.
TestsAdd a map type that rejects duplicate keys and verify its errors propagate from items and kvitems.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 3 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="src/ijson/backends/ext/_yajl2/parse_basecoro.c" line_range="104" />
<code_context>
if (KVItemsBasecoro_Check(gen->target_send)) {
- kvitems_basecoro_send_impl(gen->target_send, prefix, event, value);
+ N_N(kvitems_basecoro_send_impl(gen->target_send, prefix, event, value));
}
else if (ItemsBasecoro_Check(gen->target_send)) {
</code_context>
<issue_to_address>
**Resumed iteration uses failed builder**
When a caller catches a `map_type` exception and calls `next()` again on the same file-backed items() or kvitems() iterator, `reading_generator_next()` re-raises the saved exception without closing the iterator, so a later `next()` feeds more input to the same parser and builder. The builder remains active after `builder_event()` fails; a resumed `end_map` can therefore emit a partial object or dereference a NULL value, leaving callers with corrupted output or a crash.
Make a builder/parser error permanently terminate the reader, or reset all affected builder and parser state before allowing iteration to resume.
Also at `src/ijson/backends/ext/_yajl2/parse_basecoro.c:107`.
</issue_to_address>
### Comment 2
<location path="src/ijson/backends/ext/_yajl2/parse_basecoro.c" line_range="104" />
<code_context>
if (KVItemsBasecoro_Check(gen->target_send)) {
- kvitems_basecoro_send_impl(gen->target_send, prefix, event, value);
+ N_N(kvitems_basecoro_send_impl(gen->target_send, prefix, event, value));
}
else if (ItemsBasecoro_Check(gen->target_send)) {
</code_context>
<issue_to_address>
**Async parse errors retain input buffers**
When an async `items()` or `kvitems()` builder raises, the propagated error makes `async_reading_generator_next()` return before releasing its `Py_buffer` and decrefing the input buffer, so each failed parse retains the buffer and its export.
Release the `Py_buffer` and decref the input buffer on the error path before returning.
Also at `src/ijson/backends/ext/_yajl2/parse_basecoro.c:107`, `src/ijson/backends/ext/_yajl2/basic_parse_basecoro.c:31`.
</issue_to_address>
### Comment 3
<location path="src/ijson/backends/ext/_yajl2/parse_basecoro.c" line_range="104" />
<code_context>
if (KVItemsBasecoro_Check(gen->target_send)) {
- kvitems_basecoro_send_impl(gen->target_send, prefix, event, value);
+ N_N(kvitems_basecoro_send_impl(gen->target_send, prefix, event, value));
}
else if (ItemsBasecoro_Check(gen->target_send)) {
</code_context>
<issue_to_address>
**Failed emissions retain built results**
When a downstream callback raises while an `items()` or `kvitems()` result is being delivered, the `N_N` checks propagate the exception, but `items_basecoro_send_impl()` and `kvitems_basecoro_send_impl()` return before decrefing their local built-object or tuple references, retaining each failed emission’s result.
Decref the built object or tuple before returning from the send implementations when `CORO_SEND` fails.
Also at `src/ijson/backends/ext/_yajl2/parse_basecoro.c:107`.
</issue_to_address>Sourcery assessment
Approval pending. 3 findings to address first.
Blocking findings: src/ijson/backends/ext/_yajl2/parse_basecoro.c:104, src/ijson/backends/ext/_yajl2/parse_basecoro.c:104, src/ijson/backends/ext/_yajl2/parse_basecoro.c:104
rtobar
left a comment
There was a problem hiding this comment.
Thanks again @nabeel-dev21 for this, it looks like an easy fix 👍. I've left a couple of comments, after which I'm happy to merge.
9003227 to
d4f0136
Compare
|
As a coincidence, the |
When items()/kvitems() are chained in C, parse_basecoro_send_impl() calls items_basecoro_send_impl()/kvitems_basecoro_send_impl() directly and ignored their return value. An error while building an object, such as a map_type whose __setitem__ rejects a duplicate key, was dropped: the yajl callback reported success and parsing went on with the exception still set. The next Python call (e.g. the next map_type()) then failed with SystemError, which left the builder active with a NULL value, and the following end_map crashed in builder_value(). Return NULL when either call fails so the callback cancels the parse and the original exception propagates. add_event_and_value() now also releases the event value on that error path, which it used to leak.
d4f0136 to
ddc73f7
Compare
|
@nabeel-dev21 I took the liberty of rebasing your changes on top of the latest |
|
Thanks for sorting out the rebase and merging this. |
yajl2_c drops errors raised while items()/kvitems() build an object
When
items()/kvitems()are chained in C,parse_basecoro_send_impl()callsitems_basecoro_send_impl()/kvitems_basecoro_send_impl()directly and ignores their return value. Amap_typethat raises (here one rejecting duplicate keys) therefore leaves its exception set while yajl keeps parsing. The nextmap_type()call fails withSystemError, the builder stays active with a NULL value, and the followingend_mapsegfaults inbuilder_value():kvitems()and both async variants crash the same way. The other backends raise theValueError, and so does yajl2_c's*_coropath because it goes through the genericCORO_SEND; the #188 thread assumed yajl2_c propagated downstream errors everywhere, but that only holds for that generic path. Once the error propagates,add_event_and_value()returns before releasing the event value, leaking about 100 KB per failed parse with a 100 KB string, so it now releases the value first. The new tests segfault on master and pass with the fix on all four backends; the full suite passes on CPython 3.13 and 3.15, along with doctests and--memleaks-onlyon 3.13.Summary by Sourcery
Fix yajl2_c error handling for items() and kvitems() so downstream exceptions propagate safely without crashes or leaked event values.
Bug Fixes:
Tests: