Repository navigation
feat: end a stream's data where the file does, keep none of it, and leave a guard's cut to the guard - #297
Merged
Conversation
A stream read from a file kept a copy of its raw data for as long as the stream was held, by the object cache or by the caller. Streams whose data share a stretch of the file each kept their own: two hundred of them held 200 MB from a file of 1 MB, and validating three hundred nested object streams held 300 MB, whatever the object stream budget let go. The data is read each time it is asked for now, and kept by nothing: a source in memory lends it where it lies, and any other reads it into an array of its length, handed back whole on a short read rather than copied again. Refs #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
A stream's encoded data was bounded by the file alone. Decoding handed back data no filter decodes - none named, an image filter, one the library does not know, Flate over data that is none - whole, past MaxDecodedStreamLength and unreported, and a /Length between what an array holds and what an int does threw an OutOfMemoryException each time the data was read. PdfReaderLimits.MaxStreamLength bounds what is read of one stream at once, 256 MB by default, under the code limit.stream-length: past it the data is cut, marked cut by a guard, so that what the filters lose past the cut is no damage, and the guard is reported once for each stream, or thrown under ThrowOnLimit. Data decoding hands back as it is falls under it, and MaxDecodedStreamLength keeps bounding what filters produce. A length past what an array holds ends at the guard, whose ceiling is the array's. The stream's length stays the one the file holds. The manifest can raise the new limit. Refs #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…herever its window ends A stream's length was misjudged three ways. More than four bytes of white space before endstream, or a comment, made a right /Length a wrong one. A /Length the file cannot hold grew the window to MaxObjectLength even when an endstream was found: it read 21.5 MB for five bytes, reported limit.object and marked the object cut. And the search in the window took the first endstream it held, past the object's endobj and the next header: 49 bytes of object 6 for a stream of five. A declared length is confirmed by endstream after any white space and comments, the file read on as far as they run. Every other length - missing, unusable, past the file, or one endstream does not follow - goes to one search: the bytes the parser holds, then the file, up to the nearer of the first object header and the next object the index places, once for each stream. A stream's data never counts toward MaxObjectLength, and no window grows to find its end. With no endstream before that stop, the object's endobj plays its part: a declared length endobj follows is kept, and otherwise the endobj that ends the object before the stop ends the data, its end-of-line left out. Without one, a declared length that ends before the stop is kept, and the data otherwise runs to the stop, a declared length past it cut there. The parser alone applies the same rules to its buffer. A cross-reference stream whose /Length the chain could not read is searched as the window's search did, a carriage return kept and no header stopping it, now past the window rather than through one MaxTrailerLength bounds. Tests that declared a length past the next object, or cut stream data with MaxObjectLength, expect what the decisions give: the data ends at the object's endobj or at the next object, and stream data is cut by MaxStreamLength. PDFBox's pdfbox3947 object stream 417 keeps its declared 1,302 bytes, inside which its Flate data ends. Refs #174, #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
An object that lost both its closing delimiter and its endobj had its array or dictionary take in the next object's header and value, up to that object's endobj, and read as ending with one: the missing endobj went unreported, and a trailer that lost its >> took the next update's entries. Inside an array or a dictionary, where an element, a key or a value is expected, the next object's header now ends every container left open. A header is the one the search for a stream's endstream stops at: N G obj, digits alone, N from 1 to the largest object number and G within the generation range, white space alone between them, and obj a whole keyword a window's edge does not reach. The cut is reported once, at the innermost container, as "An object header ended the object", and the object lacks its endobj. Containers skipped past the depth the parser follows end there too. An object stream's member is never followed by a header, and a value alone is no container: neither applies the rule. Its cost: a closed container holding text that reads as a header ends there, and its endobj is judged missing. None of the 344,361 objects of the corpus holds one. Refs #230 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…s are read A cross-reference stream whose /Length the chain cannot read takes its data up to the first endstream. Rows that spell one cut the data short: the section was read in part and called malformed, and an update's objects past the spelled rows read as the revision it supersedes. Data with no endstream at all ran to the end of the file and was read as rows. Where the section's own row for the /Length lies among the rows the data holds, the object it places is read there, nothing kept or reported, and the data is taken again to the length it gives when an endstream confirms it, then decoded once more. A row that does not stand leaves the data as the chain took it, and the length is checked once the chain is read, as before. Data with no endstream before the next object the sections read so far place, or the end of the file, makes its section malformed: the file's fault, and no guard's. The search past the window this relies on came with the single search for a stream's end: a stream of 70,000 bytes of rows reads as its direct twin, with no limit.trailer, and MaxTrailerLength bounds the dictionary alone. Tests hold that too. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…e file's A /Length naming an object a guard cut was judged by what the cut left of it: null, reported as a /Length that holds no length, or an integer the cut parted, 1 of 15, reported as a declared length the data contradicts. Neither is a value the file gave. The answer is keyed on the object the reference leads to, through a chain of references too, and never on the value read: one a guard cut takes a form of its own, and one a guard left out of the index the existing one. An integer read before the cut is only a place to look for endstream at, never kept as a length; the data is otherwise found by its endstream, and nothing is said of the length. A missing endstream is still the file's, and reported. A cross-reference stream's /Length read once the chain is read is left alike. No message is made for a length not judged. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… index A /Filter, a /DecodeParms, or a value of a step's parameters that led to an object a guard cut, or left out of the index, read as no filter, or as "/Flate" of "/FlateDecode" parted: an object stream decoded so read its header as stray bytes, the catalog it held was looked for elsewhere, and a sound file under a lowered limit was judged broken. The chain of filters now ends at such a value, as at the element past MaxFilterChainLength: the steps before it run, the data is left encoded, and an object stream so left is hidden and its members taken as the guard's. A guard that loading a filter's value reached was taken for a cut of the stream's own data, which hid the stream's own faults, and made the reader and the validator disagree as read order went. Decoding now says whether the stream's own data is cut, which is what the readers of rows and members ask. While the chain is read, a cross-reference stream's /W, /Index, /Size, /Filter or /DecodeParms naming an object MaxObjectLength cuts made the section malformed, and the index was rebuilt. The section is now the guard's: the chain ends there, nothing is rebuilt, and the guard is reached once the chain is read, thrown at opening when the caller asks. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…eader's limits ADR 34 gains the amendment of 2026-10-23: MaxStreamLength is a new guard; EndStreamSearchPasses stays an internal constant, its reason rewritten; the bound on the gap before endstream is gone; EndObjLookahead is a first read's size; MaxTrailerLength bounds a cross-reference stream's dictionary alone; a value read through a reference to an object a guard cut or left out of the index is the guard's. The architecture, the threat model and the reader-limits pages say the same. Refs #170, #174, #220, #224, #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… otherwise The referee held that the reader and qpdf both took object 417's data of PDFBox's pdfbox3947 up to object 441's endstream, past a 32 KB block of zeros that erased its own: inside its window the reader did not bound its search. Its single search for a stream's end now stops at the next object the index places, in the bytes the reader holds as past them: the reader stops inside the block and keeps the declared 1,302 bytes, where qpdf, whose search no object bounds, reads 34,152. The expectation is corrected, not weakened: the stream joins the known disagreements, for qpdf's reason, as PDFium's urban planning report's object 695 did, and the remarks no longer say the reader's search is unbounded inside its window. The manifest's expectations of the document hold. Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… each time The reader-limits concepts, the guides to the reader's limits and its memory, and the diagnostics reference say what MaxStreamLength bounds, what limit.stream-length reports, and that a stream's raw data is read from the file each time it is asked for, kept by nothing. Refs #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…he references The diagnostics, object-model and validation-rules references, the concepts of diagnostics, lazy reading and validation, the threat model and M02 describe the stream-end rules, the values a guard cut or left out of the index, and the header rule. The XML documentation of the codes, the pipeline, the reader, PdfStream, Decode and the endobj rule says the same. Refs #174, #224, #230, #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…er its window ends The review of the batch found four places where a stream's end depended on how it was read rather than on its bytes. - The search for a stream's endstream is kept once per stream, but whether the object's endobj follows the declared length was kept with it. A rebuild that replaced the /Length then read the stream with the old answer. It is now read again whenever the declared end changes, and only then, so a second reading of an unchanged stream reads nothing more. - An endstream the window held, with the window ending before what follows it, left the object's endobj unseen. Past the window, what follows is read on in the file. It is now read on in both cases, so where the window ends decides nothing. - A file that ends on the stream keyword, or on its carriage return, read the stream as no data and said nothing: the parser took the end of the bytes for a window too short to say where the data starts. The end of the file says it: the data starts there and holds nothing, and the missing endstream is reported, as after a line feed. - Closing the reader released the searches but not the checks made past white space after a length. The states a missing endstream ends in, and the next object a fault names, are documented as the search now decides them. Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… keep no length from it A /Length that leads through a reference to an object a guard cut is read as the guard's: what the cut left is only a place to look for endstream at. The review of the batch found two places where the length's fault still said otherwise. - The fault named the object the /Length refers to, not the one the guard cut at the end of the chain: "names object 20 0, which one of the reader's limits cut" of an object read whole. - The integer the cut left, a parted 1 of 15, was kept as the fault's declared length. It is now only where the parser looks for endstream first, and the fault declares no length. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… an object it cut A cross-reference stream's /W, /Index, /Size, /Filter or /DecodeParms that leads to an object MaxObjectLength cuts became the guard's in this batch. A /Prev or /XRefStm written as a reference to such an object was left out. The section it names was reported missing as the file's fault, the index was rebuilt, and the validator said no section placed the object, though the newer section did. Now the chain ends there for the guard, as for the other values. Nothing is rebuilt, the older sections and their trailers are the guard's to hide, and the guard is reached once the chain is read: reported, or thrown at opening under ThrowOnLimit. A hybrid file's stream so hidden ends the chain with it, since the older sections' rows would stand for its own. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…no object it ends in a trailer
Two faults of the header rule, which ends an object's open containers at the next object's header.
- Inside a dictionary or an array, a header the window's edge cut short ("6 0", "6 0 ob") was
read for what it seemed: its number and generation as keys that are not names, or the number
as the value of the key before it. A larger window drops these reports, but where none follows,
at MaxObjectLength or in the rebuild's trailer scan, they were kept as the file's faults. The
parse now counts as having met the edge there, as it does for a reference the edge cut.
- The report said the header ended "the object" in a trailer, or a value read alone, where no
object is open. It now says the header came inside the construct left open.
A test case now holds the dictionary's value path, where a header stands for a key's value.
Refs #230
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…eferences and its guards The review of the batch found four ways a /Length the chain could not read was settled wrongly. - Only the first link of a chain of references was read while the chain was. A /Length object holding "6 0 R", where object 6 waited for the chain or was cut by MaxObjectLength, was judged to hold null, and the guard was never reached. The whole chain is followed now, and the length waits for the chain wherever a link cannot be read. - Taking the data again to the section's own row for the /Length used the first row numbering it, a row of a reserved type included, which reading the rows ignores. It now uses the row the reading keeps. - When MaxObjectLength cut that /Length object, the data was not taken again, and the rows past the endstream they spell were charged to the file: the section was said to lack them, and an older revision's rows stood for them. The integer the cut left is now where an endstream is looked for. When none confirms it, those rows are the guard's, as rows past a decoding guard are, and the chain ends there. - Data MaxStreamLength cut was still checked against the length once the chain was read, the guard's cut reported as the file's. Whether a guard cut it is known only once decoding read it, and it is checked then. Refs #220, #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…axStreamLength The helper that opens a corpus document as the acceptance tests open it mapped every reader limit a manifest entry can raise but the new MaxStreamLength. A document whose entry raises it would have been read under the default by the stream length tests, and under its own bound by the others. Refs #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…it can be The review of the batch found two places where a comment, which ISO 32000-1 (7.2.3) counts as white space outside a stream's data, was read wrongly at the end of one. Both rules were settled with the maintainer. - A /Length too short that ends on a "%" of the data: the rest of the line read as a comment before the endstream, which confirmed the wrong length, and the rest of the data was lost with no report. The same comment let an endobj after it keep the short length when the stream has no endstream. A comment now counts there only once white space comes before it: a "%" right where the length ends is the data's, the length goes unconfirmed, and the search finds the real end. - A stream with no endstream ends at its object's own endobj when only white space follows it up to the next object. A comment there kept the endobj from ending the data: the short length was kept, or the data took in the endobj. Comments after the endobj are white space now. The endobj is followed forward through the reads of the search for endstream, which reads every byte up to where it stops already, rather than read back from there: a comment can only be told from the start of its line. The bytes before the stop are no longer read a second time. Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…lter past every guard HasImageFilter says whether Decode hands back an encoded image, and read none of the filters' parameters to tell it. Since a value of a step's parameters that leads to an object a guard cut ends the chain where it is read, a Flate step before a DCT one whose /DecodeParms a guard cut left its data Flate-encoded: HasImageFilter still said it was the image, and an extractor that trusts it would have written Flate data as a JPEG. It now reads what decoding reads before the image filter, as the maintainer settled: the parameters of a Flate or LZW step that comes before it, and nothing for a chain that ends sooner. A parameter held in an object stream decodes that stream first, as decoding the stream would. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…the index A /Length that leads to an object a guard left out of the index is unknown, and not judged. With it, the endstream the stream lacked went unreported too: the data was taken where its object's endobj ends it, and neither the diagnostics nor the validator saw a fault. A /Length a guard cut already reported it as the file's fault. As the maintainer settled, both are reported alike now: "The stream's /Length names object 21 0, which one of the reader's limits left out of the index, and no endstream follows its data ...". The object named is the one the chain of references leads to, as for a cut one. A message about a stream's length is now made only when it is reported: a stream parsed again, or one whose length is read once the chain is, made each message and dropped it. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…nnot end its data A cross-reference stream whose /Length the chain cannot read takes its data up to the first endstream; with none before the next object or the end of the file, its section is malformed. A /Length that the newer section places, but that MaxObjectLength cuts, waits for the chain the same way, and was judged the same: the fault said no section placed the object, the index was rebuilt, and the validator reported a malformed section. Without the guard the length is read, and the section with it. As the maintainer settled, such a section is the guard's, as one whose /W, /Index or /Filter a guard cut is: it counts as cut, the chain ends there, nothing is rebuilt, and the guard is reached once the chain is read, thrown at opening under ThrowOnLimit. The fault names the reference the guard cut, through a chain of references too. Refs #220, #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… passes to Decode Decode promises the diagnostics a caller passes of their own what decoding meets, each time the stream is decoded, and limit.decoded-stream reaches them so. The cut MaxStreamLength makes went to the document's diagnostics alone, once, as the reader read the data: a caller who passed their own got data cut short with no report, and could not tell, the cut being marked only internally. As the maintainer settled, those diagnostics now receive limit.stream-length with each decoding that meets the cut, and the document's still hold it once. A test of ASCII85 data the guard cut added the two lists together and expected the guard once in all: it now finds it once in each, the expectation following the settled rule. The message is made once, as the reader reports it once, rather than each time the data is read past the bound. The XML documentation of Decode and of PdfStreamData.GetBytes says what the guard does to the data and where it is reported. Refs #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
A stream's raw bytes taken from a document opened over memory are a view of that memory, not a copy: they keep it alive after Dispose, for as long as they are held, and change as it does. Refs #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…s decoding does HasImageFilter now reads the parameters decoding reads before the image filter, and loading one past MaxObjectLength reaches the guard: under ThrowOnLimit, the caller is told so, as Decode would tell them. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… and HasImageFilter Decoding loads the values its filters read, and so can reach a guard under ThrowOnLimit, or meet a disposed document: the exceptions of Decode and HasImageFilter now say so. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
The batch's patch left lines of where a stream's data ends that no test reached. Each now has a file that reaches it: - an endobj that ends a longer token, read in one read or parted between two, or right where the data starts; - a stream parsed alone, in a window that is not the end of the data, with no endstream in it; - a container skipped past the depth the parser follows, which the next object's header ends; - a comment in the gap after a length that runs past the window and the first read after it, and the check the reader keeps for the stream's next reading; - "endstreamx" read on past the window, and white space read on to the end of the file; - a stream left unsearched whose declared length runs past the end of the file. The construction of streams nested in one another's dictionaries is shared by two tests now. Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… endstream is found The place a deferred cross-reference stream's endstream is looked for after is where its data starts plus the bytes the chain took, never before the start of the file. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…as they may Once the document's searches for endstream have read as much of the file as they may, white space after a declared length that runs past the window is not read on. A length the first read past the window cannot confirm is kept without a search, and one whose endstream that read ends after leaves what follows the keyword unseen. Neither path had a test. Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
A cross-reference stream whose /W gives its type field no width makes every row of type 1. The section's own row for its /Length, before rows that spell an endstream, then places it, and the data is taken again to the length it gives. No test had such a section. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…ta again A newer table that refuses its row for the object a cross-reference stream's /Length names leaves no row of that stream standing for it: the data stays as the chain took it, up to the endstream its rows spell, and the section is said to lack the rows past it. No test had such a chain. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…letes Data a proven endstream ends past white space under a hidden /Length takes in white space the length would leave out: a Flate stream short of its checksum's last byte read a space or a line feed in its place, and reported a checksum that disagrees, a warning, where read whole it is only short. A checksum that ends in that white space says nothing of the file, as one it leaves short. The decoders' docs no longer say such data decodes as read whole everywhere: LZW or RunLength data that lacks its end marker takes the white space in. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…e precedes it A cross-reference stream is read past any white space its offset names, as its header is found by reading its object. A classic table was told by the first 32 bytes alone: named 29 or more bytes before its keyword, which they held only in part or not at all, it was not read, the index was rebuilt, and startxref was said to name neither the keyword nor an object. The probe now reads on past white space, comments and a token its edge may cut, as far as a trailer's window reaches. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…gain knowing its end hidden Data decoded again knowing its end hidden held no more rows than the decoding before it, which was meant for a decoding MaxDecodedStreamLength stopped. A decoding no guard stopped, its end taken for the file's, drops a predicted row the data parts, which the second decoding undoes as far as it goes: the rows it completes, before the endstream the rows spell, were left to the guard, and the page whose row it held read as null where the whole read and the direct twin read it. The cap now holds where a guard stopped the decoding before only. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…n that reached it The proof that an endstream ends a stream's own data takes the first significant byte past the keyword from the read-on that found it, which tells whether an endobj follows, only where one did. Past an endstream the data spells, no endobj after it, it read the white space again from 16 bytes, doubling: thirty streams under a cut /Length, each spelling one before 64 KB of white space, had the searches read close to four times the file, where three are said. They read it about twice. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… end of the file The probe that looks for a classic table's xref keyword reads on past white space as far as MaxTrailerLength reaches. No test named more white space than that, nor white space that runs to the end of the file: a bound reached and read again, a loop, would have passed the suite. The guard is now held reached, reported or thrown, and white space to the end names no table. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…ong whatever it holds The subsection one of whose elements a guard cut is the file's fault where the element read is wrong whatever the cut one holds. No test held the edges: a count of 0, valid, a first number of 2147483647, valid with a count of 0 or 1, a first number past it and a count past the object numbers, nor the messages such a fault is said with. Each is now held, and a mutant of each edge or message fails. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…tever a cut value holds A cross-reference stream a value's cut keeps from being read was the guard's unless its widths or its numbering, as far as read, were wrong. Data whose length the file gives, decoded whole, may hold fewer rows than the counts read declare however few bytes the widths read allow a row: three bytes under a /Size of 8. Read whole, the section lacks rows, the file's, and the index is rebuilt; it was taken for the guard's, the chain ended there and the catalog was lost, or thrown under ThrowOnLimit. Such a section is now the file's to lack rows, there and not looked for nearby, and the index is rebuilt. Data a guard cut or whose end it hides, a /Length the chain cannot read yet, and filters a reference holds say nothing. The test that held the guard's verdict on three bytes holds 56 now. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…ngth may hold Data an endstream proven its own ends under a /Length a guard hid decoded to its end, every filter's end report silenced, a Flate checksum silenced where the data ended in white space, and a predicted row undone wherever a filter of the chain ran out. That silenced a wrong checksum whose bytes all lie before the white space, reported one a comment completes, decoded white space and a comment's digits as LZW codes or hex, faults the whole read never meets, undid rows behind ASCIIHex that ended where its format says, and charged the file with an object stream's last member the end-of-line the search left out had shortened. The first filter now decodes the data as taken, and where it ends it as its format says with nothing to report, that is the data read whole. Otherwise it decodes it again up to where the run of white space and comments that ends it starts: what it meets before is the file's, running out there says nothing, and where its output may lack what the run holds, Flate's tail or LZW and RunLength codes, the data's end is the guard's, as cut, for the filters after it, the predictor and an object stream's reader. The filters after a first that ended whole decode as read whole. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… holds beside a cut The architecture, the diagnostics reference and the threat model said data an endstream proven its own ends past white space decodes as read whole, every filter's end silenced: its first filter now decodes it again up to the run of white space and comments when it does not end it as its format says, its output's end the guard's where it may lack what the run holds, and the filters after a first that ended whole report as read whole. The known gap is narrowed to match. The lists of what is the file's beside a value a guard cut gain data that holds too few rows whatever the value holds, in ADR 34, the architecture, the concepts and the references; the reader limits reference no longer calls a subsection of no rows a fault. The search costs no longer claim about twice the file for three ways a deferred stream's white space crosses three times, the lazy reading page counts them as the others do, and the probe's comments, its regrowth among the known gaps, and the scope of a value's cut whatever the type are said as the code does them. Refs #224, #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…as read whole A cross-reference stream beside a value a guard cut whose data cannot hold the rows declared was taken for a section that cannot be read: the chain ended there, its trailer and the older ones were not merged, so /Info, /ID or /Root only they give were lost, the validator judged no trailer value of it, and at a place a few bytes from the offset named it was passed over for an older revision's stream nearby, which was read in its place. Read whole, it is read and lacks rows. It now is so under the guard too: its trailer merged, the chain going on to its /Prev, the index rebuilt, and taken where it was moved to. A width past what an int counts no longer wraps the smallest row a section's widths allow, and a lone byte or row is said as one. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…nt no data holds, may lie The first filter under a hidden end decoded the data as taken and, where it found anything to report, again up to the run of white space and comments that ends it. A % the data starts with was taken for a comment and the whole data dropped; in ASCII85, where a % is a digit, a line starting with one was dropped and not marked cut; spaces after LZW codes spelled an end code taken for the data's own; the chain's Flate predictor ran on an ASCIIHex step; raw deflate, or a zlib checksum past bytes that look like a comment, lost its last bytes to the guard; and data of white space alone under ASCIIHex was marked cut. A comment now starts only past white space within the data, and ends no ASCII85 data. Flate data whose deflate body ends in the data as taken is whole, its repair reported, a checksum that disagrees only where it reaches into the run silenced. LZW, RunLength and ASCII data decode up to the run, their end markers counting only there, and the predictor runs on Flate and LZW steps alone. Data that is all white space under ASCII85 or ASCIIHex is whole. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…tream parts A cross-reference stream whose /Length only its own rows place is taken to the first endstream, and its rows may spell one. The section's own row for that /Length is read from the rows held, but where the spelled endstream parts the predicted row that holds it, the data, decoded as data whose end the file gives, dropped that row: the length was not found, the section was charged with the rows it lacked, and a valid file's index was rebuilt. The data is now decoded once more knowing its end hidden, its parted row undone as far as it goes, and the row read from there; what that decoding reports is dropped with it. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…side a cut reads whole The architecture, the references, the concepts, the threat model and the XML docs said data a hidden length ends decodes as read whole where its first filter ends it with nothing to report. They now say what the code does: the first filter decodes up to the run of white space, and of comments past white space within the data but in ASCII85, and Flate data whose deflate body ends in the data as taken is whole. A cross-reference stream lacking rows beside a value a guard cut is said to be read as read whole, its filters written direct, rather than unreadable. A deferred length's row in a predicted row a spelled endstream parts, the search costs past the window, and the comments the concept pages left out are said as well. Refs #220, #224, #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… a guard hid A cross-reference stream whose data cannot hold the rows its values declare, beside a value a guard cut, was read as read whole: its trailer merged and the chain going on. The rows its data held, read with the cut value, were not read, and an older section's rows stood for them, so a superseded catalog was served. The cut value was never loaded, so the guard was neither reported nor thrown under ThrowOnLimit, and the validator charged the file with a wrong /Size the hidden rows would have set right. The cut value is now loaded once the chain is read. Where the data may hold a row at the fewest bytes a row may take, the section is also the guard's: the chain ends there, as where rows a guard cut, the size rule leaves it unjudged and the validator says it went unchecked. Data that holds no row leaves the chain going on. Refs #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…tream parts, guard kept A cross-reference stream whose /Length only its own rows place is taken to the first endstream its rows may spell. Where that endstream parts the predicted row holding the row for the /Length, the data was decoded once more knowing its end hidden only when the row lay past the rows the first decoding held. A PNG predictor whose first row is parted leaves the data as decoded, and the TIFF predictor leaves a parted row differenced: the row was read garbled, and a valid file's index was rebuilt. That decoding also dropped a decoding guard it reached, which ThrowOnLimit threw: the rows past the bound were charged to the file in one mode and thrown in the other. The data is now decoded once more wherever a subsection numbers the object and no row standing for it was found, and the row looked for in the rows that decoding holds. A guard that decoding reaches stands, and where it found no row the rows past the bound are the guard's, as where the first decoding reaches it. A test holds data that holds no whole row, which no row is looked for in. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
Two rules of the hidden end had no test that fails without them. Flate data whose deflate body ends in the data as taken is whole only where decoding stayed within MaxDecodedStreamLength: past it, the guard is reported and the data cut, as read whole. Data that is all white space and comments before its endstream decodes to nothing, cut by the guard, under LZW, Flate and RunLength. Refs #224 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…uard's The architecture, ADR 34, the references, the concepts and the threat model said a section whose data lacks the rows its values declare beside a value a guard cut reads as read whole, the chain going on. They now say the guard is reached once the chain is read, and that rows the data may still hold, read with the cut value, are the guard's: the chain ends there and the validator says them unchecked. A deferred length's row is said to be found in any predicted row a spelled endstream parts, a decoding guard that search reaches standing. A comment cut short is completed, a missing checksum and a first number below 0 are named, and two tests are cited. Refs #220, #224, #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
… far the guard lets rows read The last round of mutants found two rules of the deferred length's second decoding that no test held. The data is decoded again only where a subsection numbers the object the /Length names: one no row numbers is never looked for there, and a decoding guard that search would reach is not reported. Where that decoding stops at MaxDecodedStreamLength short of the /Length's row, the rows it holds are read as far as the guard lets them, a page's row in the parted predicted row among them, not only those the decoding before held. The RunLength chain's files share their predicted rows' writing. Refs #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
Remote corpus run 78 failed one test, the deferred length checked on data MaxStreamLength cuts past every row, at a bound of 29 bytes. Its bounds were those of the Flate data this container's zlib writes at CompressionLevel.Optimal; the runtime Microsoft ships, as CI installs it, bundles zlib-ng, which writes other bytes, so 28 of them held fewer rows. The same test binary, run under that runtime here, fails as CI did, and passes under the distribution's. The file now holds its Flate data in a stored block for that test, the bounds 49, 50 and 60 counted from its layout, which no compressor moves. The test that bounds the decoded data keeps the compressed form, which does not depend on it. The whole unit suite passes under both runtimes. Refs #170, #220 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…s or direct calls A coverage run of the unit suite on the committed corpus left 39 of the lines the batch changed partial or missed. Each one now has a test. A document takes the line where one reaches it; where no file can, the test calls the internal member directly. The new tests cover: - a rollback to a mark that records no decoding reports; - a range a source in memory lends nothing; - a window's edge outside the file; - a place outside the file said to end data beside white space; - data the file lost under the document; - a deferred length's chain of references one longer than a parse resolves; - sections that spend the endstream searches; - a section beside a cut /W whose data may hold rows; - ASCII85 white space under a hidden end. What no input takes stays partial: a null check on a set that is never null there, an operand a caller's precondition makes true, conditions that contradict each other, and the arms the compiler adds. Two faults the tests met are filed rather than pinned: a /Prev naming an object whose header a guard cut rebuilds the index once the chain is read, and a deferred length is reported past the chain's endstream once the searches are spent. Refs #170, #174, #220, #224, #263, #292, #293 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
…tion it records The test of a token the cross-reference probe met cut said the probe saw "xref" and the table's own window refused "xrefs". The probe now reads on past a token its edge reaches, so it reads "xrefs" whole, finds neither the keyword nor an object, and the table never sees it. The comment says so, and the test now holds the section the probe records: not found, the fault naming neither. Where MaxTrailerLength stops the probe right after "xref", the table's window refuses the token under another rule, which is filed. Refs #174, #295 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
The journal of 2026-10-25 records the batch: - the issues' shapes measured and the eleven questions settled with the maintainer; - the thirteen adversarial reviews and the decisions they asked for; - the coverage pass and the faults it filed; - what the batch did for its six issues, the tests and mutants, and the checks; - the debts it filed, and the maintainer's decision on the remote corpus's hosts. The figures at a glance follow the branch: tests, coverage and the lines left with why no input reaches them, CI and the remote corpus's runs 75 to 80, and the measurements of reading every page and of a wrong /Length. Step 4's list is brought up to date, with the fourteen debts placed in the batches they join. Refs #170, #174, #220, #224, #230, #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
AdCodicem
force-pushed
the
ccr-6d6c0ee6-dqm4lm-174
branch
from
October 25, 2026 12:35
d194cfd to
b9df537
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Owner
Author
|
Every test over the 240 other documents passed: 7,763 acceptance tests and 3,745 referee checks, none failed. This is the hosts' failure that #204 records. The fix decided today for it (the cache, more patient retries, and the copy of their own in #291) is not written yet, and none exists to port. I am re-running the job once. Generated by Claude Code |
The maintainer squashes the pull request, its 145 commits too many for a rebase. The record says so, gives the check of each commit as far as it had gone, and adds run 81 of the remote corpus. Run 81 lost two documents to web.archive.org at its first attempt and passed whole at its one rerun. Refs #170, #174, #220, #224, #230, #263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Batch 16 of M02's step 4: where a stream's data ends, what of it the reader keeps, and what a guard's cut leaves.
/Lengthis confirmed by anendstreampast any white space and comments,read on past the parser's window. An unconfirmed length gets one memoized search, in the bytes held and then the file,
up to the nearer of the next object's header and the next object the index places; it never grows a window, and stream
data never counts toward
MaxObjectLength. Ona76c9da, a/Length 2147483647before 21 MB read 21.6 MB, and fiftyof them in 17 MB read 1.08 GB. With no
endstreambefore the stop, the object'sendobjends the data, reported asthe file wrote it (
stream.truncated,stream.length-invalid).MaxEndStreamGapis removed.]or>>no longer takes the next object's entries:syntax.truncated-objectat the innermost, the object'sendobjjudged missing. Not in object-stream members, nor at top level. A trailer that lost its
>>no longer takes the nextupdate's entries.
MaxStreamLength(A stream's raw data is bounded only by the file: every cached stream keeps it, overlapping streams multiply it, and data Decode returns undecoded passes MaxDecodedStreamLength #170). A stream's encoded data is read from itssource each time it is asked for, a source held in memory lending it where it lies: 400 streams over one megabyte
held 403 MB.
PdfReaderLimits.MaxStreamLength, 256 MB by default,limit.stream-length, bounds what is read of onestream at once, data no filter decodes included; a
/Lengthpast what an array holds no longer throwsOutOfMemoryException./Lengththe chain cannot read yet (A cross-reference stream whose /Length only it indexes is found by an endstream search: past 64 KB of data it reaches limit.trailer and the index is rebuilt, and rows that spell endstream are misread #220). Its data is searched for itsendstreampast the trailer's window: 70 KB of rows reachedlimit.trailer, and now read as the direct twin. Withno
endstream, the section is malformed. Rows that spellendstreamare told from the data's end by the section'sown row for the
/Length, in a predicted row the spelledendstreamparts too, and the data is taken again to thelength it gives, where an
endstreamconfirms it./Lengtha guard cut or left out of the index (A stream whose /Length names an object a reader limit cut is reported as naming an object that holds null #224). Keyed on the object it leads to, never on the valueread: nothing is said of the length, the integer read before the cut only a place to look at. An
endstreamthesearch stops at ends data that is the file's only where what follows the object proves it; otherwise the data is the
guard's. Data whose end only the hidden length could say decodes so: what its first filter meets before the trailing
white space is the file's, its running out there the guard's.
encoded, an object stream so left hidden and its members the guard's — no catalog search, no
file.root-invalid. Across-reference stream's
/W,/Index,/Size,/Filter,/DecodeParms,/Prevor deferred/Lengthso cutmakes the section the guard's: the chain ends there, nothing is rebuilt, the guard is reached once the chain is read,
thrown at opening under
ThrowOnLimit. What makes the section unreadable whatever the value holds, or data that lacksthe rows the values read declare, stays the file's.
LimitSinceis gone: a guard reached loading a filter value nolonger hides the stream's own faults.
reference resolves, a whole reference a cut left read through; a classic table an
/XRefStmnames reads no/Prevof its own; the validator leaves unjudged what needs a chain a guard ended (
xref.checked-in-part).Why
Closes #170
Closes #174
Closes #220
Closes #224
Closes #230
Closes #263
The issues' shapes measured on
a76c9dawith a survey of the corpus's 35,716 confirmed streams. The maintainersettled the batch's eleven questions as recommended, then the questions of the reviews (R3-18, replaced by R4-1; R4-2,
R4-15, R4-23; R6-1 to R6-3; R8-1 to R8-3; R12-1 to R12-3). Thirteen adversarial review rounds, 325 findings, each put
to a refuter, then a coverage pass whose verdicts were each put to a skeptic; every major fixed in the batch with a
test that failed before its fix. The journal of 2026-10-25 in
docs/status.mdlists them all.Filed under M02:
MaxObjectLength's bound), with batch 27;checksum byte), with batch 26;
endstream's proof), White space before a section or an object counts toward MaxTrailerLength and MaxObjectLength, and a section named past more of it than the bound is charged to the file #289 (white space counted toward a window's bound) and A section lacking rows beside a cut value reads Read where its hidden value can make a whole read Malformed, and a hidden end inflates Flate up to three times #290 (thequestion R12-2 left; Flate inflated up to three times under a hidden end), with batch 27;
/Prevnaming an objectwhose header a guard cut, whose load rebuilds the index once the chain is read), Once the endstream searches have read all they may, a deferred length counting the end-of-line before the chain's endstream is reported as past it #293 (a deferred length reported
past the chain's
endstreamonce the searches are spent, the one the batch made), At the exact end of a guard's window,streamfollowed by a regular byte is taken for a stream keyword #294 (astreama guard's edgeparts taken for the keyword) and Where MaxTrailerLength stops the probe of a cross-reference section, what lies at or past its bound is judged under another rule #295 (the cross-reference probe at
MaxTrailerLength's bound);/Lengthis wrong), with batch 25.#264's first gap, a
/Prevthrough an object a guard cut, is paid here; its second stays for batch 19.Definition of done
The project's definition of done applies to every change:
docs/status.mdrecords the batch: the journal of 2026-10-25 and the figures at a glance.docs/websitematches what now exists — 263 pages built, the site's check cleandotnet formathas been runAnything reviewers should look at closely
endstreamunder a hidden/Length(R6-1). What follows the object's end — itsendobj, thenext object the index places, the end of the file,
xreforstartxref— proves theendstreamthe stream's own;otherwise its data is the guard's. Read once per place, within the search budget: about three times the file at most.
The residual risk — data that spells
endstreamthenxrefand goes on — is named in the threat model's known gaps./Length, data anendstreamproven its own ends past white spacedecodes up to that run, the first filter's running out there silenced; Flate data whose deflate body ends in the data
as taken is whole. A section lacking rows beside a cut value reads Read where its hidden value can make a whole read Malformed, and a hidden end inflates Flate up to three times #290 asks whether to stop decoding Flate two or three times there.
the guard's, the chain ending there, where it may hold one. A section lacking rows beside a cut value reads Read where its hidden value can make a whole read Malformed, and a hidden end inflates Flate up to three times #290 asks the maintainer whether a value that could make a
whole read Malformed should make the section the guard's.
?.on valuesnever null, compiler
HasValuechecks, defensive arms of private members, an operand a caller's precondition makestrue, contradicting conditions —, none missed;
src/99.42 % (8,124 of 8,171), against 99.51 % onmain. Six linesoutside the patch that
maincovered are left: one the compiler adds (SkipContainer's dispatch), five that onlyAt the exact end of a guard's window,
streamfollowed by a regular byte is taken for a stream keyword #294's and Where MaxTrailerLength stops the probe of a cross-reference section, what lies at or past its bound is judged under another rule #295's faults reach.docs/status.mdnames each.StreamLengthBenchmarksunder a medium job twice,mainand the branchalternated):
raw bytes;
/Length2 bytes too long costs about 2 µs and 120–300 B more a stream. Thesearch memo, which an in-window search now goes through and whose entries grew from 48 to 80 B, and the offsets
sorted once account for the bytes; three wastes on that path are filed (A stream whose /Length is wrong pays a boxed message, a read of bytes already held, and endobj passes it throws away #296).
241d701(3,688 tests, 3,682 passed, 6 skipped), the commits after it changing tests alone;mainand the branch, twice each: 76 lines apart, every code the same (docs/status.mdnames each);
dotnet format;Remote corpusrun 80, whole, and run 81, on the record, whole at its one rerun (two documentsweb.archive.org failed to serve at its first attempt, The remote corpus fetcher ignores Retry-After, logs no failed attempt, and downloads every document again on every run #204);
test on
d3c44d1, a commit that does not touch what it times, under load.Merge: squash, at the maintainer's choice (145 commits are too many for a rebase merge, and
maintakes no mergecommit). The title is
feat:, since the batch addsPdfReaderLimits.MaxStreamLength, so that semantic-release readsthe squashed commit as a feature.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PiFf9bygtdCAcN4va1ss8T