diff --git a/docs/adr/0034-every-valid-pdf-is-readable-and-the-readers-guards-are.md b/docs/adr/0034-every-valid-pdf-is-readable-and-the-readers-guards-are.md index 51dfad9c..74f04815 100644 --- a/docs/adr/0034-every-valid-pdf-is-readable-and-the-readers-guards-are.md +++ b/docs/adr/0034-every-valid-pdf-is-readable-and-the-readers-guards-are.md @@ -13,7 +13,7 @@ saying which bounds a sound file could reach, and follows [8](0008-facade-immuta in making them options on an immutable record. Implemented by `PdfReaderLimits`, `PdfReaderOptions.Limits` and `PdfReaderOptions.ThrowOnLimit`, -`PdfLimitExceededException`, the seven `limit.*` codes of `PdfDiagnosticCodes`, the `readerLimits` field of +`PdfLimitExceededException`, the eight `limit.*` codes of `PdfDiagnosticCodes`, the `readerLimits` field of `tests/corpus/manifest.json`, and `ReaderLimitsTests`. Amended on 2026-09-27, by the maintainer, while M08 was specified: one record bounds every hostile read, the @@ -25,7 +25,8 @@ can make it read, and is an internal constant even where a valid file reaches it of a name (`FileQuote.MaxBytes`), though PDF 2.0 allows a longer one, and at most the first and last four steps of a key path (`RuleText.PathEnds`); the whole value stays in the document. Such a constant's declaration says so, as `No guard (ADR 34): it bounds what a message repeats, not what is read`, in place of the reason no valid file -reaches it. `PdfObjectParser.EndObjLookahead`, which bounds what the reader sees of an object's end, was the first. +reaches it. `PdfObjectParser.EndObjLookahead`, which bounded what the reader saw of an object's end, was the first; the +2026-10-23 amendment made it the size of a first read, and no bound. Amended on 2026-10-14, by the maintainer, while [#164](https://github.com/AdCodicem/AdCodicem.Pdf/issues/164) and [#183](https://github.com/AdCodicem/AdCodicem.Pdf/issues/183) were fixed: a sixth guard, `MaxIndexEntries` @@ -80,6 +81,61 @@ member would start past the data, ends the header as a stray token does. And a h the pairs read. The consequence that raising `MaxDecodedStreamLength` multiplies what the cache of decoded object streams holds until M23 bounds it (T33) was stale: `ObjectStreamBudget` bounds it since 2026-09-28. +Amended on 2026-10-23, by the maintainer, while [#170](https://github.com/AdCodicem/AdCodicem.Pdf/issues/170), +[#174](https://github.com/AdCodicem/AdCodicem.Pdf/issues/174), [#220](https://github.com/AdCodicem/AdCodicem.Pdf/issues/220), +[#224](https://github.com/AdCodicem/AdCodicem.Pdf/issues/224), [#230](https://github.com/AdCodicem/AdCodicem.Pdf/issues/230) +and [#263](https://github.com/AdCodicem/AdCodicem.Pdf/issues/263) were fixed: an eighth guard, `MaxStreamLength` (256 MB, +`limit.stream-length`), bounds what is read of one stream's encoded data at once. The file alone bounded it: decoding +handed back data no filter decodes — none named, an image filter, one the library does not know, Flate over data that is +none — whole, past `MaxDecodedStreamLength` and unreported, and a `/Length` past what an array holds threw an +`OutOfMemoryException` each time the data was read. The data is now kept by nothing and read each time it is asked for, +so the guard bounds each read: past it the data is cut, what the filters lose past the cut is no damage, the stream is +reported once in its document's diagnostics and, as `limit.decoded-stream` is, with each decoding into diagnostics a +caller passes to `Decode`, and its length stays the one the file holds. `MaxDecodedStreamLength` keeps bounding what the +filters produce, so an uncompressed image past the default needs the new guard raised alone. Nor does a stream's data +count toward `MaxObjectLength`: a `/Length` the file cannot hold grew the window an object is parsed through up to that +bound, reading 21.5 MB to find five bytes and reporting `limit.object`; no window grows to find an `endstream` now, and +one search past the window does. What that search reads, with the white space read on after a declared length, is +bounded by `EndStreamSearchPasses`, four times the file for a document's searches together, which stays an internal +constant with its reason rewritten: a valid file is read past the parser's window only by the search for the data of a +cross-reference stream whose `/Length` the chain cannot read, by the search for the data of a stream whose `/Length` a +guard cut or left out of the index, and by white space and comments after a declared length that run past a first +read — about twice the file, three times over the white space a cross-reference stream's search takes as data that its +length, once read, is read on through; the proof that an `endstream` found under a hidden `/Length` is the stream's +own reads 48 or 64 bytes past each, once, and the white space it crosses once more at most — the run before an +`endobj` taken from the read-on that read it, a run a row names before the next header read once from the row —, about +three times the file in all (amended on 2026-10-24). +`MaxEndStreamGap`, the four bytes of white space accepted +between a stream's data and its `endstream`, is removed: more, or a comment, made a right `/Length` a wrong one, and +white space and comments are read on as far as they run, counted with the searches — a comment only once white space +comes before it, since a `%` right where the length ends is the data's. Comments after the `endobj` that ends a stream +with no `endstream` are white space too, read through the search's own reads: neither rule adds a bound. +`EndObjLookahead` is the size of that first read, no longer a bound. `MaxTrailerLength` bounds a cross-reference +stream's dictionary alone: the data of one whose `/Length` the chain cannot read is searched for its `endstream` in the +file rather than through a window the guard bounded — a stream of 70,000 bytes of rows reads as its twin with a direct +`/Length`, with no `limit.trailer`, and `ThrowOnLimit` no longer throws for it —, and data no `endstream` follows before +the next object the sections read so far place, or the end of the file, makes its section malformed: the file's fault, +and no guard's, unless that `/Length` leads to an object `MaxObjectLength` cut (below). A value read through a reference +to an object a guard cut or left out of the index — a stream's `/Length`, `/Filter` or `/DecodeParms`, a value of a +filter's parameters, a cross-reference stream's `/W`, `/Index` or `/Size`, a trailer's `/Prev` or `/XRefStm` — is the +guard's, not the file's: an integer the cut parted, `1` of `15`, is only a place to look for `endstream` at, never a +declared length, though a stream that lacks its `endstream` is still the file's fault, unless a header its data spells, +or an `endstream` nothing after it proves the stream's own — the next object the index places, a cross-reference +stream the chain read, the end of the file, `xref` or `startxref`, after the object's `endobj` or in its place (amended +on 2026-10-24) —, stops the search, the +data up to it then the guard's; a whole +reference the cut left leads on, as read whole; a chain of filters ends there, +its data left encoded; and a cross-reference section whose values lead to an object `MaxObjectLength` cut ends the +chain, nothing rebuilt — unless what the values read say makes it unreadable whatever the cut hides, or data whose length the file gives, its filters written direct, lacks the rows they declare whatever it hides — the file's fault, such a section read as read whole, lacking rows +(amended on 2026-10-24), the chain going on where the data holds no row: rows it may hold, read with the cut value, are +the guard's (amended on 2026-10-25) —, the older sections and their trailers the guard's to hide, and the guard reached once the chain +is read. So does a cross-reference stream's `/Length` the chain could not read that leads to such an object, wherever +only that length could say how far the data runs: no `endstream` comes before the next object or the end of the file, or +the rows spell one and none confirms what the cut left of the length, the rows past it then the guard's. A guard that +loading a value of a stream's filters reaches is that value's, not a cut of the stream's own data, whose faults it hid. +No bound is added for those values, nor for the next object's header that ends an object's open containers: no valid +object holds `obj` where a value is expected. + ## Context Invariant 4 treats everything read from a file as hostile: no allocation, loop or recursion is sized by a @@ -100,6 +156,7 @@ claims; neither bounded what it was said to, and the 2026-10-14 amendment made t | Bound | Value | What a valid file can do | |---|---|---| | A stream's decoded length, its filters before the last sharing one bound for what they pass along together, a `/Crypt` filter neither charged nor the last | 256 MB | The USGS topographic map in the remote corpus holds one image, 9,600 × 11,410 RGB, that decodes to 313 MB; qpdf reads it whole. A chain of filters whose outputs together pass the bound is valid too, and qpdf reads three thousand of them | +| A stream's encoded data, read at once | 256 MB | ISO 32000 sets no limit on a stream's length, and decoding hands back data no filter decodes as it is: the USGS map's image, stored without a filter, would be 313 MB of data | | An object's length, streams' data aside | 16 MB | A direct array or dictionary of that size: a page-tree node with a million kids | | A classic cross-reference section | 64 MB | About 3.3 million entries; the informative annex C of ISO 32000-1 gives Acrobat's own limit as 8,388,607 objects | | The number of cross-reference sections | 1,024 | A document saved incrementally more than a thousand times | @@ -123,8 +180,8 @@ hands its output out a piece at a time can read the stream (T28). We will read every PDF that is valid under ISO 32000, and treat each bound a valid file can exceed as a guard: active by default, reported when it is reached, and lifted by an option. -- **The options.** `PdfReaderOptions.Limits` is a `PdfReaderLimits`, an immutable record with the seven bounds - of the table: `MaxDecodedStreamLength`, `MaxObjectLength`, `MaxXRefSectionLength`, `MaxXRefSectionCount`, +- **The options.** `PdfReaderOptions.Limits` is a `PdfReaderLimits`, an immutable record with the eight bounds + of the table: `MaxDecodedStreamLength`, `MaxStreamLength`, `MaxObjectLength`, `MaxXRefSectionLength`, `MaxXRefSectionCount`, `MaxTrailerLength`, `MaxIndexEntries` and `MaxFilterChainLength`. `PdfReaderLimits.Default` keeps the values above; `PdfReaderLimits.Unbounded` takes the implementation's maximums — `Array.MaxLength` for a length, `int.MaxValue` for a count. A value of zero or less is refused; a value above the implementation's maximum is @@ -133,26 +190,34 @@ guard: active by default, reported when it is reached, and lifted by an option. configuring anything; one that reads a sound but exceptional file asks for it, by raising one bound or by choosing `Unbounded`. - **Reaching a guard** keeps what fits within it, as the reader does today, and reports it as a warning - under a code of its own — `limit.decoded-stream`, `limit.object`, `limit.xref-section-length`, + under a code of its own — `limit.decoded-stream`, `limit.stream-length`, `limit.object`, `limit.xref-section-length`, `limit.xref-section-count`, `limit.trailer`, `limit.index-entries`, `limit.filter-chain` — whose message names the property that - lifts it. What the parser met at the cut is the reader's, not the file's, and is dropped in favor of the guard. An - object, a section or a trailer is reported once, however often it is read again; a stream, by the first of its filters - that stopped there, each time it is decoded past the bound into diagnostics a caller supplies, and once in its - document's own, however often it is decoded (amended on 2026-10-15); a full index, once a document. The filters after one a guard stopped are given data the reader - cut: that it runs out — a lost tail, a missing checksum, a digit without its pair, a row the predictor cannot finish — - is not reported, and a fault in what was read is. The code `filter.limit-exceeded`, added by T31 and never released, becomes `limit.decoded-stream`. + lifts it. What the parser met at the cut is the reader's, not the file's, and is dropped in favor of the guard; so is a + value that leads to an object a guard cut or left out of the index (amended on 2026-10-23). An object, a section or a + trailer is reported once, however often it is read again; a stream, by the first of its filters that stopped there, + each time it is decoded past the bound into diagnostics a caller supplies, and once in its document's own, however + often it is decoded (amended on 2026-10-15); a stream's data `MaxStreamLength` cut, with each decoding into + diagnostics a caller supplies, and once in its document's own, as the reader reads it (amended on 2026-10-23); a full + index, once a document. The filters after one a guard stopped, and those of data `MaxStreamLength` cut, are given data + the reader cut: that it runs out — a lost tail, a missing checksum, a digit without its pair, a row the predictor + cannot finish — is not reported, and a fault in what was read is. The code `filter.limit-exceeded`, added by T31 and + never released, becomes `limit.decoded-stream`. - **What a guard leaves out of the index is not damage** (amended on 2026-10-14). The objects only a section past - `MaxXRefSectionCount`, rows past `MaxXRefSectionLength` or `MaxDecodedStreamLength`, or numbers past + `MaxXRefSectionCount`, rows past `MaxXRefSectionLength`, `MaxStreamLength` or `MaxDecodedStreamLength`, or numbers past `MaxIndexEntries` index read as null, and the validator treats them as unknown rather than missing, until the property is raised: the index is not rebuilt for them, which turned a sound file into an `xref.rebuilt` error (ADR 45) and could lose what the chain had read. A classic table cut by its length keeps its trailer. The chain ends - at a cut — a table's, a stream's rows, a section or a trailer cut wherever the cut falls —, since an older section, - or its trailer, would stand for what the cut left out; and a section a guard cut is where it was named, not looked for - near there. A rebuild that damage forces on such a chain keeps the entries the chain read; one the chain cannot do - without — a guard that leaves it no section — runs as before. + at a cut — a table's, a stream's rows, a section or a trailer cut wherever the cut falls, a cross-reference stream + whose `/W`, `/Index`, `/Filter` or `/DecodeParms`, or `/Size` with no `/Index`, leads to an object `MaxObjectLength` + cut, or whose `/Length` does where only it could say how far the data runs, a `/Prev` or an `/XRefStm` the reader + follows that does (amended on 2026-10-23 and 2026-10-24) —, since an older section, or its trailer, would stand for what the cut left out; and a section a guard + cut is where it was named, not looked for near there. One found near where it is named, whose values a guard keeps + from being read, or that declared itself a section — the `xref` keyword, `/Type /XRef` — before a guard cut it, is the + guard's there too (amended on 2026-10-24). A rebuild that damage forces on such a chain keeps the entries + the chain read; one the chain cannot do without — a guard that leaves it no section — runs as before. - **Or it throws.** `PdfReaderOptions.ThrowOnLimit`, false by default, turns every guard reached into a `PdfLimitExceededException`, a `PdfException` carrying the code and the name of the property: from - `PdfDocument.Open` when it is reached while opening, from `GetObject` or `Decode` when it is reached later. + `PdfDocument.Open` when it is reached while opening, from `GetObject`, `GetRawBytes` or `Decode` when it is reached later. A rebuild of the index that reaches a guard finishes first and throws after, since a rebuild is never run twice; each later lookup of an object a full index left out throws again. `Open` releases a source it was given to own when it throws. @@ -173,7 +238,8 @@ guard: active by default, reported when it is reached, and lifted by an option. with the defaults. - **Past the implementation's ceiling the rule is not yet met.** A stream that decodes past about 2 GB needs a decode that yields its output a piece at a time; that is T28, for M23, and until then - `limit.decoded-stream` reports it even under `Unbounded`. + `limit.decoded-stream` reports it even under `Unbounded`. A stream's encoded data, read in one piece, has the + same ceiling: `limit.stream-length` reports data past it even under `Unbounded` (amended on 2026-10-23). ## Consequences @@ -184,7 +250,7 @@ guard: active by default, reported when it is reached, and lifted by an option. - The defaults are judgments about what is exceptional, and one is already known to be crossed by a real document: a large-format scan decodes past 256 MB. They can be revised without breaking a caller, who may always set them. -- With `ThrowOnLimit`, any lazy operation — resolving an object, decoding a stream — may throw after `Open` +- With `ThrowOnLimit`, any lazy operation — resolving an object, reading or decoding a stream — may throw after `Open` returned; a caller who sets it must be ready for that wherever the document is used. - `Unbounded` is for documents the application trusts. Raised guards let a file make the reader hold what it asks for, up to the new bounds; the cache of decoded object streams keeps within `ObjectStreamBudget` but always diff --git a/docs/architecture.md b/docs/architecture.md index b53d9c68..2fcf86c4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -94,27 +94,71 @@ separate types, separate severity scales, and vocabularies that never share a co gave is copied before a relocation or a rebuild changes it, so that validation still sees the file's own. 3. **Lazy resolution** — `PdfReference.Resolve()` reads and parses the object on demand. A bounded cache avoids reparsing hot objects (the page tree, shared resources) without ever retaining the whole document. -4. **Streams** — stream data is neither read nor decoded until the caller asks, and then decodes under the - limits of the document it came from; a stream built in memory decodes under the defaults. A stream copied - from one document to another travels **encoded**, with no decompress/recompress cycle. Its declared - length is checked when it is parsed, against the `endstream` that must follow it: in the window the object - is parsed through, or by asking the file for the few bytes after it when the data runs past the window. - A length nothing confirms gives way to the first `endstream` after the data's start: found in the window, - grown up to the object's limit, when the declared length ends inside it or the `/Length` gives none; - searched in the file when a declared length runs past the window — once per stream, no further than the - nearer of the next object the file's index places, as the file wrote it (as the reader rebuilt it when the - document opened, when the file wrote none), and the first object header the file's bytes hold, or the end - of the file, the declared length kept when none lies before them, and the searches of a document reading - a few times the file at most. Neither bound changes as objects are read: what a search keeps depends on - the file's bytes and its index as written, and in the 401 corpus documents measured each stream — each - copy of an object at its offset — takes the same length whatever was read before it. A crafted file can - still make it depend on the number a stream is first read under, on the rebuild's scan for trailers, or, - once the searches of a document have read four times the file, on which streams were searched first - ([#138], M23). A stream the file ends inside runs to the end of the file. Each is a diagnostic. -5. **Guards** — every read sized by the file is bounded. A bound a valid file can exceed — a stream's - decoded length, an object's length, a cross-reference section's length and their number, a trailer's - length — is a `PdfReaderLimits` option, on by default and reported under its own `limit.*` code when - reached; a bound only an invalid file reaches is an internal constant (ADR 34). +4. **Streams** — stream data is neither read nor decoded until the caller asks, nor kept once read + (section 5), and decodes under the limits of the document it came from; a stream built in memory decodes + under the defaults. A stream copied from one document to another travels **encoded**, with no + decompress/recompress cycle. Its declared length is checked when it is parsed, against the `endstream` that + must follow it after white space and comments, however many — in the window the object is parsed through, or + in the file past it, read on as far as they run —, a comment counting only once white space comes before it: a + `%` right where the length ends is the data's, and leaves the length unconfirmed. A length nothing confirms, or + a `/Length` that gives none, goes to one search for the first `endstream` after the data's start ([#174]): in + the bytes the parser holds, then forward in the file — once per stream, no further than the nearer of the next + object the file's index places, as the file wrote it (as the reader rebuilt it when the document opened, when + the file wrote none), and the first object header the file's bytes hold, or the end of the file, and the + searches of a document reading a few times the file at most. A stream's data never counts toward its + object's length: no window grows to find its `endstream`. Neither bound changes as objects are read: what a + search keeps depends on the file's bytes and its index as written, and in the 401 corpus documents measured + each stream — each copy of an object at its offset — takes the same length whatever was read before it. A + crafted file can still make it depend on the number a stream is first read under, on the rebuild's scan for + trailers, or, once the searches of a document have read four times the file, on which streams were searched + first ([#138], M23). A stream whose `/Length` a guard cut or left out of the index may be valid, its length only + hidden: when a header its data spells — an embedded file's does — stops its search, no `endobj` of its own before + it, or an `endstream` it spells ends it, the data up to there is the guard's, cut by it, and nothing is said of the + file. An `endstream` is the stream's own, and its end the file's, only where what follows the object's end — its + `endobj`, or the `endstream` where none follows — is the next object the index places, a cross-reference stream + the chain read, the end of the file, or `xref` or `startxref`, read once for each place ([#224]); white space + before the keyword, which only the hidden length says is the data's or not, leaves the data's end to the guard as it + decodes (`PdfStreamData.EndHidden`). With no + `endstream` before the stop, the object's `endobj` ends the data where it + follows the declared length as an `endstream` would, or where it is the last one, a whole keyword — no regular + character, solidus or percent sign right before it, which would make it part of a token, a name or a comment that + starts there: the data before it, which no lexing reads, says nothing of one further back —, and only + white space and comments run from it to the stop. That last `endobj` is followed forward through the search's + own reads, which read every byte up to the stop already: no byte before the stop is read a second time, + backward, and a comment, which can be told only from the start of its line, is read as one. Without either, + a declared length that ends before the stop is kept, and the data otherwise runs to the stop — to the end of + the file, for a stream the file ends inside. Each is a diagnostic. What the search found is kept for the + stream; whether `endobj` follows the declared length is read again only when a rebuild changes that length. +5. **Guards** — every read sized by the file is bounded. A bound a valid file can exceed — a stream's data + read at once and its decoded length, the filters it passes through, an object's length, a cross-reference + section's length and their number, a trailer's length, the entries of the index — is a `PdfReaderLimits` + option, on by default and reported under its own `limit.*` code when reached; a bound only an invalid file + reaches is an internal constant (ADR 34). What a guard cut is the guard's, never the file's fault: a value + that leads to an object a guard cut or left out of the index — a stream's `/Length`, `/Filter` or + `/DecodeParms`, a cross-reference stream's `/W`, a trailer's `/Prev` or `/XRefStm` the reader follows — is not + judged as one the file gave. The stream's data is found by its `endstream`, an integer the cut left only a place to look for + one at, and a missing `endstream` is still the file's fault but where the search stops at a header the data + spells or an `endstream` nothing after it proves the stream's own (item 4); a reference the cut left whole is read + through, as a load reads it; its chain of filters ends there, with the data + left encoded; and the chain of cross-reference sections ends at that section, with nothing rebuilt and the + older sections and their trailers the guard's to hide, wherever near its named offset the section is found — + unless what the values read say makes it unreadable whatever the cut hides, or data whose length the file + gives, its filters written direct, lacks the rows they declare whatever it hides — the file's fault, such a section + read as read whole, lacking rows, its trailer merged and the chain going on where the data holds no row at the + fewest bytes a row may take: rows it may hold, read with the cut value, are the guard's, the chain ending there. An + object `MaxObjectLength` cut there is loaded once the chain is read, which reaches the guard — thrown at opening + under `ThrowOnLimit` —, for a value the chain keeps: not for a place it dropped as no section. A cross-reference + stream's `/Length` the chain could not read, which leads to such an object, ends the chain so wherever only it + could say how far the data runs: with no `endstream` before the next object or the end of the file, the + section is the guard's, not malformed; where the rows spell one and no `endstream` confirms what the cut left + of the length, the rows past it are the guard's — unless what follows the `endstream` the data was taken to + proves it the stream's own, as in item 4, which leaves the rows the data lacks to the file. Such data, whose end + a guard hides, is decoded again as data a guard cut, a group its end parts dropped and what a filter says of the + end silenced; a decoding guard the first decoding reached stands, and no more rows are held than it read. + Data an `endstream` proven its own ends past white space or comments (item 4) decodes up to where that run + starts — a comment past white space within the data, but in ASCII85 data, where a `%` is a digit —, the first + filter's running out there silenced, and where its output may lack what the run holds, its end the guard's, as cut; + Flate data whose deflate body ends in the data as taken is whole, its checksum alone reaching into the run. ### 3.2 The write path @@ -163,7 +207,10 @@ goes wrong. No intermediate layer may drop it. ## 5. Memory and CPU strategy - **Memory budget**: consumption follows the complexity of the **page** being processed, never the size of - the document. A ten-thousand-page report must generate in the footprint of a ten-page one. One decoded + the document. A ten-thousand-page report must generate in the footprint of a ten-page one. A stream's + encoded data is kept by nothing: it is read each time it is asked for — lent where it lies by a source held + in memory, read into an array of its length from any other —, at most `PdfReaderLimits.MaxStreamLength` of it + at once, so that streams whose data share a stretch of the file hold no copy of it each ([#170]). One decoded stream is still held whole, up to `PdfReaderLimits.MaxDecodedStreamLength` and at most `Array.MaxLength`, until M23 decodes a piece at a time ([#48]), in native memory if a measurement asks for it (ADR 35). Of a chain of filters, the filters before the last share that bound for what they pass along together, so that decoding a @@ -240,3 +287,6 @@ publication. [#34]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/34 [#48]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/48 [#138]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/138 +[#170]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/170 +[#174]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/174 +[#224]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/224 diff --git a/docs/milestones/M02.md b/docs/milestones/M02.md index 3300dfcb..5b0b0a18 100644 --- a/docs/milestones/M02.md +++ b/docs/milestones/M02.md @@ -100,6 +100,33 @@ and none lost is no finding ([#275]); and an object an object stream writes as a stream, and the definition of it, that the chain's index places: an object a rebuild serves from another, which does not list it or writes it as a stream, reads as null with no finding yet ([#276]). +Where a stream's data ends is settled once for each stream, by the same rules wherever the parser's window ends: a +declared length that an `endstream` follows, past white space and comments however many, a comment only after white +space, is the stream's; any other, or none, goes to one search for the first `endstream`, up to the nearer of the first +object header and the next object the index as written places, the object's `endobj` standing in for it when none lies +before; and a stream's data never counts toward `MaxObjectLength` ([#174]). A cross-reference stream whose `/Length` the +chain cannot read is searched alike past the window, with no header stopping it; where the row the section's reading +keeps for that `/Length` lies among rows that spell `endstream`, the data is taken again to the length it gives, and +data no `endstream` ends before the next object the sections read so far place makes the section malformed, the file's +fault and no guard's, unless that `/Length` leads to an object a guard cut ([#220]). The next object's header ends every +array and dictionary an object, or a trailer, left open, and the object lacks its `endobj` ([#230]). +A stream's `/Length`, `/Filter` or `/DecodeParms`, or a filter's parameter, that leads to an object a guard cut or left +out of the index is the guard's, never judged as a value the file gave: the length is not judged, the data is found by +its `endstream`, and one missing is still the file's fault, unless a header the data spells, or an `endstream` nothing +after it proves the stream's own — the next object the index places, a cross-reference stream the chain read, the end +of the file, `xref` or `startxref`, after the object's `endobj` or in its place —, stops the search, the data up to it then the guard's; the filters end there, +the rest of them left undone, +`HasImageFilter` seeing no image past it, and an object stream so left is hidden as one a guard cut, +`xref.checked-in-part`'s rather than `xref.object-stream-broken`'s. A cross-reference stream whose `/W`, `/Index`, +`/Filter` or `/DecodeParms`, or `/Size` with no `/Index` — or whose `/Length`, when no `endstream` ends its data — +leads to an object `MaxObjectLength` cuts while the chain is read ends the chain as a section a guard cut, found where +it is named or near there, as a `/Prev` or `/XRefStm` the reader follows that leads to one does, and nothing is rebuilt; +the rows its data lacks past the `endstream` such a `/Length` left it at are the guard's, unless that `endstream` is +proven the stream's own ([#224], [#263]). A stream's data is read each time it is asked for and kept +by nothing — validating object streams whose data share a stretch of the file holds no copy of it for each —, and no +more than `MaxStreamLength` of it is read at once, a guard reported once in the document's diagnostics and with each +decoding into diagnostics a caller passes ([#170]). + ## Slices 1. Findings, report, rule engine, profile plumbing, and a single trivial rule end to end. **Done on @@ -227,8 +254,8 @@ dictionary listed as a kid is read as a page without `/Parent`, `/MediaBox` or ` `/Contents` naming an object the file lacks (`t02-01-004`, `t02-02-001`, `t02-03-011`) is `object.reference-missing`, a warning, and `t02-01-004`'s page, which no tree lists, `page-tree.page-orphaned`. Two content-stream cases whose `stream` keyword is missing, or that have bytes between `endstream` and `endobj` (`t02-05-01-013`, `-017`), earn -`object.endobj-missing` for what follows their value; one without `endstream` (`-014`) leaves where its object ends -unknown, and its stream is the stream rules' to judge. +`object.endobj-missing` for what follows their value; one without `endstream` (`-014`) keeps the 68 bytes its `/Length` +declares, which its object's `endobj` follows ([#174]), and its stream is the stream rules' to judge. The object-shape rules answered for the rest of the catalog, page tree and page object cases. A root typed `/Pagez` (`t02-02-009`), a page typed `/Font` (`t02-03-002`) and a catalog of the wrong `/Type` (`t02-01-006`) earn @@ -406,12 +433,18 @@ By Diátaxis section, since [ADR 47](../adr/0047-the-user-documentation-follows- [#154]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/154 [#160]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/160 [#161]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/161 +[#170]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/170 +[#174]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/174 [#175]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/175 [#181]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/181 [#183]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/183 [#200]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/200 [#202]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/202 +[#220]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/220 +[#224]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/224 +[#230]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/230 [#237]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/237 +[#263]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/263 [#274]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/274 [#275]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/275 [#276]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/276 diff --git a/docs/status.md b/docs/status.md index 5bafc281..09a19c2a 100644 --- a/docs/status.md +++ b/docs/status.md @@ -21,57 +21,69 @@ Tracking workflow mirrors from `docs/roadmap.md` (*Debt and open points*, below) the maintainer's request on 2026-10-15, merged with [#254] on 2026-10-16; the thirteenth, [#128] and [#237], with [#253], with [#258] on 2026-10-17; the fourteenth, [#129] and [#175], with [#256] and [#259], with [#265] on 2026-10-21; the fifteenth, [#160] and [#161], object stream members, with [#200], [#202], [#255], [#257], [#273], - [#274] and [#275], is in [#278], under review. Then the other batches, slices 4 to 6 ([#60] to [#62]); then the + [#274] and [#275], with [#278] on 2026-10-22; the sixteenth, [#174] and [#170], stream data, with [#220], [#224], + [#230] and [#263], is in [#297], under review. Then the other batches, slices 4 to 6 ([#60] to [#62]); then the milestone's adversarial review, slice 7 ([#137], ADR 46), once every other issue filed under M02 is closed. The debts come from the reader's own work, the threat model's first version ([#135], `docs/threat-model.md`), M01's review after the fact ([#136], `docs/reviews/M01.md`), the planning of step 4 ([#199] to [#202]), its second batch ([#207] to [#210]), its third ([#212]), its fifth ([#215], [#216]), its sixth ([#218] to [#224]), its eighth ([#229], [#230]), its ninth ([#237]), its tenth ([#241], [#247]), its twelfth ([#250]), slice 4's blockers ([#253]), its thirteenth - ([#255] to [#257]), its fourteenth ([#259] to [#264]) and its fifteenth ([#273] to [#277]); [#123] moved to M20. + ([#255] to [#257]), its fourteenth ([#259] to [#264]), its fifteenth ([#273] to [#277]) and its sixteenth ([#279] to + [#283], [#287] to [#290], [#292] to [#296]); [#123] moved to M20. - **User documentation**: organized along Diátaxis since 2026-09-30 ([ADR 47](adr/0047-the-user-documentation-follows-diataxis.md), [#148]), on `main`: a tutorial held to its sample by a test, four how-to guides, four reference pages — the validation rules among them, moved from the project documents — with the API reference under them, and four explanations. Outside any milestone. - **Last milestone closed**: **M01 — Object model and tolerant reading** -- **Tests**: 4,689 unit on [#278]'s branch (19 skipped: 2 by design, the theory over the remote corpus's streams whose +- **Tests**: 5,332 unit on [#297]'s branch (20 skipped: 2 by design, the theory over the remote corpus's streams whose length is wrong, the fifteen remote documents of the theory over the corpus's filter chains, which have no document - without them, and the test held to PDFBox's remote report) + 1,495 integration (skipped without Docker) + 23 for the - remote corpus's fetcher + 43 for the roadmap's mirror on GitHub. With the remote documents fetched, on [#278]'s - branch: 7,060 unit with the 233 this container holds, 3 skipped — the laziness test on the two documents recorded as - unsupported until [#47], which every other test holds to their expectations, and the private manifest this container - lacks —; 3,688 integration against qpdf in its container, 3,682 passed and 6 skipped where qpdf cannot walk a damaged - document's pages; and, with all 242, `Remote corpus` run 74's 7,135 acceptance tests and 3,760 referee checks, on - the code [#278] ends with. + without them, the test held to PDFBox's remote report, and iPRES's stream that lost its `endstream`) + 1,495 + integration (skipped without Docker) + 23 for the remote corpus's fetcher + 43 for the roadmap's mirror on GitHub. + With the remote documents fetched, on [#297]'s branch: 7,704 unit with the 233 this container holds, 3 skipped — the + laziness test on the two documents recorded as unsupported until [#47], which every other test holds to their + expectations, and the private manifest this container lacks —, the same under the runtime Microsoft ships; 3,688 + integration against qpdf in its container, 3,682 passed and 6 skipped where qpdf cannot walk a damaged document's + pages; and, with all 242, `Remote corpus` run 80's 7,742 acceptance tests and 3,760 referee checks, on the code + [#297] ends with. - **Coverage**: on the committed corpus, as Codecov counts it (a line with an untaken branch is partial, the generated - Arlington tables left out), 99.51 % of `src/` — 7,273 of 7,309 lines on [#278]'s branch, 380 of its patch's 383 - measurable lines, two partial and one missed. The 36 left are those the rule of 2026-09-29 leaves (`CLAUDE.md`, - *Coverage*): `main`'s 34 since [#265], at the line numbers [#278] gives them, and two more. They are members that are - private, or of a private type, which no input reaches — eight lines of `ArlingtonWalk` (194, 925, 1029, 1039, 1040, - 1060, 1062, 1152) and three of a defensive branch of `PdfLexer` (161, 164, 165) —, a `?.` on an index never null - where it is read and a switch's default arm (`CrossReferenceProbe` 108 and 269, `PageTreePageOrphanedRule` 51, - `RootInvalidRule` 111 and 120), a line the compiler puts after a call that never returns (`PdfFileReader` 2218), - three branches the compiler adds that no input takes (`FileQuote` 145, an interpolation's buffer too small; - `PdfObjectParser` 875, a concatenation's null check; `PdfObjectParser` 795, the switch over a `/Length`'s presence, - whose every arm is taken); `EndStreamAfter`'s two defensive arms (`PdfFileReader` 2186, 2190, journal of batch 6), - the `HasValue` checks the compiler adds on a `long?` (3854, batch 5) and a defensive arm of the real's expansion - (`PdfReal` 66, batch 5), one of the six ways out of a cross-reference stream row's `switch` (3987) and `Repair`'s - guard against a second rebuild (5229, 5231); `KeepObjectStream`'s release of the contents it replaces (5184, 5186, - batch 13), the second of which batch 15 rewrote to weigh a header's tables with the data, and so the patch's one - missed line. Batch 14's five: the switch over a header's place (`CrossReferenceProbe` 167), whose target past the + Arlington tables left out), 99.42 % of `src/` — 8,124 of 8,171 lines on [#297]'s branch, 1,284 of its patch's 1,293 + measurable lines, nine partial and none missed. The 47 left are those the rule of 2026-09-29 leaves (`CLAUDE.md`, + *Coverage*), and five that only faults filed reach. `main`'s 36 since [#278] are 32 now, at the line numbers [#297] + gives them. They are members that are private, or of a private type, which no input reaches — eight lines of + `ArlingtonWalk` (194, 925, 1029, 1039, 1040, 1060, 1062, 1152) and three of a defensive branch of `PdfLexer` (161, 164, + 165) —, a `?.` on an index never null where it is read and a switch's default arm (`CrossReferenceProbe` 108 and 269, + `PageTreePageOrphanedRule` 51, `RootInvalidRule` 111 and 120), a line the compiler puts after a call that never + returns (`PdfFileReader` 2580), two branches the compiler adds that no input takes (`FileQuote` 145, an interpolation's + buffer too small; `PdfObjectParser` 1218, a concatenation's null check); a defensive arm of the real's expansion + (`PdfReal` 66, batch 5), one of the six ways out of a cross-reference stream row's `switch` (5187) and `Repair`'s guard + against a second rebuild (6458, 6460); `KeepObjectStream`'s release of the contents it replaces (6413, 6415, batch + 13). Batch 14's five: the switch over a header's place (`CrossReferenceProbe` 167), whose target past the enumeration's values the compiler adds and no input takes; the one over a kid's presence (`PageTreeWalk` 345), whose default the batch's arm for a cut object leaves to `Deferred`, which `GetPresence` never answers; the test that a judged catalog's `/Pages` is a dictionary or a stream (`ObjectGraph` 177), which the walk judged a root there, so is one; and two defensive lines of a rebuild: `FinishRebuild`'s `??=` when reading `/Root` reaches a guard - (`PdfFileReader` 5344), which always finds one held already — a `/Root` a guard cuts is a regular object, or a member + (`PdfFileReader` 6573), which always finds one held already — a `/Root` a guard cuts is a regular object, or a member of an object stream, that the rebuild read as it took in object streams, reaching the guard first —, and - `KeepMembersHiddenByGuard`'s position of a stream the rebuilt index lacks (5602), when every stream it marks hidden - was read from that index. Batch 15 adds two, in the probe's judging of the members only a rebuild places: the arms - that pass over an object stream the rebuilt index does not hold as a regular object, or whose header the reader - cannot read as one (`CrossReferenceProbe` 431), when the rebuild placed the members there for having found that - stream and read its header; and a `?.` on the chain's index (445), which a chain a guard stopped has read. A private - method nothing had called since the batch's reading at the chain's place, `TryParseAt`, which coverage found, is gone. - `codecov.yml` asks 95 % of each patch, and lets the project drop by half a point at most; the aim is 100 %. -- **CI**: green on `main` at `6a5aad3` (CI run 641), [#265]'s last commit. `Remote corpus` passed on every document in + `KeepMembersHiddenByGuard`'s position of a stream the rebuilt index lacks (6831), when every stream it marks hidden + was read from that index. Batch 15's two, in the probe's judging of the members only a rebuild places: the arms that + pass over an object stream the rebuilt index does not hold as a regular object, or whose header the reader cannot + read as one (`CrossReferenceProbe` 431), when the rebuild placed the members there for having found that stream and + read its header; and a `?.` on the chain's index (445), which a chain a guard stopped has read. Batch 16 takes three + off — `EndStreamAfter`'s two defensive arms, which a document whose searches are spent now takes, and the switch over + a `/Length`'s presence —, brings a fourth into its patch, the `HasValue` checks the compiler adds on a `long?` + (`PdfFileReader` 5040), and leaves fifteen. In its patch, nine no input takes: a `?.` on the set of what the chain + read, never null where it is forgotten (2517), and on a cross-reference stream's `/W` element, `/Size` and `/Index` + pair, which the parser never leaves null there (4575, 4612, 4628); those `HasValue` checks, on values already tested + (5040); a deferred length's search, always memoized by the parse before (4325), and its value, never null (4503), + defensive arms of private members; the operand of `ChecksumDisagrees` its one caller's precondition makes true + (`PdfFilterPipeline` 343); and a header's generation neither cut nor an integer, which the tests before it rule out + (`PdfObjectParser` 1570). Outside it, the dispatch the compiler builds over `SkipContainer`'s token kinds, to which + the batch's arm for a header adds a leaf no kind reaches (`PdfObjectParser` 1296); and five lines that inputs still + reach, but only to show a fault filed rather than pinned: a classic table's first token the probe's bound cut + (`PdfFileReader` 3349, 3351, 3352, [#295]), and `MarkCutByGuard`, which a `stream` a guard's edge parts reaches + (5878, 5880, [#294]). `codecov.yml` asks 95 % of each patch, and lets the project drop by half a point at most; the + aim is 100 %. +- **CI**: green on `main` at `a76c9da` (CI run 678), [#278]'s last commit. `Remote corpus` passed on every document in the nightly run 18, on `main` at `8c54490`, and in runs 19 and 20, on [#203]'s and [#211]'s branches, each rerun once after web.archive.org refused a document ([#204]), in run 21, on [#213]'s branch, at its first attempt, and in run 22, on [#214]'s branch, rerun once for the same refusal. The nightly run 23, on `main`, failed only because @@ -114,7 +126,15 @@ Tracking workflow mirrors from `docs/roadmap.md` (*Debt and open points*, below) expectation. Run 71, on `48c7172`, passed whole at its first attempt, and so did run 72, on `e69756b`. Run 73, on `0de8727`, the same tree under a corrected commit message, lost ECan's scan at its first attempt, web.archive.org refusing the connection ([#204]), every test over the 241 others passing, and passed whole at its one rerun. Run 74, - on the code [#278] ends with, passed whole at its first attempt: 7,135 acceptance tests and 3,760 referee checks. + on the code [#278] ends with, passed whole at its first attempt: 7,135 acceptance tests and 3,760 referee checks. The + nightly run 75, on `main` at `a76c9da`, passed whole; run 77 lost ECan's scan, its host timing out in the TLS + handshake, and run 79 the 88 documents of the ipres2017 archive, its host timing out ([#204]), every test over the + others passing. On [#297]'s branch, run 76, on `8f6a085`, passed whole; run 78, on `241d701`, lost the ipres2017 + archive the same way and failed one test, whose bounds the zlib of this container set and the zlib-ng of the runtime + CI installs does not keep, which `d468352` corrected; run 80, on that commit, passed whole at its first attempt: + 7,742 acceptance tests and 3,760 referee checks. Run 81, on `b9df537`, the first form of the record [#297] ends with, + lost two documents at its first attempt, web.archive.org refusing the connection or timing out ([#204]), every test + over the 240 others passing, and passed whole at its one rerun; CI run 780 is green on that code. - **Corpus**: 168 committed documents, 23.0 MB — 19 generated here, 3 from Word and PDF24 on Windows, 146 third-party files under attribution-only licenses (`docs/corpus-sources.md`). Beside them, a **remote corpus** of 242 documents we may use but not redistribute, fetched at a pinned SHA-256 and size (ADR 32), @@ -164,7 +184,7 @@ Tracking workflow mirrors from `docs/roadmap.md` (*Debt and open points*, below) | Operation | Document | Time | Allocated | |---|---|---|---| | Indexing | synthetic, 1000 pages, ~4 MB | 280 µs | 122 KB | -| Indexing, then reading every page | synthetic, 1000 pages, ~4 MB | 4.8 ms | 5.7 MB | +| Indexing, then reading every page | synthetic, 1000 pages, ~4 MB | 4.2 ms | 1.8 MB | | Indexing and walking the page tree | real ReportLab document, 1000 pages | — | 3.2 MB | | Indexing a classic table (xUnit, after a first reading) | generated, 300,000 objects | — | 16.9 MB | | Indexing a cross-reference stream whose rows Flate stores (xUnit, after a first reading) | generated, 300,000 objects | — | 29.5 MB | @@ -182,8 +202,8 @@ Tracking workflow mirrors from `docs/roadmap.md` (*Debt and open points*, below) | Decoding the same stream under a wrong checksum, kept whole | 4 MB decoded | 12.1 ms | 13.1 MB | | Decoding a stream that turns corrupt halfway, the 2 MB before the fault kept | 4 MB of content | 8.1 ms | 6.6 MB | | Decoding a chain of ASCII85 and Flate | 4 MB decoded | 10.4 ms | 13.7 MB | -| Indexing, then parsing every page's content stream, each past the reader's first window | synthetic, 1000 pages of 16 KB | 3.9 ms from memory, 7.1 ms from a file | 2.0 MB | -| The same, each stream declared 2 bytes too long and searched for its `endstream` | synthetic, 1000 pages of 16 KB | 7.3 ms from memory, 12.5 ms from a file | 2.8 MB | +| Indexing, then parsing every page's content stream, each past the reader's first window | synthetic, 1000 pages of 16 KB | 4.0 to 4.6 ms from memory, 7.2 to 7.4 ms from a file | 1.75 MB | +| The same, each stream declared 2 bytes too long and searched for its `endstream` | synthetic, 1000 pages of 16 KB | 10.6 to 11.3 ms from memory, 14.9 to 16.4 ms from a file | 2.71 MB | | Parsing one object, a medium job | Word's page dictionary | 6.0 µs | 4.6 KB | The gap between the first two rows is the library's promise: opening a document does not read its content. @@ -224,8 +244,10 @@ runs' spread. Indexing costs the map, 52 to 63 bytes an object, whatever the objects weigh, since a section makes room for its rows before reading them ([#164]); a map a rebuild grows costs 100 to 210, by where the count falls against its growth steps. A cross-reference stream adds 7 to about 42 bytes an object for its rows, and the object stream holding the catalog, -when one does, what it holds. The first two rows are batch 11's (ShortRun, twice): 394 KB and 6.1 MB on `main` in the -same runs, 304 to 324 µs and 5.5 to 6.5 ms. The third row is asserted as a budget in CI +when one does, what it holds. The first row is batch 11's (ShortRun, twice): 394 KB on `main` in the same runs, 304 to 324 +µs. The second is batch 16's ([#297], ShortRun, twice, alternated): no stream keeps its raw bytes since, each read from +its source when it is asked for, so reading every page takes 4.1 to 4.3 ms and 1.79 MB, against 6.3 to 7.1 ms and +5.82 MB on `main`. The third row is asserted as a budget in CI (`CorpusReadingTests`), so an allocation regression fails the build; it was 2.4 MB when M01 closed, and M01's review measured 3.2 MB on 2026-10-01, under the 4 MB budget. The three generated rows are held to 17.7 MB, 31.0 MB and 32.1 MB, 5 % over their figures (`DocumentReaderTests`, [#193], [#164]); growing the map allocated 31.1 MB, 43.1 MB and 33.2 MB; MB here are 2^20 bytes. A stream that ran out is @@ -253,7 +275,10 @@ read cuts of a header into the next, and the first read sized from what the pars beside it, 7.1 → 7.0 ms and 11.4 → 11.1 ms measured the same way. Data made of near-headers costs the search more than data that holds none: 16 MB of ` obj` over and over take about 63 ms to search to the end, of `0 0 obj ` 44 ms, of letters 10 ms — some 4 ms a megabyte searched at worst, and, the budget letting a document's searches read about -five times the file, some 20 ms a megabyte of file. Inside the window the check is what it was, but a wrong length +five times the file, some 20 ms a megabyte of file. Since batch 16 ([#297], medium job, twice, alternated) a sound +stream costs what it did, and a wrong length about 2 µs and 120 to 300 bytes more: one search inside the window or +past it, memoized with what follows the `endstream`, up to the next object's header or the next object the index +places; the last two rows were 8.5 to 9.1 ms and 12.4 to 13.6 ms, 2.59 MB, on `main` in the same runs ([#296]). Inside the window the check is what it was, but a wrong length now costs its record and its mark of one report: about 440 bytes more allocated per stream (the 4 KB streams, 2.20 → 2.64 MB), and about 180 bytes kept for the life of the document — 50,000 such streams, read with a cache of 64 objects, keep 9.1 MB more. A sound stream allocates and keeps nothing more. The 4 KB streams take 3.3 → 3.5 ms @@ -321,9 +346,10 @@ One pull request per batch, each design question put to the maintainer after mea cut — done, merged with [#265] on 2026-10-21, `Remote corpus` runs 63 to 65 green on its branch (journal of 2026-10-18). 15. [#160] and [#161], object-stream members, with [#200], [#202], [#255] and [#257], and [#273] to [#275], which the - batch filed — done, in [#278], `Remote corpus` runs 71 to 74 green on its branch, run 73 at its one rerun - (journal of 2026-10-22). - 16. [#174] and [#170], stream data, with [#220], [#224], [#230] and [#263]. + batch filed — done, merged with [#278] on 2026-10-22, `Remote corpus` runs 71 to 74 green on its branch, run 73 + at its one rerun (journal of 2026-10-22). + 16. [#174] and [#170], stream data, with [#220], [#224], [#230] and [#263] — done, in [#297], `Remote corpus` runs + 76 and 80 green on its branch (journal of 2026-10-25). 17. [#155]. 18. [#167] and [#168], with [#201] and [#277]. 19. [#188], [#197] and [#156], with [#207], [#208], [#210], [#221], [#222], [#241], [#261] and [#264]. @@ -357,7 +383,10 @@ One pull request per batch, each design question put to the maintainer after mea with [#174], [#170] and [#224], the stream data it concerns; [#264] with [#188] and [#197], the chain's reading of where its sections lie. Of the five batch 15 filed, [#273] to [#275] were paid in the batch at the maintainer's choice; [#276] goes with [#189], whose choice of the definition a rebuild serves it depends on; [#277] with [#168], - whose walk of the object streams' dependencies it concerns. + whose walk of the object streams' dependencies it concerns. Of the fourteen batch 16 filed, [#279], [#282], [#287], + [#289], [#290] and [#292] to [#295] go with [#163], the reader at its guards; [#280] and [#296] with [#181], the work + one document asks; [#281] with [#188], the chain's reading of where its sections lie; [#283] and [#288] with [#162], the + filter reports they reword. 5. Slice 4 ([#60]) in two pull requests, streams then fonts, after the decisions it waits on: the severity of a font that is not embedded, the standard 14's aliases, where text is "meant to be extractable", how the rules that need content are left out and shown so, and the tools that referee both families. @@ -379,6 +408,176 @@ not a fault of the file: the rules on it report at most, as information, that it ## Journal +### 2026-10-25 — Batch 16: where a stream's data ends, and what a guard's cut leaves ([#170], [#174], [#220], [#224], [#230], [#263]) +- **The question.** Measured on `a76c9da` by seven agents and a completeness critic. [#174]: a `/Length` no `endstream` + confirmed was searched for within the parser's window, then past it, a window grown for it counting toward + `MaxObjectLength` — a `/Length 2147483647` before 21 MB read 21.6 MB and reported `limit.object`, fifty such objects + in 17 MB read 1.08 GB, 63 times the file —, and a stream that lost its `endstream` took 49 bytes across the next + object's header; the corpus's 35,716 confirmed streams leave 0 to 2 bytes before `endstream`. [#170]: every stream + kept its raw bytes — 400 streams over one megabyte held 403 MB —, a `/Length` past what an array holds threw + `OutOfMemoryException`, and 50 MB no filter decodes came back whole under a 1 MB `MaxDecodedStreamLength`. [#220]: a + cross-reference stream whose `/Length` only its own rows place was read within the trailer's 64 KB window — 70 KB of + rows reached `limit.trailer` —, and rows that spell `endstream` served a superseded object. [#224]: a `/Length` a guard + cut read as "holds null", or as the integer the cut parted. [#263]: a `/Filter` or `/DecodeParms` naming an object a + guard cut made an object stream unreadable and set off a catalog search and Errors, and a cut value of a + cross-reference stream rebuilt the index; `LimitSince` blamed a guard reached while a filter value loaded on the + stream. [#230]: an object that lost its `]` or `>>` took the next object's entries. Eleven questions followed. +- **Settled with the maintainer**, all eleven as recommended (2026-10-22 and 23): + - a `/Length` is confirmed by an `endstream` past any white space and comments, read on past the window and counted + in `EndStreamSearchPasses`; `MaxEndStreamGap` removed (Q1); + - one search for every unconfirmed length, in the bytes held then the file, up to the nearer of the first header and + the next object the index places, memoized; stream data never counts toward `MaxObjectLength` (Q2); + - with no `endstream` before that stop, the object's `endobj` plays its part, each case reported as the file wrote it + (Q3); + - the next object's header ends every container left open, with `HeaderText`'s rule, in files and trailers, not in + object-stream members nor at top level (Q4); + - nothing of a stream's raw data is kept, a source held in memory lending it; the overlap's reads on a file source + filed as debt (Q5); + - a new guard, `MaxStreamLength`, 256 MB, `limit.stream-length`, bounding what is read of one stream's encoded data + (Q6); + - a deferred cross-reference stream's data searched past the window, no header stopping it, a carriage return before + the end-of-line kept; `MaxTrailerLength` bounds the dictionary alone (Q7); its data settled before its rows merge, + taken again where its own row for the `/Length` gives a length an `endstream` confirms (Q8); + - a `/Length` keyed on the object it leads to: one a guard cut or left out of the index is the guard's, the integer + read before the cut only a place to look at (Q9); + - a cross-reference stream's value that leads to an object a guard cut makes the section the guard's, the chain + ending there, nothing rebuilt (Q10); + - `LimitSince`, the trailer that lost its `>>`, and `FileStreamData`'s second copy paid here; the parted tokens and + the overlap's reads filed (Q11). +- **Reviewed** thirteen times, each finding put to a refuter in its own worktree, 325 findings in all. Every major was + fixed in the batch, with a test that failed before its fix; what was left is filed below. + - The first round, in two passes over six lenses, gave 30 findings, 6 major, then 22. Decided, as recommended: a + comment counts as white space after a declared length only past white space (F6), and after the object's `endobj` + (F5); a missing `endstream` under a `/Length` left out of the index reported as under one cut (F8); a deferred + cross-reference `/Length` cut with no `endstream` the guard's (G3, X1); `limit.stream-length` reported to a + caller's diagnostics on each decode (G4); `HasImageFilter` reading what `Decode` reads (G2). + - The second, 22 findings, two settled as the first round's rules; the third, 20. Decided after the third: a header + the data spells, under a hidden `/Length`, passed for the next object the index places (R3-18), which the fourth + replaced. + - The fourth, 25 findings, 11 major. Decided: no second pass — data a spelled header stops under a hidden `/Length`, + with no `endobj` of the object's own before it, is the guard's (R4-1); a reference the guard's edge parts after + `N G ` no missing `endobj` (R4-23); a member value ending at `MaxObjectLength` filed (R4-2, [#282]); a hybrid stream's + own `/Prev` judged as written (R4-15). + - The fifth, 24 findings, 20 kept, 5 major; the sixth, 22, 6 major. Decided after the sixth: an `endstream` under a + hidden `/Length` ends data that is the file's only where what follows the object proves it — its `endobj`, the + next object the index places, the end of the file, `xref` or `startxref` (R6-1), the chain aligned on that proof + (R6-2); a filter fault reported over data a guard cut filed (R6-3, [#283]). + - The seventh, 27 findings, 24 kept, 6 major; the eighth, 23, 20 kept, 8 major. Decided after the eighth: data an + `endstream` proven its own ends, past white space only a hidden length says is the data's or not, has its end + decoded as hidden (R8-1); the cross-reference streams the chain read place the object after a stream, rows a newer + section superseded filed (R8-2, [#287]); a place a section may have moved to is kept as the guard's only where it + declared itself one before the cut (R8-3). + - The ninth, 35 findings, 7 major; the tenth, 22, 15 kept; the eleventh, 29: the hidden end made precise — the first + filter decodes up to the run of white space and comments, what it meets before the file's —, a section lacking rows + beside a cut value the file's, and the cost of the proofs bounded to about three times the file. + - The twelfth, 30 findings, 14 major. Decided: each hidden-end case fixed as it stands, the two decodings kept (R12-1); + a section whose data lacks rows beside a cut value read as read whole (R12-2); one round more, limited to the + eleventh's and twelfth's changes, its majors fixed and the rest filed (R12-3). + - The thirteenth, 16 findings, 5 major, all fixed: a section lacking rows whose data may hold rows, read with the cut + value, is the guard's, the chain ending there, the guard reached once the chain is read and the validator saying the + rows unchecked; a deferred length's row looked for in any predicted row a spelled `endstream` parts — the first row + of a PNG predictor, a row the TIFF predictor left differenced —, a decoding guard that search reaches standing. + Its minors are tests, its nits documents; the question it raised of R12-2, and a Flate stream inflated up to three + times under a hidden end, filed ([#290]). + - The coverage of the patch, once the reviews were done, by an agent per group of lines and a skeptic per agent: of + the 39 lines the unit suite left partial or missed, 30 are held by tests through documents or direct calls, and 9 + no input takes. A second pass took the six lines outside the patch that `main` covered and the batch left: one the + compiler adds, five reached only by faults. The passes met four faults, each reproduced and filed rather than + pinned ([#292] to [#295]): one the batch made, a false warning once a crafted file has spent the searches + ([#293]), and three older ones, each of which the batch narrowed. +- **Done**, in [#297], 145 commits, squashed into one at the maintainer's choice, too many for a rebase: + - [#174]: a `/Length` confirmed past any white space and comments, read on past the window; one memoized search for + every unconfirmed length, up to the next object's header or the next object the index places, which never grows a + window; with no `endstream`, the object's `endobj` ending the data, as the file wrote it, `stream.truncated` or + `stream.length-invalid`; a found `endstream` settling a `/Length` the file cannot hold; + - [#230]: the next object's header ends every container left open, `syntax.truncated-object` at the innermost, and a + trailer that lost its `>>` no longer takes the next update's entries; + - [#170]: no stream keeps its raw data, a source held in memory lending it; `MaxStreamLength`, a guard, 256 MB, + `limit.stream-length`, bounds what is read of one stream at once, data no filter decodes included; + - [#220]: a deferred cross-reference stream's data searched past the window, no `limit.trailer`; with no `endstream` + the section malformed; rows that spell `endstream` told from the data's end by the section's own row for the + `/Length`, in a predicted row the spelled `endstream` parts too, the data taken again to it; + - [#224]: a `/Length` that leads to an object a guard cut or left out of the index is the guard's, nothing said of it, + an `endstream` the search stops at the file's only where what follows the object proves it, and data whose end only + the hidden length could say decoded as such — what its first filter meets before the trailing white space the + file's, its running out there the guard's; + - [#263]: a filter chain ends at a value a guard cut or left out of the index, the data left encoded, an object stream + so left hidden, its members the guard's; a cross-reference stream's value so cut makes the section the guard's, the + chain ending there, the guard reached once the chain is read, but for what makes the section unreadable whatever the + value holds, or data that lacks the rows the values read declare, the file's; `LimitSince` gone; + - the chain read without loading, its values followed as far as a reference resolves, a whole reference a cut left + read through; a classic table named by `/XRefStm` reading no `/Prev` of its own; the validator leaving unjudged what + needs a chain a guard ended. + + Docs: ADR 34's classifications (`MaxStreamLength` a guard; `EndStreamSearchPasses` an internal constant, its reason + rewritten; `MaxEndStreamGap` removed; a value through a reference to what a guard cut the guard's), amended on + 2026-10-23, 24 and 25; `architecture.md`; the threat model's rows on stream ends, deferred lengths, guard-cut values, + the chain's reads, costs and the known gaps; `reference/reader-limits.md`, `reference/diagnostics.md`, + `reference/validation-rules.md`, `concepts/reader-limits.md`, `concepts/validation.md`, `concepts/diagnostics.md`, + `concepts/lazy-reading.md`, a guide to `MaxStreamLength`; M02. +- **Tests**: 643 new unit tests, 4,689 to 5,332 — with the remote documents fetched, 7,060 to 7,704 —, in + `StreamLengthTests`, `GuardCutValueTests`, `WindowEdgeTests`, `DocumentReaderTests`, `CrossReferenceRuleTests`, + `ReaderLimitsTests`, `HostileInputTests`, `FilterDamageTests`, `ParserTests`, `DiagnosticsTests` and their neighbors, + with `ChainFiles` and `HandWrittenFile` building the files. +- **Mutation**: mutants of each round's fixes, built and the tests of their classes run on them, every survivor killed by + a test written for it or shown equivalent; the last round, 16 mutants over the eleventh to thirteenth rounds' code, + 14 killed by the tests as they stood and the other two by a test written for each. +- **Tracking.** Filed under M02: [#279], the parted tokens, and [#282], a member value at `MaxObjectLength`'s bound, with + batch 27; [#280], the overlap's reads on a file source, with batch 25; [#281], a section relocated onto one the chain + read, with batch 19; [#283], a filter fault's size over data a guard cut, and [#288], a deferred stream's kept carriage + return read as a checksum byte, with batch 26; [#287], superseded rows in the proof of an `endstream`, [#289], white + space counted toward a window's bound, and [#290], the question R12-2 left and the inflation under a hidden end, with + batch 27; and, from the coverage pass, [#292], a `/Prev` naming an object whose header a guard cut, whose load + rebuilds the index once the chain is read, [#293], a deferred length reported past the chain's `endstream` once the + searches are spent, [#294], a `stream` a guard's edge parts taken for the keyword, and [#295], the cross-reference + probe at `MaxTrailerLength`'s bound, with batch 27; and, from the benchmarks, [#296], three wastes on the path of a + stream whose `/Length` is wrong, with batch 25. [#264]'s first gap, a `/Prev` through an object a guard cut, is + paid here, the issue told so; [#173], [#181], [#279] and [#282] were told of the shapes the reviews met in their + classes. +- **The remote corpus's hosts.** RADAR timed out on the ipres2017 archive in runs 78 and 79, as in run 42, 88 + documents lost at once. Decided by the maintainer on 2026-10-25 ([#204]): the remote documents a license lets the + project copy are kept, unmodified, in a repository of their own ([#291]), `tests/corpus/remote/` is cached between + runs where no pull request can restore it, and the fetcher waits as hosts ask; one pull request after this batch. +- **Checked**: + - each commit of the branch built alone with no warning and run through the unit suite, a check still running when + this record was written: the first 30 build and pass, 7,065 to 7,297 tests with the remote documents fetched, but + for one run of the test of the validator's linear time over an object stream's members, on `d3c44d1`, a commit + that does not touch what it times, while other work loaded the machine. The squashed tree is the one CI and the + checks below ran on. commitlint finds no problem in the messages; + - the integration suite twice on `241d701`, 3,688 tests, 3,682 passed and 6 skipped, both runs; the commits after it + change tests alone; + - the unit suite under the runtime Microsoft ships, as CI installs it, zlib-ng included, at `2055eeb`: 7,704 tests, + 3 skipped, as under the distribution's; + - the 401 documents this container holds, 168 committed and 233 remote, opened and validated on `main` and on the + branch at `241d701`, twice each, each tree giving the same lines both times: the trees differ in 76 lines, their + codes the same. `stream.truncated` names the declared length and the bytes to the end of the file in + `invoice-truncated-tail` and PDFBox's `pdfbox3949` and `pdfbox3950`, and JHOVE's `hul-138`; PDFBox's `pdfbox3947` + keeps the 1,302 bytes its object stream 417 declares, `stream.length-invalid` saying so and `filter.failed` that its + Flate stream ends before its data does, its header listing 47 objects in 3,414 decoded bytes rather than 81 from a + 32 KB block of zeros, and an Arlington finding's element moving from 182 to 192; iPRES's `t02-05-01-014` keeps the + 68 bytes its `endobj` follows; + - the documentation site, 263 pages, its check clean; + - `ReaderBenchmarks`, `ValidationBenchmarks`, `ParserBenchmarks`, `FilterBenchmarks`, `LexerBenchmarks`, + `StreamOpenBenchmarks` and `StreamLengthBenchmarks` (ShortRun), twice, `main` and the branch alternated, then + `StreamLengthBenchmarks` under a medium job, twice. Reading every page of a thousand takes 4.1 to 4.3 ms and 1.79 MB + against 6.3 to 7.1 ms and 5.82 MB, no stream keeping its raw bytes; opening allocates 450 bytes more, the reader's + new fields and five empty memos; the parser, the filters, the lexer and the validation rows allocate what they did, + to 40 bytes, opening and validating a thousand pages 7 KB less, their times within the runs' spread. A sound stream + costs what it did. A stream whose `/Length` is 2 bytes too long costs about 2 µs and 120 to 300 bytes more: 6.6 to + 7.8 ms against 5.2 to 5.8 ms and 2.71 MB against 2.41 MB for a thousand of 4 KB from memory, 10.6 to 11.3 ms against + 8.5 to 9.1 ms and 2.71 MB against 2.59 MB for 16 KB. The search memo, which a search inside the window now goes + through and whose entries grew from 48 to 80 bytes, and the index's offsets sorted once account for the bytes, to + the byte; three wastes on that path, a boxed `long?`, a read of bytes the window holds and `endobj` passes thrown + away, are filed ([#296]); + - `Remote corpus`: run 76, on `8f6a085`, passed whole. Run 78, on `241d701`, lost the 88 documents of the ipres2017 + archive, its host timing out ([#204]), and failed one test, a deferred length checked on data `MaxStreamLength` cuts + past every row, whose bounds were those of the Flate data this container's zlib writes, which the zlib-ng of the + runtime CI installs writes otherwise: `d468352` stores that data, and run 80, on it, passed whole, 242 documents, + 7,742 acceptance tests and 3,760 referee checks. Run 81, on `b9df537`, the first form of this record, lost two + documents at its first attempt, web.archive.org refusing the connection or timing out ([#204]), every test over the + 240 others passing, and passed whole at its one rerun. +- **Next**: batch 17, [#155]. + ### 2026-10-22 — Batch 15: object stream members ([#160], [#161], [#200], [#202], [#255], [#257], [#273], [#274], [#275]) - **The question.** [#160]: a member of an object stream was parsed over the whole decoded data, with no `MaxObjectLength`, and the header's tables were allocated from `/N` before a pair was read — at `13e06bb`, as the @@ -2349,3 +2548,19 @@ Until 2026-09-27 this section was a table whose rows were numbered T01 to T40; t [#276]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/276 [#277]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/277 [#278]: https://github.com/AdCodicem/AdCodicem.Pdf/pull/278 +[#279]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/279 +[#280]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/280 +[#281]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/281 +[#282]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/282 +[#283]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/283 +[#287]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/287 +[#288]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/288 +[#289]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/289 +[#290]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/290 +[#291]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/291 +[#292]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/292 +[#293]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/293 +[#294]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/294 +[#295]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/295 +[#296]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/296 +[#297]: https://github.com/AdCodicem/AdCodicem.Pdf/pull/297 diff --git a/docs/threat-model.md b/docs/threat-model.md index 4b072606..8104fcab 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -30,7 +30,7 @@ it stands, and each test it names was read. A defense found missing is not writt What a caller entrusts to the library when it opens a file it did not produce: - **The process's memory.** Memory follows the heaviest object the caller touches, not the size of the file - (invariant 2). No allocation is sized by a value from the file without a checked bound (invariant 4). The seven + (invariant 2). No allocation is sized by a value from the file without a checked bound (invariant 4). The eight guards cap what a valid but exceptional file may make the reader hold. - **The process's time, and the calling thread.** The aim is that every call does work in proportion to the bytes it reads, under the guards. Several shapes break it today ([#155], [#168], [#181]). No call can be @@ -97,19 +97,28 @@ programs, image files, content streams, XML, encryption, signatures and the netw | `PdfReaderLimits` | Default | Code | Where it applies | |---|---|---|---| | `MaxDecodedStreamLength` | 256 MB | `limit.decoded-stream` | Every filter's output: the last filter of a chain alone, the filters before it together, for what they pass along (`PdfFilterPipeline`) | -| `MaxObjectLength` | 16 MB | `limit.object` | The window a regular object is parsed through, grown ×8 from 8 KB (`PdfFileReader.TryParseNumberedAt`), and an object stream member's, from its start to the next member's (`ObjectStreamMemberWindow.Of`, `ReachMemberLimit`) | +| `MaxStreamLength` | 256 MB | `limit.stream-length` | One stream's encoded data, read in one piece each time it is asked for, by `GetRawBytes` or a decoding (`FileStreamData.GetBytes`): past it the data is cut, and what lies within it is decoded as data the reader cut, whose lost tail is no damage. Data decoding hands back as it is — no filter, an image filter, one the library does not know, Flate over data that is none — is bounded by it, not by `MaxDecodedStreamLength`. Its ceiling is `Array.MaxLength`, at which a `/Length` past what an array holds ends; the stream's length stays the one the file holds ([#170]) | +| `MaxObjectLength` | 16 MB | `limit.object` | The window a regular object is parsed through, grown ×8 from 8 KB (`PdfFileReader.TryParseNumberedAt`), and an object stream member's, from its start to the next member's (`ObjectStreamMemberWindow.Of`, `ReachMemberLimit`). A stream's data never counts toward it: no window is grown to find an `endstream` ([#174]) | | `MaxXRefSectionLength` | 64 MB | `limit.xref-section-length` | The window a classic cross-reference table is read through | | `MaxXRefSectionCount` | 1,024 | `limit.xref-section-count` | The `/Prev` links the chain follows | -| `MaxTrailerLength` | 64 KB | `limit.trailer` | A trailer the table's window cut, and a cross-reference stream's dictionary | +| `MaxTrailerLength` | 64 KB | `limit.trailer` | A trailer the table's window cut, and a cross-reference stream's dictionary alone, not its data ([#220]) | | `MaxIndexEntries` | 8,388,607 | `limit.index-entries` | The object numbers the index holds, refused ones counted: a row the chain reads, a header a rebuild reads, a member a rebuild expands, and the room a section makes before its rows (`PdfFileReader.AddFromChain`, `Refuse`, `HasRoomInRebuild`, `ReserveIndex`) | | `MaxFilterChainLength` | 64 | `limit.filter-chain` | The filters a stream's data passes through, counted as its `/Filter` array names them, `/Crypt` included: decoding stops before the first past them, and nothing past them is read (`FilterChain`). What comes back is still encoded, so a cross-reference stream or an object stream the guard ends is read as cut before its data (`PdfFilterPipeline.Decode`'s `leftEncoded`) | Reaching a guard keeps what fits and reports its code (`PdfLimitGuard.Reach`): once per position for an object written -in the file, a section or a trailer, once for each member of each definition of an object stream — at that stream's +in the file, a section, a trailer or a stream's data, once for each member of each definition of an object stream — at that stream's data start, which its members share —, once per document for the index, and for a stream decoded past the bound once in its document's -diagnostics, each time into diagnostics the caller passes, counted each time either way ([#144]). What a guard +diagnostics, each time into diagnostics the caller passes, counted each time either way ([#144]); data `MaxStreamLength` +cut, reported once at its position in the document's diagnostics, reaches diagnostics the caller passes with each decoding +that meets the cut too ([#170]). What a guard leaves out of the index reads as null until it is raised, and the index is not rebuilt for it: a guard's cut is no -damage. When the document was opened with +damage. A value read through a reference to an object a guard cut or left out of the index, through a chain of +references too, is the guard's as well, whatever the cut left of it — null, or a token it parted —, and the reader +judges none of it as the file's — but for a whole reference the cut left after the value, which is read through as a +load reads it, what it leads to judged as read whole. That holds for a stream's `/Length` ([#224]) and the values its filters read +([#263]), for a cross-reference stream's `/W`, `/Index`, `/Filter` or `/DecodeParms`, or `/Size` with no `/Index`, +that `MaxObjectLength` cut as the chain was read, and its `/Length` when no `endstream` ends its data, and for a `/Prev` +or `/XRefStm` the reader follows that leads to such an object (*Indexing*); it adds no bound. When the document was opened with `ThrowOnLimit`, reaching a guard throws `PdfLimitExceededException` instead. A stream read from a document decodes under that document's guards, however long after opening. `PdfReaderLimits` refuses a bound of zero or less and takes a length past `Array.MaxLength` as that length. Held by `ReaderLimitsTests` throughout; the @@ -239,7 +248,8 @@ Known gaps: |---|---|---|---| | The lexer looping, or reading past its buffer | Every token but the end of input consumes a byte. Look-aheads check the length, and a string that never closes ends at the buffer's end, and is told from a closed one by its token's length | `PropertyTests.The_lexer_terminates_on_any_bytes_and_stays_inside_the_buffer`; `LexerTests.Never_loops_on_a_stray_delimiter`, `Recovers_from_an_unterminated_literal_string`; `ParserTests.Reports_nothing_of_a_construct_closed_at_the_end_of_the_data` | Property, over buffers of about 100 bytes | | Nesting that exhausts the stack | `PdfObjectParser.MaxDepth` (128) and a stack probe at each container. Past either, `SkipContainer` counts brackets without recursion, and `syntax.depth-exceeded` is reported | `ParserTests.Refuses_to_follow_nesting_deeper_than_its_limit`; `HostileInputTests.Survives_deeply_nested_containers_inside_an_object`, `A_file_nesting_objects_deeper_than_the_stack_allows_reads_them_as_null_without_crashing` | No | -| A token or an object of any length, or one that never closes | The window grows ×8 from 8 KB up to `MaxObjectLength`, and the parser is told whether its window ends the data: only then is what it leaves open reported, once, `syntax.truncated-object` where the innermost construct opens, and a window that ends where the file does cuts nothing however long it is. Past the bound, the object is kept as far as it was read and `limit.object` is reported, beside what the parse met before the cut; nothing is reported of the token the cut reaches, nor of what follows once a look-ahead met it (ADR 34). An array or a dictionary an `endobj` finds open ends there, rather than take the objects after it — but for one that lost its own `endobj` too, which takes the next object up to its `endobj` ([#230]). The parses of one object add up to about 1.3 times the last window | `HostileInputTests.Survives_a_string_longer_than_the_largest_window`, `A_string_that_never_closes_takes_the_end_of_the_file_and_is_reported_once_where_it_opens`; `ReaderLimitsTests.An_object_is_read_no_further_than_its_bound`, `An_object_cut_by_its_bound_is_not_reported_as_damage`, `An_object_its_bound_cuts_anywhere_in_its_last_tokens_reports_the_bound_alone`, `What_an_object_its_bound_cuts_met_before_the_cut_is_still_reported`, `A_fault_after_a_stream_whose_length_the_file_cannot_hold_is_still_reported_beside_the_bound`, `An_object_its_bound_cuts_after_more_faults_than_the_diagnostics_hold_still_reports_the_bound`, `A_number_its_bound_cuts_is_no_number_out_of_range`, `An_object_as_long_as_its_bound_that_ends_the_file_is_read_whole`; `WindowEdgeTests.An_anomaly_inside_an_object_longer_than_the_window_is_reported_once`; `PropertyTests.What_a_window_reports_of_its_syntax_the_whole_data_reports_too_in_the_same_order`, `A_buffer_that_ends_at_a_window_s_edge_reports_nothing_of_the_end_of_the_data`; `SyntaxFaultTests.A_construct_the_file_ends_inside_is_reported_where_it_opens`, `A_container_an_endobj_ends_takes_nothing_of_the_objects_after_it`, `A_fault_of_an_object_read_through_a_window_the_file_goes_on_past_is_reported`, `A_rebuild_reports_no_fault_where_its_window_cuts_a_trailer`, `A_trailer_a_source_cut_short_ends_inside_is_reported_as_the_end_of_the_data`; `FileRuleTests.A_trailer_whose_closing_brackets_end_its_table_s_window_is_judged_by_what_they_close` | Growth yes, the bound no; the window's edge by property | +| A token or an object of any length, or one that never closes | The window grows ×8 from 8 KB up to `MaxObjectLength`, and the parser is told whether its window ends the data: only then is what it leaves open reported, once, `syntax.truncated-object` where the innermost construct opens, and a window that ends where the file does cuts nothing however long it is. Past the bound, the object is kept as far as it was read and `limit.object` is reported, beside what the parse met before the cut; nothing is reported of the token the cut reaches, nor of what follows once a look-ahead met it (ADR 34). An array or a dictionary an `endobj` finds open ends there, rather than take the objects after it, and one that lost its own `endobj` too ends at the next object's header (below). The parses of one object add up to about 1.3 times the last window | `HostileInputTests.Survives_a_string_longer_than_the_largest_window`, `A_string_that_never_closes_takes_the_end_of_the_file_and_is_reported_once_where_it_opens`; `ReaderLimitsTests.An_object_is_read_no_further_than_its_bound`, `An_object_cut_by_its_bound_is_not_reported_as_damage`, `An_object_its_bound_cuts_anywhere_in_its_last_tokens_reports_the_bound_alone`, `What_an_object_its_bound_cuts_met_before_the_cut_is_still_reported`, `A_fault_after_a_stream_whose_length_the_file_cannot_hold_is_still_reported_beside_the_bound`, `An_object_its_bound_cuts_after_more_faults_than_the_diagnostics_hold_still_reports_the_bound`, `A_number_its_bound_cuts_is_no_number_out_of_range`, `An_object_as_long_as_its_bound_that_ends_the_file_is_read_whole`; `WindowEdgeTests.An_anomaly_inside_an_object_longer_than_the_window_is_reported_once`; `PropertyTests.What_a_window_reports_of_its_syntax_the_whole_data_reports_too_in_the_same_order`, `A_buffer_that_ends_at_a_window_s_edge_reports_nothing_of_the_end_of_the_data`; `SyntaxFaultTests.A_construct_the_file_ends_inside_is_reported_where_it_opens`, `A_container_an_endobj_ends_takes_nothing_of_the_objects_after_it`, `A_fault_of_an_object_read_through_a_window_the_file_goes_on_past_is_reported`, `A_rebuild_reports_no_fault_where_its_window_cuts_a_trailer`, `A_trailer_a_source_cut_short_ends_inside_is_reported_as_the_end_of_the_data`; `FileRuleTests.A_trailer_whose_closing_brackets_end_its_table_s_window_is_judged_by_what_they_close` | Growth yes, the bound no; the window's edge by property | +| An object that lost both its closing delimiter and its `endobj` taking in the next object, up to that object's `endobj`, its own missing `endobj` unreported; a trailer that lost its `>>` taking the next update's entries; a header the window's edge cut short reported as the file's stray keys | Inside an array or a dictionary, where an element, a key or a value is expected, the next object's header ends every container left open: `N G obj`, digits alone, `N` in 1..`int.MaxValue` and `G` in 0..65,535, white space alone between them, and `obj` a whole keyword the window's edge does not reach — the header the search for a lost `endstream` stops at (`HeaderText`). A header the edge cuts short, `6 0` or `6 0 ob`, is the edge's, as a reference it cuts is: where no larger window follows — at `MaxObjectLength`, in the rebuild's trailer scan —, its numbers are not reported as keys that are not names, unless a whole token already rules a header out — a number 0, a generation past 65,535, a member, which no header follows —; `6 0 ob`, a keyword `obj` starts after two integers, which no reference can be, ends the containers as a header does, its number given to no key and nothing reported, while `6 0` alone, what the edge may have left of a reference, is kept as far as it was read, as any value the edge cuts. It is reported once, at the innermost container, as `syntax.truncated-object`, "An object header ended the object inside …", or, in a trailer or a value read alone, where no object is open, "An object header came inside …", and the object lacks its `endobj`. Containers `SkipContainer` passes over past `MaxDepth` end there too, and a trailer is read so as well. An object stream's member, which no header follows, and a value alone, which is no container, do not apply it: in `4 0 obj 5 0 obj`, object 4 holds the integer 5. Its cost, accepted: a closed container holding text that reads as a header ends there, and its `endobj` is judged missing; none of the 344,361 objects of the corpus holds one ([#230]). An `endobj` the window of `MaxObjectLength` ends exactly with is asked of the file, which says whether a regular character goes on past it: an object exactly as long as the guard reads whole, loaded or read by the chain, wherever its `endobj` comes — after its value, where a container it left open is, or as all an empty object holds ([#174]). A reference alone that the window's edge cuts after its generation leaves what follows it the edge's: the object is not said to lack its `endobj` at the guard's cut ([#224]). An `R` the edge reaches is asked of the file too: where a regular byte goes on past it, `6 0 Rx` or `6 0 RG`, the edge cut what may be a reference, and a `/Filter` or `/Length` read through the object goes no further than under a bound one byte smaller, rather than follow a reference the file never wrote past the guard's cut ([#224], [#263]) | `SyntaxFaultTests.An_object_that_lost_its_closing_delimiter_and_its_endobj_ends_at_the_next_object_s_header`, `The_next_object_s_header_ends_every_container_left_open_and_is_reported_at_the_innermost`, `Text_that_is_no_object_header_ends_no_container`, `A_header_inside_a_closed_container_ends_it_there_and_its_endobj_is_judged_missing`, `An_object_stream_member_holding_a_header_reads_it_as_tokens`, `Nesting_deeper_than_the_reader_follows_ends_at_the_next_object_s_header_too`, `A_trailer_that_lost_its_closing_delimiter_takes_nothing_of_the_update_after_it`, `A_header_the_guard_s_cut_leaves_undecided_is_the_guard_s`; `ParserTests.Ends_the_containers_an_object_left_open_at_the_next_object_s_header_and_leaves_it_to_be_read`, `Takes_an_object_header_a_window_s_edge_reaches_for_none_until_a_larger_window_shows_obj_whole`, `A_header_where_a_key_or_a_value_is_expected_ends_the_dictionary`, `A_value_alone_that_is_a_header_s_number_is_an_integer`, `Tokens_a_window_s_edge_leaves_undecided_as_a_header_are_no_stray_keys`; `WindowEdgeTests.An_object_that_lost_its_end_ends_at_the_next_header_wherever_the_window_edge_cuts_it`; `SyntaxFaultTests.A_header_the_guard_s_cut_leaves_undecided_gives_the_key_before_it_no_value`, `A_value_no_header_can_begin_with_keeps_its_place_at_the_guard_s_cut`, `An_object_a_guard_cuts_at_a_header_it_leaves_undecided_is_not_said_to_lack_its_endobj`, `Tokens_no_header_begins_with_are_reported_at_the_guard_s_cut_as_past_it`, `An_object_exactly_as_long_as_MaxObjectLength_reads_whole`, `An_object_the_chain_reads_exactly_as_long_as_MaxObjectLength_reads_whole`, `An_object_whose_endobj_ends_its_containers_at_MaxObjectLength_reads_as_under_a_larger_bound`, `An_object_a_guard_cuts_inside_a_reference_it_holds_alone_is_not_said_to_lack_its_endobj`; `GuardCutValueTests.An_R_a_window_s_edge_reaches_makes_a_reference_only_where_the_file_ends_it_there`; `ParserTests.Ends_a_container_skipped_for_its_depth_at_the_next_object_s_header`, `Reports_the_keys_of_a_member_a_window_s_edge_leaves_as_a_header_none_can_be` | Plausibly | | Look-ahead that makes parsing superlinear | Look-ahead is a constant number of tokens, so each token is lexed three times at most | `ParserTests.Distinguishes_a_reference_from_two_integers`, `Backtracks_when_two_integers_are_not_followed_by_R` | Property | | A number past a `long` or a `double` | The number parser never wraps: an integer past a `long` reads as the real nearest to it, and a number past a double's range as an infinity, which the object parser reads as null under `syntax.number-out-of-range`, quoting the number only in a report the diagnostics keep. A real reads as the double nearest to its decimal. The shape — signs, then digits around at most one period — is checked before the framework's parse, which would read `NaN` and `Infinity`; each digit is read twice at most, and nothing is allocated | `LexerTests.Reads_an_integer_too_large_for_a_long_as_a_real`, `Reads_a_number_past_the_largest_double_as_an_infinity_its_reader_can_tell`, `Parsing_a_number_allocates_nothing_however_long_it_is`; `PropertyTests.The_integer_parser_reads_every_long_and_never_wraps_past_one`, `The_real_parser_reads_the_double_nearest_to_the_decimal`; `ParserTests.Reads_a_number_past_the_largest_real_as_null_and_reports_it`, `Reports_no_number_its_double_holds_however_it_rounds`; `HostileInputTests.Reads_a_number_fifteen_million_digits_long_in_time`, `Quotes_no_number_beyond_what_a_real_can_hold_once_the_diagnostics_are_full` | Property | | An object number or a generation overflowing its cast | The parser builds a header only from a number in 1..`int.MaxValue`, and a reference only from one in 0..`int.MaxValue` — object 0 heads the free list and reads as null —, each with a generation in 0..65,535 (`PdfObjectId.MaxNumber` and `MaxGeneration`, with their reasons), checked before the cast. The index and object streams check theirs the same way, before narrowing: see *Indexing* and *Resolving objects and object streams* | `ParserTests.Refuses_an_object_header_whose_number_or_generation_is_out_of_range`, `Makes_no_reference_of_numbers_an_object_identifier_cannot_hold`; `HostileInputTests.Serves_no_object_under_a_number_past_a_long_that_once_wrapped_to_its_number` | No | @@ -255,10 +265,12 @@ Known gaps: - [#163] Nesting past 128 levels, and loads nested past 64, are cut though a valid file can reach both. They become guards. - [#262] A reference a guard cuts between its generation and its `R` reads as two integers. +- [#279] A name, an integer or a real that a guard's cut parts is kept as the part read — + `/Flate` of `/FlateDecode`, `1` of `15`, `1.0` of `1.5` — by every reader of the object it cut. A stream's `/Length` + and the values its filters read take such an object for the guard's ([#224], [#263]); a caller reading it is told of + the cut, not of the token it parted. - [#36] `PdfString.ToText` misreads PDFDocEncoding, PDF 2.0 UTF-8 strings and UTF-16 language escapes. - [#37] Interned names grow for the life of the process. -- [#230] An object that lost both its closing delimiter and its `endobj` takes the next object up to that object's - `endobj`, and `object.endobj-missing` is not reported for it. ### Indexing: the cross-reference chain, trailers and the rebuild @@ -266,16 +278,17 @@ Known gaps: |---|---|---|---| | A `/Prev` chain that loops | The offsets the file writes are kept, and a repeat ends the chain with `xref.chain-cycle`, placed at an offset in the file; two offsets past what a `long` holds are two, not one | `DocumentReaderTests.Stops_when_the_chain_of_sections_loops`; `CrossReferenceChainTests.Reports_a_loop_in_the_chain_at_the_section_it_loops_back_to_whatever_precedes_the_header`, `Places_a_loop_back_to_a_section_named_outside_the_file_at_what_named_it`; `CrossReferenceRuleTests.A_prev_that_names_a_section_already_read_is_a_loop_and_an_error`; `HeaderOffsetTests.A_chain_that_names_the_same_offset_outside_the_file_twice_loops`, `Two_offsets_past_what_a_long_holds_name_two_sections_not_one_loop`; `CrossReferenceRuleTests.A_loop_through_an_offset_the_chain_only_named_says_no_section_was_read_there` | Yes | | A chain without end | `MaxXRefSectionCount` counts the `/Prev` links; a cut where the chain goes on outside the file is placed at the section that names it, the offset in the message | `ReaderLimitsTests.Reaching_a_guard_keeps_what_fits_and_names_the_property_that_lifts_it`; `CrossReferenceChainTests.Reads_an_object_only_a_section_past_the_section_count_indexes_as_null_until_the_guard_is_lifted`; `HeaderOffsetTests.A_chain_cut_where_it_goes_on_past_what_a_long_holds_is_placed_at_the_section_naming_it`, `A_chain_cut_outside_the_file_names_where_it_goes_on_as_the_file_writes_it_the_header_added` | No | -| A section offset outside the file; `/Prev` or `/XRefStm` given as a reference | Offsets are checked against the file, before the header's offset is added. A `/Prev` or an `/XRefStm` written as a reference is read where a section already read places its object, without loading it, and a real with no fractional part as its integer; anything else names no section | `CrossReferenceChainTests.Reports_a_section_that_prev_names_past_the_end_of_the_file_once_at_the_section_naming_it`, `Reports_a_prev_that_is_not_an_offset_and_finds_what_it_named_when_asked`; `CrossReferenceRuleTests.A_prev_written_as_a_reference_or_a_real_is_followed_and_reported`, `A_hybrid_file_whose_xrefstm_is_a_reference_or_a_real_is_read_whole` | Yes | +| A section offset outside the file; `/Prev` or `/XRefStm` given as a reference | Offsets are checked against the file, before the header's offset is added. A `/Prev` or an `/XRefStm` written as a reference is read where a section already read places its object, without loading it, and a real with no fractional part as its integer; anything else names no section, but for one that leads to an object a guard cut or hid, which ends the chain for the guard (below) | `CrossReferenceChainTests.Reports_a_section_that_prev_names_past_the_end_of_the_file_once_at_the_section_naming_it`, `Reports_a_prev_that_is_not_an_offset_and_finds_what_it_named_when_asked`; `CrossReferenceRuleTests.A_prev_written_as_a_reference_or_a_real_is_followed_and_reported`, `A_hybrid_file_whose_xrefstm_is_a_reference_or_a_real_is_read_whole` | Yes | | A missing section tried at many places | Candidates within 512 bytes either side, nearest first, and 32 attempts at most per document (`MaxRelocationCandidates`); a section a guard cut is where it was named, and is tried nowhere else | `CrossReferenceChainTests.Reads_a_section_that_prev_names_a_few_bytes_off`, `Sections_named_a_few_bytes_off_are_looked_for_at_thirty_two_places_a_document_at_most`; `HostileInputTests.Opening_a_chain_of_hybrid_sections_whose_streams_never_close_reads_a_bounded_amount` | Yes | | A classic table of any length | Its window grows ×4 from 64 KB up to `MaxXRefSectionLength`, and the rows read are kept. A table the guard cuts, or whose rows run past a full index, keeps its trailer, found past the rows left unread by a search a megabyte at a time (`ReadTrailerPastRows`): comments are skipped, and anything else no row holds is the section's fault, reported, the search going on to the `trailer` keyword, so that its `/Encrypt` still refuses the document. The search starts at the start of the line the cut fell in, outside any comment and past the table's own keyword; it reads each megabyte once, every byte of it looked at before the next is read, however many comments or stray bytes it holds, and is not made for a place a section may have been moved to, a guess. The chain ends there, since an older section's rows would stand for the objects the unread rows define. What those rows place reads as null until the guard is raised | `ReaderLimitsTests.A_table_whose_keywords_the_bound_cuts_reports_the_bound`, `A_table_its_length_bound_cuts_keeps_its_trailer_and_the_encryption_it_names`, `A_table_its_length_bound_cuts_ends_the_chain_though_its_trailer_names_an_older_section`, `What_ends_the_rows_a_length_bound_cut_is_judged_as_it_is_where_none_did`, `A_table_a_length_bound_cuts_that_runs_to_the_end_of_the_file_has_no_trailer`, `A_table_its_length_bound_cuts_keeps_the_encryption_its_trailer_names_past_what_no_row_holds`, `A_table_cut_before_its_keyword_ends_keeps_its_trailer`, `Rows_past_a_full_index_are_not_read_and_the_trailer_past_them_is`, `A_cut_in_a_comment_among_the_rows_searches_for_the_trailer_from_the_comment_s_start`, `The_trailer_past_rows_left_unread_is_found_wherever_the_search_s_window_ends`, `A_trailer_past_rows_left_unread_that_its_own_bound_cuts_keeps_what_precedes_the_cut`; `DocumentReaderTests.The_search_for_the_trailer_past_rows_left_unread_reads_the_file_once`; `CrossReferenceChainTests.Reads_an_object_a_length_bound_cut_out_of_its_table_as_null_until_the_guard_is_lifted`, `A_section_found_near_where_prev_names_it_is_not_searched_past_the_rows_a_guard_left_unread`; `WindowEdgeTests.A_classic_table_reads_whole_wherever_its_window_edge_falls_near_its_trailer` | No: seeds stay under 120 KB | | A subsection claiming a huge count, or numbering objects past `int.MaxValue` | The count is not negative, and the first number and the last, first + count − 1, compared by subtraction so that a count as large as a `long` cannot wrap it, are at most `int.MaxValue`; past either, the table is malformed. The rows are bounded by the window, the entries they give by `MaxIndexEntries`, and a count claiming more rows than the table holds makes it malformed where the rows end. A row giving an object in use a generation outside 0..65,535 is refused alone, and the rows after it read; no older section's row then stands for its object, which is the rebuild's to find. A free row is read whatever its generation, as producers give the free list's head 65,536 | `CrossReferenceRuleTests.A_subsection_header_without_a_first_number_and_a_count_it_can_hold_makes_the_table_malformed`, `A_subsection_numbering_rows_past_the_largest_object_number_makes_the_table_malformed`, `A_subsection_claiming_more_rows_than_the_table_holds_makes_it_malformed_where_the_rows_end`, `A_row_giving_an_object_in_use_a_generation_no_object_can_have_is_refused_alone`, `A_free_row_whose_generation_is_past_65535_is_read_in_silence`; `DocumentReaderTests.Serves_no_older_revision_of_an_object_whose_newest_row_is_refused` | Yes | -| A trailer, or a cross-reference stream's dictionary, of any length | `MaxTrailerLength`. The chain ends at a section the guard cut, or whose trailer it cut: an older section, an `/XRefStm` or a `/Root` past the cut is unknown, and what only older sections index reads as null until the guard is raised, rather than as the revision an update replaced — a section found a few bytes from where it was named too, its trailer's cut kept. A `/Root` a guard hid — past a trailer it cut, in an older trailer the chain stopped short of, at the section count or at a section whose rows a guard cut or with which the index filled — is unknown, no fault of the file: the reader looks for the catalog among the objects it indexed, rebuilds nothing when none is one, and `file.root-invalid` says nothing ([#175]); a `/Root` that names an object a guard left out of the index — at a section count, rows a guard cut or a full index, as the chain was read or as a rebuild filled the index —, through a chain of references too, is taken as read, null until the guard is lifted, and no other catalog is looked for, which could only be another object, one an update superseded or a stray one ([#175]); rows a guard cut that run on to no trailer, which the reader looks for to the file's end, lack one, which is the file's fault, and the reader rebuilds for it as a read of the whole table does; so does a trailer whose `/Prev` or `/XRefStm` is no offset in the file, or one the chain has read — at a section count, a full index or rows a guard cut alike —, and a cross-reference stream whose data a guard cut past its last row, the chain going on; a `/Prev`, or a table's `/XRefStm`, written as a reference, or a chain of them as far as a resolution follows one, is unknown where it leads to an object that runs past `MaxObjectLength` — under its own header, not another object's —, or to one whose number the index read so far lacks and whose row may lie among those the guard left unread: any row past the cut of a table; of a cross-reference stream, a row its `/Index` or `/Size` numbers from where the guard stopped it — the row the index had no room for, up to the last its data holds, or past the data a guard cut —. Elsewhere it is read as a whole read reads it: object 0, a number whose row was refused, a free row, another object's, or a row a stream's data decoded whole does not hold names no section in either read, which is reported. A section the guard cut is where it was named, and is not looked for near there: what lies near it would stand for it. An older cross-reference stream whose dictionary the guard cut before it said what it is — no `/Type` yet — is a section the guard cut, not one missing; an array, a dictionary of another type, or a stream whose dictionary was read whole is no section, cut or not | `ReaderLimitsTests.A_trailer_that_the_table_bound_cuts_is_read_through_a_window_of_its_own`; `HostileInputTests.Opening_a_chain_of_trailers_that_never_close_reads_a_bounded_amount`; `CrossReferenceChainTests.An_object_only_a_section_past_a_cut_trailer_indexes_is_unknown_rather_than_missing`, `A_cross_reference_stream_whose_dictionary_a_guard_cut_before_it_declared_itself_is_no_section_missing`, `A_trailer_a_guard_cut_past_its_prev_ends_the_chain`, `ReaderLimitsTests.A_section_a_prev_names_a_few_bytes_off_ends_the_chain_where_its_trailer_is_cut`, `A_root_a_trailer_cut_before_it_hides_is_not_judged_missing`, `A_root_a_guard_kept_the_trailers_from_giving_is_unknown_and_rebuilds_nothing`, `A_root_naming_an_object_a_guard_left_out_of_the_index_is_taken_as_read_and_no_other_is_looked_for`, `A_root_a_rebuild_that_fills_the_index_leaves_out_is_taken_as_read_and_no_other_is_looked_for`, `Rows_a_guard_cut_that_run_to_no_trailer_leave_the_root_the_file_lacks_to_be_found`, `A_trailer_past_a_guard_whose_older_sections_lie_nowhere_hides_no_root`, `A_cross_reference_stream_a_limit_cut_past_its_rows_hides_no_trailer`, `A_prev_written_as_a_reference_reads_past_a_guard_as_a_whole_read_reads_it`, `A_prev_a_stream_that_fills_the_index_writes_as_a_reference_past_the_full_point_is_no_fault_of_the_file`, `A_prev_a_stream_that_fills_the_index_writes_as_a_reference_no_row_past_it_places_reads_as_a_whole_read_reads_it`, `A_prev_a_stream_that_fills_the_index_writes_as_a_reference_to_an_object_past_the_object_guard_hides_the_trailers`, `A_prev_written_as_a_chain_of_references_past_a_guard_is_followed_as_far_as_a_whole_read_follows_it`, `A_prev_a_stream_a_guard_stopped_writes_as_a_reference_is_unknown_only_for_a_row_it_left_unread`, `A_table_that_fills_the_index_hides_the_trailer_of_the_hybrid_stream_its_xrefstm_names`; `CrossReferenceChainTests.A_cross_reference_stream_a_guard_cut_is_not_looked_for_near_where_it_was_named`, `A_stream_a_guard_cut_where_prev_names_it_is_not_found_again_in_the_table_that_names_it`, `An_object_a_guard_cut_where_prev_names_a_section_is_a_section_missing_when_it_cannot_be_one` | Yes | -| What the chain refers to — a cross-reference stream's dictionary written with references, an indirect `/Length`, a relocation candidate's, a trailer followed by `stream` — loading objects while the index is still being read, and a correction, a rebuild or a null cached then served afterward as what the file wrote | Nothing loads while the chain is read (`PdfFileReader.ReadWithoutLoading`): an object is read where a section already read places it, at the offset its row gives, under the reference's number and generation, within `MaxObjectLength` and `MaxNestedLoads`, and is not a stream; a reference to a reference is followed no further than `PdfReference.Resolve` follows it; nothing is corrected, rebuilt, cached, recorded or reported, and a guard that cuts it leaves it unread. A value the rows need that cannot be read so makes the section malformed; a `/Length` is read and checked once the chain is, behind a copy of the index as the file wrote it. An object read at a section's offset that is no section leaves nothing of its reading | `DocumentReaderTests.Changes_nothing_of_the_index_while_the_chain_is_read`, `A_value_the_chain_reads_that_a_guard_cuts_leaves_the_section_whole`, `Reads_a_value_the_chain_refers_to_past_the_window_it_starts_with`, `A_place_a_section_may_have_moved_to_that_holds_another_object_leaves_nothing_of_its_reading`, `A_cross_reference_stream_whose_dictionary_the_trailer_guard_cuts_before_it_declares_itself_reaches_the_guard`, `Reads_a_cross_reference_stream_whose_length_only_it_indexes_as_its_direct_twin`, `Checks_a_deferred_length_against_the_data_once_the_chain_is_read`, `Reads_every_row_of_a_cross_reference_stream_whose_data_a_deferred_length_leaves_to_its_end_of_line`, `A_guard_a_deferred_length_reaches_throws_once_the_chain_is_read_when_asked_to`; `CrossReferenceRuleTests.A_cross_reference_stream_value_the_chain_cannot_read_leaves_the_index_as_the_file_wrote_it`, `A_trailer_followed_by_stream_whose_length_the_chain_cannot_read_has_its_entry_judged_as_the_file_wrote_it`, `A_value_the_rows_need_that_the_chain_cannot_follow_to_its_end_makes_the_section_malformed`; `HostileInputTests.Opening_a_chain_whose_value_leads_through_lengths_longer_than_the_stack_is_deep_reads_a_bounded_amount` | No | +| A trailer, or a cross-reference stream's dictionary, of any length | `MaxTrailerLength`, which bounds a cross-reference stream's dictionary alone, not its data ([#220]). The chain ends at a section the guard cut, or whose trailer it cut: an older section, an `/XRefStm` or a `/Root` past the cut is unknown, and what only older sections index reads as null until the guard is raised, rather than as the revision an update replaced — a section found a few bytes from where it was named too, its trailer's cut kept. A `/Root` a guard hid — past a trailer it cut, in an older trailer the chain stopped short of, at the section count or at a section whose rows a guard cut or with which the index filled — is unknown, no fault of the file: the reader looks for the catalog among the objects it indexed, rebuilds nothing when none is one, and `file.root-invalid` says nothing ([#175]); a `/Root` that names an object a guard left out of the index — at a section count, rows a guard cut or a full index, as the chain was read or as a rebuild filled the index —, through a chain of references too, is taken as read, null until the guard is lifted, and no other catalog is looked for, which could only be another object, one an update superseded or a stray one ([#175]); rows a guard cut that run on to no trailer, which the reader looks for to the file's end, lack one, which is the file's fault, and the reader rebuilds for it as a read of the whole table does; so does a trailer whose `/Prev` or `/XRefStm` is no offset in the file, or one the chain has read — at a section count, a full index or rows a guard cut alike —, and a cross-reference stream whose data a guard cut past its last row, the chain going on; a `/Prev`, or a table's `/XRefStm`, written as a reference, or a chain of them as far as a resolution follows one, is unknown where it leads to an object that runs past `MaxObjectLength` — under its own header, not another object's —, or to one whose number the index read so far lacks and whose row may lie among those the guard left unread: any row past the cut of a table; of a cross-reference stream, a row its `/Index` or `/Size` numbers from where the guard stopped it — the row the index had no room for, up to the last its data holds, or past the data a guard cut —. Such a `/Prev` or `/XRefStm` that leads to an object `MaxObjectLength` cut ends the chain there for the guard, as a cross-reference stream's `/W` does (below): nothing is rebuilt, the older sections and their trailers are the guard's to hide, and the guard is reached once the chain is read, reported or thrown at opening under `ThrowOnLimit`; a hybrid table's `/XRefStm` so hidden ends the chain with it, since the older sections' rows would stand for its stream's ([#263]). A hybrid file's stream's own `/Prev`, which the chain does not follow, going on through the table's, is not read at all, nor are the `/Prev` and `/XRefStm` of a table an `/XRefStm` names in a stream's place: neither names a section missing nor ends the chain for a guard, and the rule on trailer values judges them as written, a reference they hold not followed ([#263]). An object the guard cut after a whole reference it holds hides nothing: the chain reads through it, as a load does, and loads it once the chain is read. Elsewhere it is read as a whole read reads it: object 0, a number whose row was refused, a free row, another object's, or a row a stream's data decoded whole does not hold names no section in either read, which is reported. A section the guard cut is where it was named, and is not looked for near there: what lies near it would stand for it. An older cross-reference stream whose dictionary the guard cut before it said what it is — no `/Type` yet — is a section the guard cut, not one missing; an array, a dictionary of another type, or a stream whose dictionary was read whole is no section, cut or not | `ReaderLimitsTests.A_trailer_that_the_table_bound_cuts_is_read_through_a_window_of_its_own`; `HostileInputTests.Opening_a_chain_of_trailers_that_never_close_reads_a_bounded_amount`; `CrossReferenceChainTests.An_object_only_a_section_past_a_cut_trailer_indexes_is_unknown_rather_than_missing`, `A_cross_reference_stream_whose_dictionary_a_guard_cut_before_it_declared_itself_is_no_section_missing`, `A_trailer_a_guard_cut_past_its_prev_ends_the_chain`, `ReaderLimitsTests.A_section_a_prev_names_a_few_bytes_off_ends_the_chain_where_its_trailer_is_cut`, `A_root_a_trailer_cut_before_it_hides_is_not_judged_missing`, `A_root_a_guard_kept_the_trailers_from_giving_is_unknown_and_rebuilds_nothing`, `A_root_naming_an_object_a_guard_left_out_of_the_index_is_taken_as_read_and_no_other_is_looked_for`, `A_root_a_rebuild_that_fills_the_index_leaves_out_is_taken_as_read_and_no_other_is_looked_for`, `Rows_a_guard_cut_that_run_to_no_trailer_leave_the_root_the_file_lacks_to_be_found`, `A_trailer_past_a_guard_whose_older_sections_lie_nowhere_hides_no_root`, `A_cross_reference_stream_a_limit_cut_past_its_rows_hides_no_trailer`, `A_prev_written_as_a_reference_reads_past_a_guard_as_a_whole_read_reads_it`, `A_prev_a_stream_that_fills_the_index_writes_as_a_reference_past_the_full_point_is_no_fault_of_the_file`, `A_prev_a_stream_that_fills_the_index_writes_as_a_reference_no_row_past_it_places_reads_as_a_whole_read_reads_it`, `A_prev_a_stream_that_fills_the_index_writes_as_a_reference_to_an_object_past_the_object_guard_hides_the_trailers`, `A_prev_written_as_a_chain_of_references_past_a_guard_is_followed_as_far_as_a_whole_read_follows_it`, `A_prev_a_stream_a_guard_stopped_writes_as_a_reference_is_unknown_only_for_a_row_it_left_unread`, `A_table_that_fills_the_index_hides_the_trailer_of_the_hybrid_stream_its_xrefstm_names`; `CrossReferenceChainTests.A_cross_reference_stream_a_guard_cut_is_not_looked_for_near_where_it_was_named`, `A_stream_a_guard_cut_where_prev_names_it_is_not_found_again_in_the_table_that_names_it`, `An_object_a_guard_cut_where_prev_names_a_section_is_a_section_missing_when_it_cannot_be_one`; `GuardCutValueTests.A_section_offset_a_guard_cut_ends_the_chain_as_the_guard_s`; `GuardCutValueTests.A_prev_whose_chain_ends_at_its_last_object_a_guard_cut_ends_the_chain_as_the_guard_s`, `A_cross_reference_stream_s_prev_a_guard_cut_ends_the_chain_as_the_guard_s`, `A_section_whose_prev_a_guard_cut_keeps_the_chain_it_hides_when_it_is_found_where_it_was_moved`, `A_chain_a_guard_ends_at_a_hidden_prev_leaves_unjudged_what_needs_the_whole_chain`, `A_hybrid_stream_s_own_prev_is_not_followed_the_chain_going_on_through_the_table_s`, `A_table_an_xrefstm_names_in_a_stream_s_place_has_its_own_links_left_unread`, `A_prev_written_as_a_reference_to_a_real_is_judged_through_it_only_where_the_reader_follows_it` | Yes | +| What the chain refers to — a cross-reference stream's dictionary written with references, an indirect `/Length`, a relocation candidate's, a trailer followed by `stream` — loading objects while the index is still being read, and a correction, a rebuild or a null cached then served afterward as what the file wrote | Nothing loads while the chain is read (`PdfFileReader.ReadWithoutLoading`): an object is read where a section already read places it, at the offset its row gives, under the reference's number and generation, within `MaxObjectLength` and `MaxNestedLoads`, and is not a stream; a reference to a reference is followed no further than `PdfReference.Resolve` follows it; nothing is corrected, rebuilt, cached, recorded or reported, and a guard that cuts it leaves it unread — but for a whole reference the cut left after the value, which is read through as a load reads it, the object then loaded once the chain is read, which reaches the guard, or throws it at opening under `ThrowOnLimit`. A value the rows need that cannot be read so makes the section malformed — but for one that leads to an object `MaxObjectLength` cut, a `/W`, `/Index`, `/Filter` or `/DecodeParms`, a `/Size` with no `/Index`, an element or a parameter of one, or a `/Length` whose data no `endstream` ends (below): the section is then the guard's, counted as cut by it, the chain ends there and nothing is rebuilt, and that object is loaded once the chain is read, which reports the guard, or throws it at opening under `ThrowOnLimit` ([#263], [#224]); a `/Prev` or an `/XRefStm` the reader follows that leads to one ends the chain alike (above). A `/Length` that cannot be read so — no section read so far places it, or a guard cut it, at any link of its chain of references — waits for the chain: it is read and checked once the chain is, behind a copy of the index as the file wrote it, but for one a guard cut or left out of the index, which is not judged ([#224]). An object read at a place a section may have been moved to that is no section leaves nothing of its reading, nor what it read through; one read at the offset the chain names keeps all of its reading, as a load would — its reports, the guards it reached, what it read through —: an object read through a whole reference is loaded once the chain is read only for a value the chain keeps or what that offset holds — not for a place it guessed and dropped, nor for a `/Size` beside an `/Index`, which numbers no row. A cross-reference stream found near the offset it is named at, whose values lead to an object `MaxObjectLength` cut, is the section, the guard's there as where it is named exactly: the chain ends at it, nothing rebuilt ([#263]); so is one that declared itself a section before a guard cut it — a table's `xref` keyword, `/Type /XRef`. A place that declares another type — a CIDFont, whose `/W` gives widths —, or whose dictionary a guard cut before it said what it is, is a guess dropped, the next place tried; where the chain names a section, a guard that cut the dictionary of what declares another type cut no section — a CIDFont's, which read whole no stream data follows —, and relocation is tried, while one that cut a value of a stream there leaves it the section the guard cut, whatever type it gives, as read whole it is the section. What makes a section unreadable whatever a value a guard cut holds is the file's fault, as read whole, judged before the cut is taken for the guard's (`PdfFileReader.FaultBesideCut`): a value past the cut that no section read so far places, which the sections a whole read reads before make the file's to lack; a `/W` that gives no three widths, or none past 0; rows numbered wrongly — an `/Index` that holds no pairs, whatever its elements, a subsection out of range, or wrong by the element read whatever the cut one holds, a `/Size` with no `/Index` that gives no count —, as far as the values read, written direct or through references, say: a value a guard cut says nothing, nor does what depends on it, the `/Size` an `/Index` so cut may stand beside; and data whose length the file gives, its filters written direct, decoded whole, that holds fewer rows than the counts read declare however few bytes the widths read leave a row — a width past what an int counts taken as such —, which lacks them whatever the cut hides: the section is read as read whole, lacking rows, its trailer merged and the index rebuilt, where it was moved to too, and the object the guard cut loaded once the chain is read, which reaches the guard; the rows its data may hold at the fewest bytes a row may take, read with the cut value, are the guard's, the chain ending there, as where rows a guard cut, and the section said unchecked, its highest number not judged — the chain goes on to its `/Prev` only past data that holds none | `DocumentReaderTests.Changes_nothing_of_the_index_while_the_chain_is_read`, `A_value_the_chain_reads_that_a_guard_cuts_leaves_the_section_whole`, `Reads_a_value_the_chain_refers_to_past_the_window_it_starts_with`, `A_place_a_section_may_have_moved_to_that_holds_another_object_leaves_nothing_of_its_reading`, `A_place_a_section_may_have_moved_to_whose_dictionary_a_guard_cuts_is_no_section_the_chain_ends_at`, `A_cross_reference_stream_whose_dictionary_the_trailer_guard_cuts_before_it_declares_itself_reaches_the_guard`, `Reads_a_cross_reference_stream_whose_length_only_it_indexes_as_its_direct_twin`, `Checks_a_deferred_length_against_the_data_once_the_chain_is_read`, `Reads_every_row_of_a_cross_reference_stream_whose_data_a_deferred_length_leaves_to_its_end_of_line`, `A_guard_a_deferred_length_reaches_throws_once_the_chain_is_read_when_asked_to`, `A_deferred_length_whose_reference_leads_to_an_object_only_its_own_section_places_is_read_once_the_chain_is`; `CrossReferenceRuleTests.A_cross_reference_stream_value_the_chain_cannot_read_leaves_the_index_as_the_file_wrote_it`, `A_trailer_followed_by_stream_whose_length_the_chain_cannot_read_has_its_entry_judged_as_the_file_wrote_it`, `A_value_the_rows_need_that_the_chain_cannot_follow_to_its_end_makes_the_section_malformed`; `HostileInputTests.Opening_a_chain_whose_value_leads_through_lengths_longer_than_the_stack_is_deep_reads_a_bounded_amount`; `GuardCutValueTests.A_cross_reference_stream_value_a_guard_cut_ends_the_chain_as_the_guard_s`, `A_deferred_length_a_guard_cut_is_not_judged_once_the_chain_is_read`, `A_deferred_length_whose_reference_leads_to_an_object_a_guard_cut_is_not_judged_once_the_chain_is_read`; `DocumentReaderTests.A_deferred_length_read_once_the_chain_is_says_what_its_chain_leads_to`, `A_deferred_length_s_fault_names_the_reference_at_the_end_of_its_chain`, `A_deferred_length_whose_chain_is_too_long_to_follow_is_reported_at_opening_as_a_later_load_reads_it`; `GuardCutValueTests.A_cross_reference_stream_s_length_whose_chain_ends_at_its_last_object_a_guard_cut_waits_for_the_chain`, `A_value_the_chain_reads_through_a_whole_reference_a_guard_cut_after_is_read_and_reaches_the_guard`, `An_object_read_through_for_a_place_that_holds_no_section_reaches_the_guard_only_where_a_kept_value_reads_it`, `What_the_offset_a_section_is_named_at_holds_keeps_what_it_read_through_as_a_load_would`, `A_size_beside_an_index_that_names_an_object_a_guard_cut_after_a_whole_reference_is_not_read`, `A_cross_reference_stream_a_guard_cut_a_value_of_is_found_where_it_was_moved`, `A_cross_reference_stream_whose_deferred_length_a_guard_cut_is_found_where_it_was_moved`, `A_cross_reference_stream_a_guard_cut_the_dictionary_of_after_its_type_is_the_section_where_it_was_moved`, `A_classic_table_a_guard_cut_the_rows_of_is_the_section_where_it_was_moved`, `A_stream_with_no_numbering_whose_width_a_guard_cut_is_no_section_the_guard_cut`, `A_font_whose_widths_a_guard_cut_where_the_chain_names_a_section_is_no_section_the_guard_cut`, `A_place_a_section_may_have_moved_to_that_cannot_be_read_leaves_what_it_read_through_to_the_rebuild`, `What_makes_a_cross_reference_stream_unreadable_whatever_a_value_a_guard_cut_holds_is_the_file_s`, `A_place_a_section_may_have_moved_to_that_a_value_past_a_guard_s_cut_makes_unreadable_is_dropped`, `A_stream_whose_value_a_guard_cut_where_the_chain_names_a_section_is_the_section_the_guard_cut_whatever_its_type`, `A_cross_reference_stream_that_misspells_its_type_and_whose_value_a_guard_cut_is_the_section_the_guard_cut`, `A_section_a_value_s_cut_keeps_from_being_read_is_the_guard_s_where_nothing_read_says_it_is_wrong`, `A_section_whose_data_holds_fewer_rows_than_it_declares_whatever_a_value_a_guard_cut_holds_is_the_file_s_to_lack`, `Rows_a_section_lacking_rows_holds_beside_a_value_a_guard_cut_are_the_guard_s_and_no_older_section_stands_for_them`, `A_section_lacking_rows_beside_a_value_a_guard_cut_reaches_the_guard_once_the_chain_is_read`, `The_validator_charges_the_file_with_no_size_rows_a_section_lacking_rows_holds_beside_a_value_a_guard_cut_hide`, `Data_a_length_the_chain_cannot_read_yet_leaves_to_a_spelled_endstream_says_nothing_of_the_rows_it_holds` | No | +| A cross-reference stream whose `/Length` the chain cannot read: its data searched through a window `MaxTrailerLength` bounds, ended early by rows that spell `endstream` or an object header, or, with no `endstream`, read to the end of the file as rows; a guard's cut of that `/Length` charged to the file | Its data runs to its first `endstream`, searched for once in the bytes the parser holds and then in the file past them, up to the next object the sections read so far place or the end of the file, as any stream's whose length is not confirmed (*Stream data and its length*), but stopped by no object header its rows spell and keeping a carriage return before the end-of-line as data, which a row may end in. With no `endstream` before that stop, or none searched for once the document's searches have read as much of the file as they may, the section is malformed, the file's fault and no guard's: "its /Length is the reference N G R, which no section read before it places where it can be read, and no endstream follows its data before …" — but where the `/Length` leads to an object `MaxObjectLength` cut, through a chain of references too, the section is the guard's, as one whose `/W` a guard cut is (above): "its /Length is the reference 5 0 R, which one of the reader's limits cut" ([#224]). Rows that spell an `endstream` are told from the data's end by the row for the `/Length` that reading the section's rows keeps — a row of a reserved type is ignored, as the specification says —, when it lies among the rows taken, in a predicted row the spelled `endstream` parts too — the row the data decoded as the file's drops, leaves as decoded, the first parted, or still differenced under the TIFF predictor —, the data decoded once more knowing its end hidden, that row undone as far as it goes and the reports dropped, but for a guard on decoding that decoding reached, which the whole data reaches too: short of the row, the rows past the bound are then the guard's, as where the decoding before reached it, and `ThrowOnLimit` throws as the report stands: the object it places is read there as the chain reads one, nothing kept, corrected or reported, and where an `endstream` confirms the length it gives, the data is taken again to it and decoded once more, what the first decoding reported dropped. Where `MaxObjectLength` cuts the object that row places — or the object a chain of references from it leads to, or the one a newer section places for the `/Length`, or one a chain from it leads to —, the integer read before the cut is only a place to look for `endstream` at, read once with the cut however many sections ask — a whole reference it holds is followed on, as read whole, to a length the file gave where it reaches an object read whole, rows the data taken again lacks then the file's —: the data is taken again where one confirms it — rows it still lacks then the guard's, the integer no length the file gave —, and otherwise the rows past the `endstream` the rows spell are the guard's, as rows past a decoding guard are — the section counts as cut, the chain ends there, nothing is charged to the file, and the length is not checked —; but where what follows the `endstream` the data ends at proves it the stream's own (*Stream data and its length*), the rows the data lacks are the file's, as read whole, and the index is rebuilt ([#224]). Otherwise, a row that does not stand — a newer section gave the number, or refused it; the row places it in an object stream, under another generation, or holds no length an `endstream` confirms — leaves the data as taken, and the length, checked once the chain is read, is reported where an `endstream` confirms it past the rows the chain read; data a guard cut that lacks rows — `MaxStreamLength` as it is read, or a guard on decoding it, `MaxDecodedStreamLength` or `MaxFilterChainLength`, which keeps it from being taken again — is the guard's, not checked against it; data that holds every row the section declares is checked whatever a guard did, but not against a length a guard cut. Wherever the guard hides where the data ends — a guard on decoding kept it, which lacks rows, from being taken again, or a guard's cut of the `/Length` gave its end —, it is decoded once more knowing that, as data a guard cut — a group ASCII85 or ASCIIHex holds in part, which the hidden end may have parted, dropped and counted against `MaxDecodedStreamLength` all the same, a predictor's last row undone as far as it goes, so that no row is read from bytes a guess completed —, a decoding guard the first decoding reached standing, with its report, though a filter after the one that drops a group, given less, may stay within the bound — the group counted against what the filters before the last share as well, and no more rows held than the first decoding read —: a filter run out at its end, over data a spelled `endstream` cut short, says nothing of the file, while a fault in what it read stands, as for the stream's direct twin, and so does a report the capacity counted ([#220]); data that holds every row the section declares is not decoded again, its end the file's | `GuardCutValueTests.A_decoding_guard_deferred_data_reached_stands_when_the_data_is_decoded_again_knowing_its_end_hidden`, `A_decoding_guard_deferred_data_reached_is_reported_once_whatever_room_the_diagnostics_have`, `A_group_an_unproven_endstream_parts_where_a_guard_cut_the_length_leaves_its_rows_to_the_guard`, `A_predicted_row_an_unproven_endstream_parts_where_a_guard_cut_the_length_is_undone_as_far_as_it_goes`; `DocumentReaderTests.A_cross_reference_stream_whose_length_only_it_indexes_is_read_past_the_first_window_as_its_direct_twin`, `A_deferred_length_s_data_MaxStreamLength_cuts_past_every_row_is_checked_against_its_length`, `A_deferred_length_s_data_is_read_once_more_than_its_direct_twin_s`, `Rows_that_spell_an_object_header_do_not_end_a_deferred_length_s_data`, `A_deferred_length_s_data_with_no_endstream_before_the_end_of_the_file_makes_its_section_malformed`, `A_deferred_length_s_data_with_no_endstream_before_the_next_object_makes_its_section_malformed`, `Rows_that_spell_endstream_after_the_length_s_own_row_are_read_as_the_length_gives`, `Rows_that_spell_endstream_in_an_update_leave_none_of_its_objects_to_the_revision_it_supersedes`, `Rows_that_spell_endstream_in_an_older_section_are_read_as_its_length_gives`, `A_length_row_that_does_not_stand_leaves_the_data_up_to_the_endstream_the_rows_spell`, `Updates_each_with_a_long_deferred_length_are_all_searched`, `A_deferred_length_is_taken_again_by_the_row_the_section_keeps_for_it`, `A_deferred_length_is_not_checked_against_data_a_guard_cut`, `A_filter_guard_that_keeps_a_deferred_length_s_data_from_being_taken_again_leaves_no_filter_fault`, `A_guard_s_report_the_capacity_drops_beside_a_deferred_length_s_decoding_is_counted`, `A_fault_a_deferred_length_s_data_holds_before_a_guard_stops_it_is_the_file_s_as_for_a_direct_one`, `A_deferred_length_s_data_whose_last_digit_alone_crosses_MaxDecodedStreamLength_is_cut_as_its_direct_twin_s`, `Data_whose_end_a_guard_hides_leaves_the_filters_after_a_dropped_group_the_room_its_direct_twin_s_have`, `A_deferred_length_s_data_whose_rows_a_guard_on_decoding_holds_all_of_reports_its_filters_ends`; `GuardCutValueTests.A_deferred_length_a_guard_cut_leaves_a_stream_with_no_endstream_the_guard_s`, `The_integer_a_guard_cut_a_deferred_length_after_is_where_rows_that_spell_an_endstream_are_taken_again_to`, `Rows_past_an_endstream_they_spell_are_the_guard_s_when_it_cut_the_deferred_length_before_its_value`, `Rows_past_a_spelled_endstream_are_the_guard_s_when_the_length_leads_to_an_object_a_newer_section_places_and_it_cut`, `The_integer_a_guard_cut_a_length_a_newer_section_places_after_is_where_spelled_rows_are_taken_again_to`, `Rows_data_taken_again_to_an_integer_a_guard_parted_still_lacks_are_the_guard_s`, `Rows_data_taken_again_to_an_integer_a_guard_parted_in_the_own_row_s_object_still_lacks_are_the_guard_s`, `A_whole_reference_a_guard_cut_after_leads_the_own_row_s_length_on_as_a_whole_read_follows_it`, `A_whole_reference_a_guard_cut_after_leads_a_chain_on_whichever_section_reads_it_first`, `A_filter_run_out_where_a_guard_s_cut_of_the_length_leaves_the_data_says_nothing_of_the_file`; `ReaderLimitsTests.MaxTrailerLength_bounds_a_cross_reference_stream_s_dictionary_and_not_its_data`; `HostileInputTests.A_deferred_length_s_data_with_no_endstream_makes_its_section_malformed_reading_the_file_a_few_times`, `Sections_whose_lengths_name_one_object_a_guard_cut_read_it_once`; `DocumentReaderTests.A_deferred_length_a_newer_section_refused_takes_no_data_again`, `A_deferred_length_is_taken_again_by_a_row_with_no_type_field`; `GuardCutValueTests.A_cross_reference_stream_with_no_endstream_whose_length_s_chain_ends_at_its_last_object_a_guard_cut_is_the_guard_s`, `Rows_past_a_spelled_endstream_are_the_file_s_when_the_length_s_chain_gives_a_wrong_length`, `Rows_past_a_spelled_endstream_are_the_guard_s_when_the_length_s_chain_ends_at_its_last_object_a_guard_cut`, `A_cross_reference_stream_s_length_a_guard_cut_leaves_rows_its_proven_endstream_ends_to_the_file`, `A_cross_reference_stream_the_chain_read_proves_the_endstream_of_the_older_one_before_it`, `GuardCutValueTests.Rows_a_predicted_row_undone_as_far_as_it_goes_completes_are_read_though_no_guard_stopped_the_decoding_before`; `DocumentReaderTests.Reads_a_cross_reference_stream_whose_length_row_a_spelled_endstream_parts_the_predicted_row_of`, `A_decoding_guard_a_deferred_length_s_data_reaches_once_its_predicted_row_is_undone_is_reached_as_read_whole`, `Reads_a_cross_reference_stream_whose_rows_a_length_it_places_runs_past_spell_an_endstream_before_the_first` | No | | `/W` widths that overflow a field or give empty rows; `/Size` or `/Index` claiming rows the data lacks, or numbering objects no object can be; row fields an entry cannot hold | At least three widths, each an integer that is not negative, not all zero, narrowed only once it is known to be within the decoded data: a row longer than the data is one the data does not hold, and the widths' sum is then within a `long`. A field wider than 8 bytes is read when its leading bytes are zero, its every byte still read. `/Index` is an array of pairs of integers, each numbering its rows within 0..`int.MaxValue`, compared by subtraction, with a count that is not negative, each read once; without `/Index`, `/Size` is a non-negative integer. A fault in this numbering makes the section malformed before any row is read. Rows are read only while they fit in the decoded data, and the shortfall is computed by division, where a `/Size` as large as a `long` cannot overflow. Each field is read as an unsigned 64-bit value, one wider than 8 bytes whose leading bytes are not all zero as the largest; a row is refused alone when no entry can hold it — an offset of 2^63 or more, a generation past 65,535, an object stream numbered outside 1..`int.MaxValue`, an index past `int.MaxValue`, no older section's row then standing for its object — and a type past 2 is ignored, as ISO 32000-1 asks | `CrossReferenceRuleTests.A_cross_reference_stream_whose_widths_or_ranges_cannot_be_read_is_malformed`, `A_cross_reference_stream_whose_size_declares_more_rows_than_it_holds_is_malformed`, `A_cross_reference_stream_row_whose_fields_no_entry_can_hold_is_refused_alone`, `A_cross_reference_stream_row_of_a_type_past_2_is_ignored_however_wide_its_type_field`, `A_cross_reference_stream_whose_fields_are_wider_than_8_bytes_is_read_when_their_leading_bytes_are_zero`, `A_cross_reference_stream_row_whose_wide_field_holds_more_than_64_bits_is_refused_alone`, `A_cross_reference_stream_whose_rows_are_wider_than_its_data_holds_fewer_rows_than_it_declares`; `HostileInputTests.Survives_a_cross_reference_stream_with_impossible_field_widths`; `CrossReferenceChainTests.Looks_for_an_object_a_cross_reference_stream_holds_no_row_for` | Seldom: the rows are compressed | | Entries without end — rows of one byte decoding to 256 MB from a few hundred bytes, refused rows, a rebuild's headers, the members of the object streams a rebuild expands | `MaxIndexEntries` counts the distinct object numbers the index holds, in use, free or refused: a row or a header for a number held already takes no room, so that millions of copies of one header hide nothing behind them, and a rebuild reads the file to its end. The first number with no room is reported once, where it was met; the chain stops at the section that filled the index, and a stream's rows past it are not read ([#164], [#183]) | `HostileInputTests.A_cross_reference_stream_of_one_byte_rows_decoding_to_the_bound_indexes_no_more_entries_than_the_guard_allows`, `Rows_a_section_refuses_count_toward_the_entries_the_index_may_hold`, `A_rebuild_finds_a_document_behind_millions_of_repeated_headers_of_one_object`; `CrossReferenceChainTests.The_entry_guard_counts_the_entries_of_every_section_and_stops_the_chain_at_the_one_that_fills_it`; `ReaderLimitsTests.A_rebuild_indexes_object_numbers_up_to_the_entry_guard_and_reports_the_first_it_had_no_room_for`, `A_rebuild_past_the_entry_guard_still_takes_the_newest_definition_of_a_number_it_holds`, `A_rebuild_takes_in_the_members_of_object_streams_no_further_than_the_entry_guard`, `A_rebuild_that_reaches_the_entry_guard_finishes_the_index_before_it_throws`, `A_row_an_older_section_refuses_for_a_number_a_newer_one_gave_takes_no_room` | Yes | -| A declared count the index makes room for before its rows are read | A section makes room for its rows before they are read, so that the map is sized once: no more rows than a stream's data holds, or a classic table's bytes hold — a line each, of 19 bytes at least — up to the first byte no row holds, read in the window that holds that byte or more rows than the index has room for, and made once however often the rows are read; no more numbers than `MaxIndexEntries` lets the index take, nor than the newest section's `/Size` numbers, or the highest number the sections read so far give when it is stale, an older section's superseded rows adding none. A `/Size` alone makes no room. Room is never added by less than the map holds, so that a chain of many sections does not resize it at each one; once a doubling comes within a doubling of the newest `/Size`, room is made for that size, no more than four times what the map held. Rows that index nothing — a reserved type, a refused row, a number held already — still get room, at most 52 bytes a number the guard lets the index take, once a document | `HostileInputTests.A_declared_size_far_past_the_rows_a_file_holds_makes_room_for_no_more_than_they_hold`, `Rows_that_index_nothing_make_room_for_no_more_entries_than_the_guard_allows`, `Blank_lines_among_a_table_s_rows_make_room_for_no_more_rows_than_their_bytes_hold`; `ReaderLimitsTests.A_table_its_length_guard_cuts_makes_room_for_no_more_rows_than_the_guard_lets_the_reader_read`, `A_newest_size_makes_room_for_no_object_the_sections_hold_no_row_for`; `XRefTableTests.Room_for_a_few_rows_more_than_the_table_holds_doubles_it_rather_than_growing_it_by_them`, `Room_is_made_for_no_more_entries_than_the_most_asked`; `DocumentReaderTests.Opening_a_chain_of_a_thousand_sections_resizes_the_index_no_more_often_than_doubling_it`, `An_update_s_rows_and_the_original_s_make_room_for_the_objects_the_newest_size_numbers`, `A_table_of_many_subsections_makes_room_for_its_rows_once`, `A_subsection_claiming_millions_of_rows_makes_room_for_the_rows_its_bytes_hold`, `A_table_read_again_in_a_larger_window_makes_room_for_its_rows_once`, `Rows_written_a_byte_short_make_room_for_each_of_them`, `A_newest_size_an_update_did_not_raise_makes_room_for_the_objects_its_rows_add`, `A_lowered_entry_guard_reads_a_table_in_no_larger_window_than_its_room_needs` | Yes | -| A rebuild run again and again | At most one per document, and none once a validation's first rule has begun: before it, the validator has it run when a later read still could, so that no read of a rule's rebuilds the index halfway through ([#237]). A missing object rebuilds only when the index is known to be incomplete through damage: what a guard left out of it — a section past `MaxXRefSectionCount`, rows past `MaxXRefSectionLength` or `MaxDecodedStreamLength`, numbers past `MaxIndexEntries` — reads as null, unknown to the validator rather than missing, and each later lookup of it throws under `ThrowOnLimit`. A rebuild damage forces on a chain a guard cut keeps the chain's entries the scan does not find. The chain ends wherever the index fills, and wherever a stream's rows are cut, a hybrid file's stream included, so that no older section stands for what the cut rows define; a cut is known by the count of guards reached, which full diagnostics do not drop, not by its report. A catalog or an object stream a guard hid, and the objects such a stream holds, are no reason to rebuild; the validator takes an object stream the chain's index lacks as unknown once a guard stopped the chain, though a rebuild found the stream since | `DocumentReaderTests.A_reference_the_chain_cannot_read_rebuilds_nothing_while_the_chain_is_read`; `CrossReferenceChainTests.Reads_a_reference_to_an_object_the_file_does_not_define_as_null_without_rebuilding`; `ReaderLimitsTests.A_lookup_past_a_full_index_reads_null_and_rebuilds_nothing`, `A_catalog_past_a_guard_cut_reads_as_null_and_rebuilds_nothing`, `A_rebuild_damage_sets_off_keeps_the_entries_a_chain_a_guard_cut_gave_that_the_scan_cannot_find`; `ReadOrderTests.An_entry_whose_object_a_read_would_not_find_has_the_index_rebuilt_before_the_first_rule`; `CorpusValidationTests.No_read_rebuilds_the_index_after_a_validation`; `CrossReferenceRuleTests.An_object_a_guard_left_out_of_the_index_is_no_reference_the_file_lacks`, `An_index_the_entry_bound_filled_is_said_once_to_be_checked_in_part`; `CrossReferenceChainTests.A_full_index_ends_the_chain_at_the_stream_of_a_hybrid_section`, `An_older_section_does_not_stand_for_what_a_cross_reference_stream_s_cut_rows_define`, `A_rebuild_reports_a_full_index_once_though_an_object_stream_lies_before_the_header`; `ReaderLimitsTests.A_catalog_packed_in_an_object_stream_past_a_full_index_reads_as_null_and_rebuilds_nothing`, `A_row_refused_before_a_full_index_is_the_file_s_and_an_object_stream_past_it_stays_unknown`; `DiagnosticsTests.A_guard_reached_is_counted_though_the_capacity_drops_its_report_and_follows_it_when_moved_or_rolled_back` | Yes | +| A declared count the index makes room for before its rows are read | A section makes room for its rows before they are read, so that the map is sized once: no more rows than a stream's data holds, or a classic table's bytes hold — a line each, of 19 bytes at least — up to the first byte no row holds, read in the window that holds that byte or more rows than the index has room for, and made once however often the rows are read; no more numbers than `MaxIndexEntries` lets the index take, nor than the newest section's `/Size` numbers, written direct — an integer, or a real with no fractional part; one written as a reference is not read through for it, and bounds nothing —, or the highest number the sections read so far give when it is stale, an older section's superseded rows adding none. A `/Size` alone makes no room. Room is never added by less than the map holds, so that a chain of many sections does not resize it at each one; once a doubling comes within a doubling of the newest `/Size`, room is made for that size, no more than four times what the map held. Rows that index nothing — a reserved type, a refused row, a number held already — still get room, at most 52 bytes a number the guard lets the index take, once a document | `HostileInputTests.A_declared_size_far_past_the_rows_a_file_holds_makes_room_for_no_more_than_they_hold`, `Rows_that_index_nothing_make_room_for_no_more_entries_than_the_guard_allows`, `Blank_lines_among_a_table_s_rows_make_room_for_no_more_rows_than_their_bytes_hold`; `ReaderLimitsTests.A_table_its_length_guard_cuts_makes_room_for_no_more_rows_than_the_guard_lets_the_reader_read`, `A_newest_size_makes_room_for_no_object_the_sections_hold_no_row_for`; `XRefTableTests.Room_for_a_few_rows_more_than_the_table_holds_doubles_it_rather_than_growing_it_by_them`, `Room_is_made_for_no_more_entries_than_the_most_asked`; `DocumentReaderTests.Opening_a_chain_of_a_thousand_sections_resizes_the_index_no_more_often_than_doubling_it`, `An_update_s_rows_and_the_original_s_make_room_for_the_objects_the_newest_size_numbers`, `A_table_of_many_subsections_makes_room_for_its_rows_once`, `A_subsection_claiming_millions_of_rows_makes_room_for_the_rows_its_bytes_hold`, `A_table_read_again_in_a_larger_window_makes_room_for_its_rows_once`, `Rows_written_a_byte_short_make_room_for_each_of_them`, `A_newest_size_an_update_did_not_raise_makes_room_for_the_objects_its_rows_add`, `A_lowered_entry_guard_reads_a_table_in_no_larger_window_than_its_room_needs` | Yes | +| A rebuild run again and again | At most one per document, and none once a validation's first rule has begun: before it, the validator has it run when a later read still could, so that no read of a rule's rebuilds the index halfway through ([#237]). A missing object rebuilds only when the index is known to be incomplete through damage: what a guard left out of it — a section past `MaxXRefSectionCount`, rows past `MaxXRefSectionLength`, `MaxStreamLength` or `MaxDecodedStreamLength`, a section whose rows need a value `MaxObjectLength` cut ([#263]), numbers past `MaxIndexEntries` — reads as null, unknown to the validator rather than missing, and each later lookup of it throws under `ThrowOnLimit`. A rebuild damage forces on a chain a guard cut keeps the chain's entries the scan does not find. The chain ends wherever the index fills, and wherever a stream's rows are cut, a hybrid file's stream included, so that no older section stands for what the cut rows define; a cut is known by the count of guards reached, which full diagnostics do not drop, not by its report. A catalog or an object stream a guard hid, and the objects such a stream holds, are no reason to rebuild; the validator takes an object stream the chain's index lacks as unknown once a guard stopped the chain, though a rebuild found the stream since | `DocumentReaderTests.A_reference_the_chain_cannot_read_rebuilds_nothing_while_the_chain_is_read`; `CrossReferenceChainTests.Reads_a_reference_to_an_object_the_file_does_not_define_as_null_without_rebuilding`; `ReaderLimitsTests.A_lookup_past_a_full_index_reads_null_and_rebuilds_nothing`, `A_catalog_past_a_guard_cut_reads_as_null_and_rebuilds_nothing`, `A_rebuild_damage_sets_off_keeps_the_entries_a_chain_a_guard_cut_gave_that_the_scan_cannot_find`; `ReadOrderTests.An_entry_whose_object_a_read_would_not_find_has_the_index_rebuilt_before_the_first_rule`; `CorpusValidationTests.No_read_rebuilds_the_index_after_a_validation`; `CrossReferenceRuleTests.An_object_a_guard_left_out_of_the_index_is_no_reference_the_file_lacks`, `An_index_the_entry_bound_filled_is_said_once_to_be_checked_in_part`; `CrossReferenceChainTests.A_full_index_ends_the_chain_at_the_stream_of_a_hybrid_section`, `An_older_section_does_not_stand_for_what_a_cross_reference_stream_s_cut_rows_define`, `A_rebuild_reports_a_full_index_once_though_an_object_stream_lies_before_the_header`; `ReaderLimitsTests.A_catalog_packed_in_an_object_stream_past_a_full_index_reads_as_null_and_rebuilds_nothing`, `A_row_refused_before_a_full_index_is_the_file_s_and_an_object_stream_past_it_stays_unknown`; `DiagnosticsTests.A_guard_reached_is_counted_though_the_capacity_drops_its_report_and_follows_it_when_moved_or_rolled_back` | Yes | | The rebuild's scan | 1 MB pieces overlapping by 64 bytes, each piece after the first searched from where the one before could no longer hold a whole `obj`, so that a header is read once. A header is read backwards, each run of digits judged by its value however many zeros lead it, as the parser judges it: a number in 1..`int.MaxValue` and a generation in 0..65,535, which the entry records; the search near an entry's offset, a load's, settling's and the validator's probe's, reads a header so too. The definitions it finds again of a number are counted, with the first ten numbers, in a fixed-size record, and reported once as `object.redefined` | `DocumentReaderTests.Rebuilds_an_index_whose_objects_lie_past_the_first_megabyte`, `A_rebuild_ignores_object_headers_numbered_zero_or_beyond_what_an_object_number_holds`, `A_rebuild_finds_a_header_whose_number_and_generation_zeros_lead`, `A_rebuilt_entry_records_the_generation_its_header_gives`, `Counts_two_definitions_the_scan_finds_in_the_overlap_of_two_windows_as_one_redefinition`, `Reads_a_header_that_straddles_the_start_of_a_scan_window_under_its_own_number`; `HostileInputTests.Survives_a_file_made_only_of_object_headers`; `CrossReferenceRuleTests.An_entry_at_a_header_the_parser_refuses_is_broken` | Partly | | A catalog a guard cut — its `/Type` or its `/Pages` past the cut —, which leads to no catalog as read | The object `/Root` names, through a chain of references too, is taken for the catalog as read — one an object stream's cut leaves unread, past the cut data or in a stream the guard made unreadable, as one it cut ([#259]), and one written in a trailer `MaxTrailerLength` cut —, at opening and after any rebuild: no search among the indexed objects, which could take a catalog an update superseded, and no rebuild, which would read it cut again; `FileStructure.RootCutByGuard` records it, and `file.root-invalid` does not judge it ([#175]). A rebuild after a search reads the file's own `/Root` again, not the search's guess, and a guard it reaches there is held, as any in a rebuild. A rebuild keeps in its index the members the chain's index places in an object stream a guard hides — the scan finds no header of theirs —, which stay unread for the guard's reasons and throw again when opened to throw; a stream refused for a fault of its dictionary, read whole, hides none, whatever a limit cut of its data | `ReaderLimitsTests.A_catalog_a_guard_cut_is_taken_as_read_and_no_other_is_looked_for_even_by_a_later_rebuild`, `A_catalog_an_object_stream_s_cut_leaves_unread_stays_taken_as_read_through_a_rebuild`, `A_catalog_written_in_a_trailer_a_guard_cut_is_taken_as_read`, `A_catalog_written_in_a_trailer_a_guard_cut_stays_taken_as_read_through_a_rebuild`, `A_member_an_object_stream_s_cut_leaves_unread_throws_again_after_a_rebuild`, `An_object_stream_refused_for_its_own_fault_hides_no_member_where_a_limit_also_cut_its_data`, `A_rebuild_after_a_search_reads_the_file_s_own_root_again`, `A_rebuild_reaching_a_guard_on_a_catalog_a_search_took_still_looks_for_another_when_opened_to_throw`, `A_root_that_leads_through_a_chain_to_a_catalog_a_guard_cut_is_taken_as_read`, `A_catalog_an_object_stream_s_cut_leaves_unread_is_taken_as_read`; `FileRuleTests.A_catalog_a_limit_cut_inside_its_type_is_not_judged`, `A_catalog_a_limit_cut_before_its_type_is_taken_as_read_rather_than_an_older_one`, `A_catalog_that_is_none_is_still_judged_beside_one_a_limit_cut_elsewhere` | No | | A guard reached in the middle of a rebuild | The guard's exception is held, and thrown once the index is whole | `ReaderLimitsTests.A_rebuild_that_reaches_a_guard_finishes_the_index_before_it_throws`, `A_rebuild_that_reaches_a_guard_looking_for_the_catalog_still_finds_it`; `ReadOrderTests.A_guard_the_rebuild_before_the_first_rule_reaches_throws_in_the_first_validation_only` | No | @@ -289,12 +302,12 @@ Known gaps: bucket, and every lookup walks it. - [#171] A rebuild takes headers cut at a chunk's edge, or lying in stream data or strings, and serves another object under their number. -- [#220] A cross-reference stream whose `/Length` only it indexes is found by an `endstream` search through a window - `MaxTrailerLength` bounds, a guard named for a dictionary; rows that spell `endstream` end its data early. - [#221] A cross-reference stream row of a reserved type is skipped, letting an older section's row stand where ISO 32000-1 reads the object as null. - [#156] An `/XRefStm` naming a section already read, or a classic table, leaves no record and no finding. -- [#47] Each section is read through a window of up to 64 KB, whatever its size. +- [#47] Each section is read through a window of up to 64 KB, whatever its size; and the probe that looks for a + classic table's `xref` keyword past white space an offset falls short by reads again from the offset as it grows, + eight times larger each time up to `MaxTrailerLength`: about nine times the run it crosses, once per section. - [#49] A rebuild reads a 64 KB window at every `trailer` keyword. - [#188] Relocating a section the chain cannot read can land on one it already read: a loop and a shift the file does not have are reported, or a hybrid table taken for its own `/XRefStm` loses what only the stream indexes. @@ -311,15 +324,16 @@ Known gaps: | Threat | Defense | Held by | Fuzzed | |---|---|---|---| | An object whose load needs itself, or loads chained without end | `_loading` turns a load of an object being loaded into null. `PdfFileReader.MaxNestedLoads` (64) and a stack probe bound the chain, reported once as `syntax.depth-exceeded` | `HostileInputTests.Survives_an_object_whose_length_refers_to_itself`, `Survives_a_chain_of_lengths_longer_than_the_stack_is_deep`, `Reports_chains_that_run_too_deep_once` | The load path, not the bound | -| Decodes nested through filter parameters, each holding its output while the next decodes | A stream resolves its filters, its parameters and the values they hold before it reads any of its data, and holds them as plain values: only what a step reads — LZW's `/EarlyChange`, a Flate or LZW step's predictor values —, nothing past an image filter, an unknown one or an element that names none. A chain of loads holds what `ObjectStreamBudget` keeps, the object stream decoded last, and one decoding, whatever its depth ([#165]), besides the raw data of each object stream on it, which the object cache keeps until it lets the stream go ([#170]) | `FilterTests.Resolves_every_value_a_step_reads_before_any_step_reads_the_data`, `Resolves_nothing_a_step_does_not_read`, `Reads_a_stream_s_data_from_the_file_once_every_value_its_filters_read_is_loaded`; `HostileInputMemoryTests.A_chain_of_object_streams_each_needing_the_next_to_decode_holds_one_level_s_output_at_once` (64 levels of 4 MB, held to 32 MB plus three bounds, live at once) | No | +| Decodes nested through filter parameters, each holding its output while the next decodes | A stream resolves its filters, its parameters and the values they hold before it reads any of its data, and holds them as plain values: only what a step reads — LZW's `/EarlyChange`, a Flate or LZW step's predictor values —, nothing past an image filter, an unknown one or an element that names none. A chain of loads holds what `ObjectStreamBudget` keeps, the object stream decoded last, and one decoding, whatever its depth ([#165]); no object stream on it keeps the raw data it read ([#170]) | `FilterTests.Resolves_every_value_a_step_reads_before_any_step_reads_the_data`, `Resolves_nothing_a_step_does_not_read`, `Reads_a_stream_s_data_from_the_file_once_every_value_its_filters_read_is_loaded`; `HostileInputMemoryTests.A_chain_of_object_streams_each_needing_the_next_to_decode_holds_one_level_s_output_at_once` (64 levels of 4 MB, held to 32 MB plus three bounds, live at once) | No | | References to references that loop | `PdfReference.Resolve` gives up after 32 fetches and reads null, with no report ([#173]) | `ObjectModelTests.A_chain_of_references_that_loops_resolves_to_null` | No | | An object stream that needs an object it holds, or two that need each other | While an object stream is decoded, an object it holds reads as null and is reported once as `stream.self-reference`, at the stream's entry, or nowhere when the entry lies outside the file; the null is remembered, so a later decode reads the same | `HostileInputTests.An_object_stream_that_needs_an_object_it_holds_reads_the_same_after_the_budget_lets_it_go`, `Two_object_streams_that_need_each_other_read_the_same_after_the_budget_lets_them_go`; `HeaderOffsetTests.An_object_stream_that_needs_itself_found_near_a_row_past_the_end_is_reported_nowhere` | Object-stream seeds, not the cycle | | An offset table that lies | `/First` is checked against the decoded data, and the stream against its `/Type`, which must be `/ObjStm` ([#274]). The header is read as far as it goes, whatever `/N` claims, up to its first pair that is not two integers, that gives a number outside 1..`int.MaxValue` or an offset outside 0..`int.MaxValue`, whose offset is not past the one before it, or whose member would start at or past the end of the decoded data unless a guard cut it ([#259]) — the reader, the rebuild and the validator alike ([#202]) —: the pairs before it are kept, the fault is reported once as `object-stream.unreadable`, and the validator counts the objects the index places in the stream that the header does not list before it, and lost — none, and no finding, when it lists them all ([#275]). The header's tables grow as its pairs are read, never past the quarter of `/First`'s bytes a header holds at most, so that a `/N` no header holds allocates nothing ([#160]). A wrong index falls back on a lookup by number, sorted and built once (`ObjectStreamPlaces`), reported once for each object and definition of its stream as `object-stream.member-moved`: the object is read at the first place the header lists it. The validator's probe finds each entry's object through the same lookup, and the members written as a stream by a search, so that judging a stream costs what its members do | `ReaderTimingTests.An_object_stream_whose_every_index_is_wrong_is_read_in_linear_time`; `DocumentReaderTests.Reads_an_object_where_its_stream_s_header_lists_it_rather_than_where_the_index_says`, `Finds_an_object_its_entry_misplaces_at_the_first_place_its_object_stream_lists_it`, `Reports_an_object_stream_whose_dictionary_or_header_cannot_be_believed_where_its_data_starts`, `Reads_no_member_an_object_stream_header_numbers_as_no_object_can_be`, `Reads_no_member_an_object_stream_header_places_where_no_member_can_start`, `Ends_an_object_stream_header_at_a_member_that_starts_before_the_one_before_it_ends_or_past_the_data`, `Reads_none_of_an_object_stream_whose_first_member_starts_past_its_data`, `Reads_none_of_the_members_of_a_stream_that_is_not_of_type_object_stream`, `Reads_an_object_stream_header_that_lists_the_largest_number_a_member_can_have`, `Indexes_no_member_a_rebuild_finds_listed_under_a_number_no_object_can_have`, `Reports_an_object_at_another_index_of_its_stream_once_however_often_it_is_parsed`; `HostileInputTests.An_object_stream_that_claims_more_objects_than_it_could_hold_serves_those_it_lists_and_allocates_nothing_for_the_rest`, `An_object_stream_whose_N_its_blank_header_cannot_hold_allocates_no_table_for_it`, `An_object_stream_that_lists_one_offset_over_and_over_keeps_one_member`; `CrossReferenceRuleTests.An_object_stream_whose_header_lists_something_other_than_objects_cannot_be_read`, `An_object_stream_whose_header_breaks_after_an_object_says_which_objects_are_lost`, `An_object_stream_whose_first_member_starts_past_its_data_cannot_be_read`, `An_object_stream_that_cannot_serve_its_objects_is_reported_once`, `An_object_stream_that_lists_every_object_its_index_places_before_its_header_breaks_loses_none_and_is_no_finding`, `An_object_stream_broken_after_an_object_the_index_misplaces_shifts_that_object_and_loses_the_other`; `HostileInputTests.An_object_stream_header_s_tables_start_no_larger_than_its_bytes_can_hold`, `An_object_stream_header_s_tables_grow_no_larger_than_its_bytes_can_hold`; `ReadOrderTests.A_member_the_definition_a_rebuild_serves_lists_elsewhere_is_reported_moved_whatever_was_read_before`; `ReaderTimingTests.The_validator_judges_an_object_stream_s_members_in_linear_time` | Yes | -| A member that runs on — into the next member, or for megabytes | A member is read to the next member's start, the last to the end of the data — the offsets increase, so the members' reads together cover the data once ([#200]) —, and no further than `MaxObjectLength`, as no object of the file is ([#160]). A member left open ends there, reported as `syntax.truncated-object`, though a guard cut the data exactly at the next member's start; one the guard cut is read as far as the bound — a dictionary the bound leaves no room to say whether `stream` follows included, the bound ending after it or on a keyword that begins `stream`, or is `stream` and may be longer; a `]`, a `>>` or another keyword it ends on is whole — and reported once for each member of each definition of its stream as `limit.object`, at the stream's data, and a document opened to throw throws each time it is read | `SyntaxFaultTests.A_member_left_open_ends_at_the_next_member_s_start`; `ReaderLimitsTests.A_member_longer_than_MaxObjectLength_is_read_to_the_bound_and_each_is_reported_once`, `A_member_longer_than_MaxObjectLength_throws_each_time_it_is_read_when_the_document_throws_on_a_limit`, `A_member_within_MaxObjectLength_is_read_whole_and_reported_nowhere`; `HostileInputTests.A_member_of_millions_of_arrays_is_read_no_further_than_MaxObjectLength`, `An_object_stream_of_members_one_byte_apart_reads_each_to_the_next_one_s_start_in_time`; `ReaderTimingTests.The_members_of_an_object_stream_one_byte_apart_are_read_in_linear_time`; `ReaderLimitsTests.A_member_dictionary_whose_stream_keyword_MaxObjectLength_leaves_out_is_cut_there_and_reported`, `A_member_whose_value_ends_where_MaxObjectLength_ends_its_window_is_read_whole`, `A_member_dictionary_followed_by_a_keyword_that_cannot_be_stream_where_MaxObjectLength_ends_its_window_is_whole`, `A_member_whose_array_ends_where_a_guard_cuts_the_data_is_read_whole`, `A_member_left_open_where_a_guard_cuts_the_data_at_the_next_member_s_start_is_the_file_s_fault` | Yes | +| A member that runs on — into the next member, or for megabytes | A member is read to the next member's start, the last to the end of the data — the offsets increase, so the members' reads together cover the data once ([#200]) —, and no further than `MaxObjectLength`, as no object of the file is ([#160]). A member left open ends there, reported as `syntax.truncated-object`, though a guard cut the data exactly at the next member's start; one the guard cut is read as far as the bound — a dictionary the bound leaves no room to say whether `stream` follows included, the bound ending after it or on a keyword that begins `stream`, or is `stream` and may be longer; a `]`, a `>>` or another keyword it ends on is whole, an `R` too where the decoded +byte past the window does not go on it, `6 0 R` read as the reference and `6 0 RG` cut ([#224]) — and reported once for each member of each definition of its stream as `limit.object`, at the stream's data, and a document opened to throw throws each time it is read | `SyntaxFaultTests.A_member_left_open_ends_at_the_next_member_s_start`; `ReaderLimitsTests.A_member_longer_than_MaxObjectLength_is_read_to_the_bound_and_each_is_reported_once`, `A_member_longer_than_MaxObjectLength_throws_each_time_it_is_read_when_the_document_throws_on_a_limit`, `A_member_within_MaxObjectLength_is_read_whole_and_reported_nowhere`; `HostileInputTests.A_member_of_millions_of_arrays_is_read_no_further_than_MaxObjectLength`, `An_object_stream_of_members_one_byte_apart_reads_each_to_the_next_one_s_start_in_time`; `ReaderTimingTests.The_members_of_an_object_stream_one_byte_apart_are_read_in_linear_time`; `ReaderLimitsTests.A_member_dictionary_whose_stream_keyword_MaxObjectLength_leaves_out_is_cut_there_and_reported`, `A_member_whose_value_ends_where_MaxObjectLength_ends_its_window_is_read_whole`, `A_member_whose_reference_ends_where_MaxObjectLength_ends_its_window_is_the_reference`, `A_member_whose_R_the_byte_past_MaxObjectLength_continues_is_cut`, `A_member_dictionary_followed_by_a_keyword_that_cannot_be_stream_where_MaxObjectLength_ends_its_window_is_whole`, `A_member_whose_array_ends_where_a_guard_cuts_the_data_is_read_whole`, `A_member_left_open_where_a_guard_cuts_the_data_at_the_next_member_s_start_is_the_file_s_fault` | Yes | | A stream written inside an object stream, which ISO 32000 forbids | The parser stops at its `stream` keyword: the object reads as null, nothing of it is decoded nor kept, and it is reported once for each definition of its stream and number as `object-stream.member-stream`; the validator reports the entry that places it there as `xref.entry-broken`, naming the index its header gives when the entry gives another, and the entry of the index a rebuild made where the file's own places nothing — or, where a guard stopped the chain, the part of it the reader read ([#161]) | `DocumentReaderTests.Reads_an_object_written_as_a_stream_inside_an_object_stream_as_null_and_reports_it_once`; `HostileInputTests.A_stream_inside_an_object_stream_is_not_decoded_whatever_it_would_decode_to`; `HostileInputMemoryTests.Object_streams_each_holding_a_stream_keep_no_more_than_their_budget_once_their_members_are_read`; `StreamLengthTests.A_stream_inside_an_object_stream_has_no_length_read_nor_reported_however_often_it_is_read`; `CrossReferenceRuleTests.An_object_written_as_a_stream_inside_an_object_stream_is_broken`, `An_object_whose_bytes_merely_hold_stream_inside_an_object_stream_is_sound`, `An_object_written_as_a_stream_that_its_entry_misplaces_is_broken_where_its_header_lists_it`, `An_object_written_as_a_stream_that_only_a_rebuilt_index_places_is_broken`, `An_object_written_as_a_stream_that_the_chain_and_a_rebuilt_index_both_place_is_its_entry_s_finding_alone`; `ReadOrderTests.A_member_fault_of_the_definition_a_rebuild_serves_is_reported_whatever_was_read_before` | Object-stream seeds, not a stream member | | Many large object streams held at once | `ObjectStreamBudget` (32 MB, with its reason), which weighs each stream's decoded data with its header's tables and lookup ([#160]): the oldest are let go and decoded again on demand, and nothing a member gave keeps its stream's data alive past them ([#161]) | `HostileInputTests.Object_streams_are_kept_decoded_within_a_budget_and_decoded_again_when_needed`, `The_object_stream_budget_weighs_the_header_tables_with_the_data`; `HostileInputMemoryTests.Disposing_a_document_lets_go_of_the_object_streams_it_decoded_though_a_reference_it_gave_is_held` | No | | A stream decoded again and again, each decoding reporting into its document | Its document's diagnostics hold what decoding a stream met once — a filter's fault, a guard reached on its data —: a decoding reports apart, and a report the document holds already, the whole of it alike, is not moved in, so that it takes no room there; two filters of one decoding that report alike are each held. An object stream decoded again after the budget let it go, by a rebuild, by the validator or by a caller adds nothing it holds already. The guard is still counted, so that what it cut is never taken for damage, and a document opened to throw throws each time, what the decoding met before moved in all the same ([#144]) | `HostileInputTests.A_fault_of_an_object_stream_is_held_once_by_its_document_however_often_the_stream_is_decoded`, `A_guard_an_object_stream_reaches_is_held_once_by_its_document_and_counted_each_time`, `A_document_holds_alike_reports_of_two_streams_and_of_two_filters_of_one_stream_each_once_a_decoding`, `What_a_decoding_met_before_a_guard_threw_is_held_once_though_each_decoding_throws`, `A_report_its_document_holds_takes_no_room_when_a_stream_is_decoded_again_into_full_diagnostics`, `A_report_the_capacity_leaves_out_of_one_decoding_is_counted_by_its_document`, `A_report_full_diagnostics_left_out_is_counted_again_each_time_its_stream_is_decoded`; `DiagnosticsTests.A_copy_of_a_report_the_capacity_left_out_is_counted_again_each_time_a_decoding_meets_it`, `A_decoding_moves_in_only_what_no_decoding_moved_in_before_two_alike_of_its_own_included`, `A_report_a_decoding_moved_in_before_takes_no_room_once_the_diagnostics_are_full_and_a_guard_stays_counted`; `CorpusReadingTests.A_document_holds_each_report_once_however_often_its_streams_are_decoded`, `A_damaged_object_stream_s_fault_is_held_once_after_a_full_read_and_a_validation` | No | -| Parsed objects piling up | The object cache is FIFO, bounded by count (`ObjectCacheCapacity`, 8,192), not by weight ([#37], [#170]), and keyed by number, so an object named under many generations is parsed once | `HostileInputTests.An_object_referenced_under_many_generations_is_read_once`; `StreamLengthTests.A_stream_whose_length_is_wrong_is_searched_and_reported_once_however_often_the_cache_lets_it_go` | No | +| Parsed objects piling up | The object cache is FIFO, bounded by count (`ObjectCacheCapacity`, 8,192), not by weight ([#37]) — a stream it holds keeps none of its data ([#170]) —, and keyed by number, so an object named under many generations is parsed once | `HostileInputTests.An_object_referenced_under_many_generations_is_read_once`; `StreamLengthTests.A_stream_whose_length_is_wrong_is_searched_and_reported_once_however_often_the_cache_lets_it_go` | No | | An object that is not where its entry says | The entry is checked against the file, before the header's offset is added, a header is looked for near it — within 512 bytes, though a window clamped at the file's start reaches further ([#197]); none for an offset more than 512 bytes before the file's start, nor past what a `long` holds —, and then the index is rebuilt once: three counted attempts. This answers a stack overflow the fuzzing found in M01. A moved entry records the generation of the header found, not that of the reference that asked, and that the reader found it | `DocumentReaderTests.Finds_objects_whose_recorded_offsets_are_wrong`, `Finds_an_object_whose_entry_points_past_the_end_of_the_file`, `An_object_a_rebuild_does_not_find_either_is_null`, `A_relocated_entry_records_the_generation_its_header_gives_whatever_reference_found_it`; `HeaderOffsetTests.An_entry_past_what_a_long_holds_is_not_looked_for_at_the_file_s_start_whatever_precedes_the_header`, `An_entry_far_before_the_file_s_start_is_not_looked_for_at_it`, `An_object_found_near_its_entry_before_the_header_is_placed_where_it_was_found` | Yes | | A reference to what the index lacks, forcing a rebuild each time | A rebuild runs only when the index is known to be incomplete through damage, and once; what a guard left out reads as null with none | `CrossReferenceChainTests.Reads_a_reference_to_an_object_the_file_does_not_define_as_null_without_rebuilding`; `ReaderLimitsTests.A_lookup_past_a_full_index_reads_null_and_rebuilds_nothing` | Yes | @@ -337,26 +351,49 @@ Known gaps: | Threat | Defense | Held by | Fuzzed | |---|---|---|---| -| A `/Length` that is negative, past `int.MaxValue`, not an integer or unresolvable | Only 0..`int.MaxValue` is taken as a length. Anything else sends the reader to the first `endstream` in the window | `StreamLengthTests.A_length_that_cannot_be_taken_is_reported_as_the_file_wrote_it` | Partly | +| A `/Length` that is negative, past `int.MaxValue`, not an integer or unresolvable | Only 0..`int.MaxValue` is taken as a length. Anything else sends the reader to the one search for the stream's first `endstream` (below) | `StreamLengthTests.A_length_that_cannot_be_taken_is_reported_as_the_file_wrote_it` | Partly | +| A `/Length` naming an object a guard cut or left out of the index, judged by what the cut left of it: null, reported as no length, or the `1` it parted of `15`, reported as a length the data contradicts; or the stream's missing `endstream` left unreported along with the length | Keyed on the object the reference leads to, through a chain of references too, never on the value read (`StreamLengthForm.ObjectCut`, `ObjectUnindexed`): an integer read before the cut is only a place to look for `endstream` at, never kept as a length, and the data is otherwise found by its `endstream`. Nothing is said of the length, whose fault is the guard's, reported where it was reached; a missing `endstream` is still the file's, and reported — but where the search stops at a header the data spells, or at an `endstream` nothing after it proves the stream's own (below) —, naming the object at the end of the chain of references, the one the guard cut or left out: "The stream's /Length names object N G, which one of the reader's limits cut, and no endstream follows its data …", or "…, which one of the reader's limits left out of the index, and no endstream follows its data …". A cross-reference stream's `/Length` so cut is left alike once the chain is read, and makes its section the guard's when no `endstream` ends its data, as *Indexing* says ([#224]) | `GuardCutValueTests.A_length_a_guard_cut_is_not_judged_and_the_data_is_found_by_its_endstream`, `A_length_a_guard_cut_leaves_a_missing_endstream_the_file_s_fault`, `A_length_an_object_stream_s_guard_left_unread_is_not_judged`, `A_length_whose_reference_leads_to_an_object_a_guard_left_out_of_the_index_is_not_judged`, `A_missing_endstream_names_the_object_a_guard_cut_at_the_end_of_the_length_s_chain_and_keeps_no_length`, `A_missing_endstream_is_the_file_s_fault_when_a_guard_left_the_length_out_of_the_index`, `A_deferred_length_a_guard_cut_is_not_judged_once_the_chain_is_read`; `GuardCutValueTests.A_length_whose_chain_ends_at_its_last_object_a_guard_cut_is_the_guard_s`, `A_missing_endstream_names_the_last_object_of_a_length_s_chain_a_guard_cut_or_left_out_of_the_index`, `A_missing_endstream_no_header_stops_the_search_at_is_the_file_s_where_a_guard_hid_the_length` | No | | An indirect `/Length` leading back to its stream, or chained | The loading set and `MaxNestedLoads`, as above | `HostileInputTests.Survives_an_object_whose_length_refers_to_itself`, `Survives_a_chain_of_lengths_longer_than_the_stack_is_deep` | No | -| A declared length past the file | It is checked against the file before the file is asked, and the length a stream is given is clamped to what the file holds | `HostileInputTests.Ignores_a_stream_length_larger_than_the_file`; `WindowEdgeTests.A_stream_whose_length_the_file_cannot_hold_is_read_to_its_endstream` | Yes | -| Confirming a length by reading past the buffer or the file | The look-ahead after the data is a fixed stack buffer of 13 bytes, or 64 past the window, refused outside the file and clamped to it | `WindowEdgeTests.A_stream_the_file_cuts_short_is_reported_without_asking_the_source_past_its_end`; `StreamLengthTests.A_stream_past_the_window_whose_length_endstream_follows_is_read_as_declared_for_a_few_bytes_more` | Property | -| A search for a lost `endstream` running into other objects | It stops at the next object the index places, the first object header the bytes hold, or the end of the file. When none of these comes first, the declared length is kept and reported | `StreamLengthTests.A_stream_past_the_window_with_no_endstream_before_the_next_object_keeps_its_declared_length`, `The_search_stops_at_the_next_object_of_an_index_rebuilt_as_the_document_opened` | Plausibly | -| Many searches over one stretch of the file | The searches past the window read at most four times the file in a document (`EndStreamSearchPasses`, with its reason). Past that, the declared length is kept unsearched | `StreamLengthTests.Streams_whose_data_share_a_stretch_read_the_file_a_bounded_number_of_times_over` | No | -| One search reading without end | Reads of at most 64 KB, each strictly after the last, ending at the bound, at the end of the file, at a header, or on an empty read | `StreamLengthTests.A_file_of_data_with_no_endstream_is_searched_once_to_its_end`, `A_source_that_returns_less_than_it_holds_ends_the_search` | No | -| The same search made at every new parse | The result is kept by data start for the life of the document | `StreamLengthTests.A_stream_whose_length_is_wrong_is_searched_and_reported_once_however_often_the_cache_lets_it_go` | — | +| A declared length past the file | It is checked against the file before the file is asked, and confirms nothing: the stream goes to the search, whose `endstream` settles its length, and with none its data ends as the rows below say. The length a stream is given is clamped to what the file holds | `HostileInputTests.Ignores_a_stream_length_larger_than_the_file`, `A_length_the_file_cannot_hold_is_settled_by_the_endstream_found_without_reaching_a_guard`, `Many_lengths_the_file_cannot_hold_read_the_file_once_not_once_each`, `A_length_the_file_cannot_hold_with_no_endstream_takes_the_rest_of_the_file_without_reaching_a_guard`; `WindowEdgeTests.A_stream_whose_length_the_file_cannot_hold_is_read_to_its_endstream` | Yes | +| A stream's data read through the window its object is parsed through: a wrong `/Length` growing it to `MaxObjectLength` for a few bytes of data, and the object then reported cut | A stream's data never counts toward `MaxObjectLength`, however its end is found, nor the end-of-line after its `stream` keyword: no window is grown to find an `endstream`, nor where the data starts, which the file says past a window a guard ended on the keyword; the data past the bytes the parser holds is searched in the file, and a fault of its length is the file's, not the guard's ([#174]) | `StreamLengthTests.A_stream_s_data_counts_for_nothing_toward_MaxObjectLength_and_its_fault_is_the_file_s`, `A_stream_whose_keyword_a_limit_cuts_records_nothing_of_its_length`; `WindowEdgeTests.A_stream_reads_the_same_when_a_guard_s_window_ends_on_its_keyword`, `A_cross_reference_stream_reads_whole_when_its_guard_s_window_ends_on_its_keyword`; `HostileInputTests.A_stream_whose_endobj_follows_its_length_without_endstream_reads_no_further_than_its_window_whatever_follows`; `WindowEdgeTests.A_keyword_longer_than_stream_reads_the_same_wherever_the_window_edge_cuts_it`; `ParserTests.Leaves_a_stream_whose_window_ends_on_its_keyword_to_a_larger_one`, `Leaves_a_stream_with_no_endstream_in_a_window_to_a_larger_one` | No | +| Confirming a length by reading past the buffer or the file, or through white space without end; a `%` of the data read as a comment that confirms a length too short | An `endstream` after the declared length confirms it, after any white space and comments, however many ([#174]); the 35,716 streams of the corpus whose length is confirmed have a gap of 0 to 2 bytes. A comment counts there only once white space comes before it: a `%` right where the length ends is the data's, so the length goes unconfirmed and the search below finds where the data ends. The bytes the parser holds decide when they can, and what follows an `endstream` they hold, when they end before it shows, is read on in the file, as past them: where the window ends decides nothing of the object's `endobj`. Past them, a first read of 64 bytes on the stack (`PdfObjectParser.EndObjLookahead`, a first read and no bound), refused outside the file and clamped to it, then reads that double up to 64 KB as far as the white space and comments run, once for each place, counted with the searches below; once those have read as much as they may, the first read alone decides. A length past the file confirms nothing | `WindowEdgeTests.A_stream_the_file_cuts_short_keeps_its_declared_length_without_asking_the_source_past_its_end`, `What_follows_an_endstream_the_window_holds_is_read_wherever_the_window_edge_falls`; `StreamLengthTests.A_stream_past_the_window_whose_length_endstream_follows_is_read_as_declared_for_a_few_bytes_more`, `A_length_any_white_space_or_comment_separates_from_endstream_is_confirmed`, `White_space_before_endstream_that_runs_past_the_first_read_is_read_on_and_confirms_the_length`, `What_follows_the_endstream_of_a_stream_past_the_window_is_seen_however_far_white_space_runs`, `A_percent_sign_where_a_short_length_ends_is_data_and_no_comment_that_confirms_the_length`, `A_comment_after_white_space_where_the_length_ends_confirms_it`; `ParserTests.Reads_endstream_after_any_white_space_and_comments_or_says_the_bytes_cannot_tell`, `Bytes_that_end_the_file_before_endstream_hold_none`; `StreamLengthTests.A_comment_in_the_gap_that_runs_past_the_window_is_read_on_and_its_check_kept_for_a_second_reading`, `An_endstreamx_read_on_past_the_window_confirms_the_length_and_is_no_endobj`, `White_space_after_a_length_past_the_window_is_read_on_to_the_end_of_the_file` | Property | +| A search for a lost `endstream` running into other objects, or past its own object's end | A length unconfirmed, or none to confirm, sends the stream to one search for its first `endstream`: in the bytes the parser holds, then in the file, up to the nearer of the first object header the bytes hold (`HeaderText`) and the next object the index as written places, or to the end of the data. A stream whose `/Length` leads to an object a guard cut or left out of the index may be valid, its length only hidden, and its data may spell a header, as an embedded file's does — and the `endstream` of a stream the embedded file holds: a header the data spells that stops the search, no `endobj` of the object's own before it, leaves the data up to it to the guard, cut by it, the object's `endobj` unjudged, nothing said of the file; so does an `endstream` the data spells — a string's, say, an `endobj` after it or not —, which only the hidden length could tell from the stream's own. An `endstream` ends data that is the file's only where what follows the object's end — its `endobj`, or the `endstream` where none follows — is the next object the index places — at its row, or past the white space a row names before its header —, a cross-reference stream the chain read, which no row of its own need place, or the end of the file, which data that goes on is never followed by, or a whole `xref` or `startxref` keyword, which data may spell (below): read once for each place, within what the searches may read (`IPdfStreamDataProvider.IsOwnEndStream`). Proven, it ends data that is the file's, its `endobj` judged; but white space before the keyword — an end-of-line the search leaves out, or white space and comments the data ends in — only the hidden length says is the data's or not: the data's end is decoded as one the guard hides (`PdfStreamData.EndHidden`). The first filter decodes the data up to where that run starts, found in time linear in the data — a comment past white space within the data, never at its first byte, and none in ASCII85 data, where a `%` is a digit —: what it meets before is the file's, its end marker counting only there, its running out there — a Flate checksum short, an end marker missing — says nothing of the file, and where its output may lack what the run holds — Flate that lost its tail, LZW or RunLength data without its end marker — the output's end is the guard's, cut, for the predictor, which runs on Flate and LZW steps only and undoes the last row as far as it goes, the filters after it and an object stream's last member. Flate data whose deflate body ends in the data as taken decodes whole, its repair reported and a checksum that disagrees only where it reaches into the run silenced; white space decodes to nothing in ASCII85 or ASCIIHex data, whose output is whole, and so does a comment in ASCIIHex, which no valid hexadecimal data holds; the filters after a first whose output is whole decode as read whole, their faults the file's. A reading that cannot tell where the object ends takes back what an earlier one recorded of its `endobj` ([#224]). With none before that stop, the object's `endobj` plays its part ([#174]). A declared length that ends before the stop and that its `endobj` follows, past white space and comments as above, is kept, `stream.truncated`. Otherwise the object's own `endobj`, with only white space and comments between it and the stop, ends the data, its end-of-line left out, `stream.truncated`; an `endobj` a token follows past its comments ends nothing. That `endobj` is followed forward through the search's own reads, which read every byte up to the stop already (`EndObjTail`): no byte is read again to find it. Otherwise a declared length that ends before the stop is kept, `stream.length-invalid`, and its `endobj` is not judged. Otherwise the data runs to the stop, `stream.truncated` at the end of the file and `stream.length-invalid` at the next object, and the object lacks its `endobj`, unless the data taken holds one | `StreamLengthTests.A_stream_past_the_window_with_no_endstream_before_the_next_object_ends_at_its_object_s_endobj`, `A_stream_past_the_window_with_no_endstream_before_an_object_the_window_holds_ends_at_its_endobj_unsearched`, `The_search_stops_at_the_next_object_of_an_index_rebuilt_as_the_document_opened`, `A_stream_that_lost_its_endstream_takes_nothing_of_the_object_after_it`, `A_stream_that_lost_its_endstream_runs_to_a_header_its_object_has_no_endobj_before_and_lacks_its_endobj`, `A_stream_that_endobj_follows_without_its_endstream_keeps_its_declared_length`, `A_stream_whose_endobj_the_trailer_follows_runs_to_the_end_of_the_file_and_leaves_its_endobj_unjudged`, `IPRES_s_stream_that_lost_its_endstream_keeps_the_length_its_endobj_follows`, `A_percent_sign_where_a_short_length_ends_is_data_before_the_object_s_endobj_too`, `An_object_s_endobj_ends_data_with_no_endstream_whatever_comments_follow_it`, `An_endobj_a_token_follows_past_its_comments_ends_no_data`, `A_file_that_ends_on_the_stream_keyword_ends_the_stream_s_data_there_and_says_so`; `WindowEdgeTests.An_object_s_endobj_ends_data_with_no_endstream_wherever_the_reads_part_what_follows_it`; `ParserTests.A_stream_that_lost_its_endstream_ends_at_its_own_endobj_before_the_next_object`, `Ends_a_stream_s_data_at_its_object_s_endobj_that_only_white_space_and_comments_follow`, `A_stream_that_lost_its_endstream_runs_to_the_header_its_bytes_hold_when_no_endobj_precedes_it`, `A_window_given_to_the_parser_alone_that_cannot_say_where_a_stream_ends_asks_for_more`; `GuardCutValueTests.Data_a_header_stops_where_a_guard_hid_the_length_is_the_guard_s`, `A_stream_a_guard_hid_the_length_of_ends_at_its_own_endobj_before_a_header`, `An_endstream_the_data_spells_where_a_guard_hid_the_length_ends_data_that_is_the_guard_s`, `An_endstream_the_next_object_follows_where_a_guard_hid_the_length_lacks_its_endobj_as_the_file_s_fault`, `An_endstream_where_a_guard_hid_the_length_ends_data_that_is_the_file_s_where_what_follows_proves_it_its_own`, `A_cross_reference_stream_with_no_row_of_its_own_proves_the_endstream_before_it`, `A_cross_reference_stream_named_before_white_space_proves_the_endstream_before_it_however_much`, `Flate_data_whose_last_byte_is_a_carriage_return_before_a_proven_endstream_is_no_fault_of_the_file`, `Data_that_ends_in_white_space_right_before_a_proven_endstream_ends_where_the_guard_hides`, `A_checksum_short_of_a_byte_white_space_before_a_proven_endstream_completes_is_no_fault_of_the_file`, `A_checksum_that_disagrees_before_the_white_space_a_hidden_length_leaves_is_the_file_s`, `A_filter_after_one_that_ends_where_its_format_says_decodes_as_read_whole_whatever_hides_the_end`, `Predicted_rows_a_hidden_end_leaves_a_byte_short_are_undone_as_far_as_they_go`, `Lzw_rows_a_hidden_end_leaves_a_byte_short_are_undone_as_far_as_they_go`, `Lzw_data_that_lacks_its_end_code_before_a_proven_endstream_past_an_end_of_line_is_no_fault_of_the_file`, `White_space_and_comments_a_hidden_length_leaves_are_not_decoded_as_data_that_lacks_its_end_marker`, `An_object_stream_whose_last_code_a_hidden_end_leaves_a_byte_short_ends_where_the_guard_hides`, `Data_a_hidden_length_ends_decodes_as_read_whole_but_for_what_the_run_after_it_may_hold`, `Flate_data_a_hidden_length_ends_where_its_deflate_body_does_decodes_as_read_whole`, `An_ascii85_object_stream_that_lacks_its_end_marker_before_a_proven_endstream_decodes_as_read_whole`, `An_endstream_and_an_endobj_the_data_spells_where_a_guard_hid_the_length_end_data_that_is_the_guard_s`, `Rows_past_a_spelled_endstream_are_the_guard_s_when_the_length_s_chain_ends_at_its_last_object_a_guard_cut`; `StreamLengthTests.A_stream_left_unsearched_is_reported_once_however_often_it_is_parsed`, `A_stream_left_unsearched_whose_length_runs_past_the_file_takes_the_data_its_object_holds`, `A_stream_read_once_the_searches_read_as_much_as_they_may_is_not_read_on_past_its_window`; `WindowEdgeTests.An_endobj_a_regular_byte_goes_on_is_missing_wherever_the_window_edge_cuts_it`, `An_endobj_that_ends_a_longer_token_ends_no_data_wherever_the_reads_part_it` | Plausibly | +| Many searches, and reads on through white space, over one stretch of the file | What the searches read past the bytes the parser holds, and the reads on through white space and comments after a declared length, add up to at most four times the file in a document (`EndStreamSearchPasses`, an internal constant, with its reason): a valid file is read past the window only by a cross-reference stream whose `/Length` the chain cannot read, searched once, by a stream whose `/Length` a guard cut or left out of the index, searched once, and by white space after a declared length that runs past a first read, about twice the file, three times over the white space a cross-reference stream's search takes as data that its length, once read, is read on through, and the proof that an `endstream` found under a hidden `/Length` is the stream's own, 48 or 64 bytes past each, once, and the white space it crosses once more at most — the run before an `endobj`, or before what follows the keyword where no `endobj` does, taken from the read-on that read it, a run a row names before the next header read once from the row —, about three times the file in all; any other search is of an invalid stream, and reads about twice what its object spans. Past the bound, only the bytes the parser holds are searched: a declared length that ends before the next object the index places, or the end of the file, is kept, and the data otherwise runs to it, each reported as taken without a search — but where a guard hid the `/Length`, which would say how far the data runs, the data up to the stop is the guard's, its object's `endobj` unjudged, nothing reported ([#224]) | `StreamLengthTests.Streams_whose_data_share_a_stretch_read_the_file_a_bounded_number_of_times_over`; `DocumentReaderTests.Deferred_lengths_followed_by_long_white_space_have_it_read_on_once_however_their_data_ends`; `GuardCutValueTests.The_proofs_of_small_streams_whose_length_a_guard_hid_read_less_than_the_file`, `The_proofs_of_streams_white_space_follows_read_less_than_three_times_the_file`, `The_proofs_of_endstreams_data_spells_before_white_space_read_less_than_three_times_the_file`, `The_place_a_row_names_before_the_next_header_is_read_once_however_often_the_stream_is_parsed`; `StreamLengthTests.A_stream_left_unsearched_whose_length_a_guard_hid_is_the_guard_s`; `HostileInputTests.Many_lengths_the_file_cannot_hold_read_the_file_once_not_once_each`, `A_deferred_length_s_data_with_no_endstream_makes_its_section_malformed_reading_the_file_a_few_times` | No | +| One search reading without end | Reads that double from twice what the parser holds of the data up to 64 KB, each starting further on than the last, ending at the next object the index places, at the end of the file, at a header, at an `endstream`, or on an empty read | `StreamLengthTests.A_file_of_data_with_no_endstream_is_searched_once_to_its_end`, `A_source_that_returns_less_than_it_holds_ends_the_search` | No | +| The same search made at every new parse | The result is kept by data start for the life of the document, and so is what the white space after a declared length leads to. Whether the object's `endobj` follows the declared length is read again only when that length changes, as a rebuild that replaces the `/Length` changes it | `StreamLengthTests.A_stream_whose_length_is_wrong_is_searched_and_reported_once_however_often_the_cache_lets_it_go`; `ReadOrderTests.A_stream_read_again_after_a_rebuild_changed_its_length_ends_as_one_read_after_the_rebuild_does` | — | +| Encoded data of any length: a `/Length` of a few digits claiming gigabytes, one past what an array holds, which would exhaust memory at each read, or data decoding hands back as it is, which `MaxDecodedStreamLength` does not bound | `MaxStreamLength` (*The guards*): what is read of one stream's data at once stops at it, what lies past it is cut by a guard and reported as `limit.stream-length` — once for the stream in its document's diagnostics, and with each decoding that meets the cut in diagnostics a caller passes to `Decode`, as `limit.decoded-stream` is —, and the filters decode what was read as data the reader cut ([#170]) | `ReaderLimitsTests.A_stream_cut_at_MaxStreamLength_decodes_what_lies_within_it_as_data_the_reader_cut_and_is_reported_once`, `Data_decoding_hands_back_as_it_is_is_bounded_by_MaxStreamLength_and_not_by_MaxDecodedStreamLength`, `A_length_past_what_an_array_holds_ends_at_the_guard_and_never_in_an_exhausted_memory`, `Decoding_into_diagnostics_of_one_s_own_reports_the_cut_max_stream_length_made_each_time` | No | +| A stream's raw data held for as long as the stream is, and streams whose data share a stretch of the file each holding a copy of it: two hundred sharing a megabyte, 200 MB | Nothing keeps a stream's encoded data (`FileStreamData`): it is read each time it is asked for, by `GetRawBytes` or a decoding. A source in memory lends it where it lies, with no copy — a view of that memory, which keeps it alive after the document is disposed for as long as it is held, and changes as it does —, and any other reads it into an array of its length. A stream keeps its place and length, not its bytes ([#170]) | `DocumentReaderTests.A_stream_s_data_is_read_from_the_file_each_time_it_is_asked_for_and_kept_by_nothing`; `HostileInputMemoryTests.Streams_whose_data_share_a_stretch_of_the_file_hold_none_of_it_once_read`, `Validating_object_streams_whose_data_share_a_stretch_of_the_file_holds_no_copy_of_their_data`; `HostileInputTests.Reading_streams_whose_data_share_a_stretch_of_a_file_held_in_memory_allocates_none_of_their_data` | No | | A stream decoded long after opening escaping its document's guards | The stream's data carries its document's guard and diagnostics | `ReaderLimitsTests.A_stream_decoded_without_diagnostics_reports_the_bound_to_its_document` | Yes | Known gaps: -- [#170] Raw data is bounded only by the file. Every cached stream keeps its own copy, overlapping streams multiply - it, and data passed through undecoded escapes `MaxDecodedStreamLength`. -- [#174] A length is misjudged when more than four white-space bytes precede `endstream`, when the file cannot hold - the `/Length`, or when the search inside the window crosses into the next object. +- [#280] Streams whose data share a stretch of the file each read that stretch again from a + source not held in memory, whenever their data is asked for. What overlapping streams cost is time rather than + memory, bounded for each read by `MaxStreamLength` alone; a check for overlapping data is proposed. - [#138] A stream searched for its `endstream` can keep a length that depends on the object read first. -- [#53] Encoded data past about 2 GB cannot be represented, and a length just under `int.MaxValue` throws - `OutOfMemoryException`. +- [#287] Under a `/Length` a guard hid, an `endstream` followed by an object only a superseded row places — an older + definition an update replaced — is not proven the stream's own: its data, whole, is the guard's. +- [#53] Encoded data past about 2 GB cannot be represented: a `/Length` past what an array holds is cut at + `MaxStreamLength`, whose ceiling is `Array.MaxLength`. - [#48] Data that decodes past about 2 GB cannot be read whole. +- Accepted, with no issue: the last object of a classic file, whose stream lost its `endstream` and declares no length + within the file the reader can take, takes as its data everything to the end of the file, its cross-reference table + and trailer included: neither `xref` nor `trailer` stops the search ([#174]). +- Accepted, with no issue: under a `/Length` a guard cut or left out of the index, data that spells `endstream`, an + `endobj` or not, then `xref` or `startxref`, and goes on, ends there as the file's data, and its object as ending + there: only the hidden length could tell, and a valid file's last stream before its table is told so ([#224]). +- Accepted, with no issue: under a `/Length` a guard cut or left out of the index, data an `endstream` proven its own + ends past white space or comments — the end-of-line the search leaves out before the keyword, above all — decodes + up to where that run starts: its first filter's running out there — a Flate stream that lost its tail or its + checksum, ASCII85, ASCIIHex, RunLength or LZW data without its end marker — is not reported, though a whole read + reports it, nor what the filters after it meet of an end the guard's; a checksum whose bytes reach into the run says + nothing; and where the data's last bytes, read whole, look like white space or a comment after white space, what a + binary filter that ran out decodes from them is cut, the guard's. Only the hidden length says whether that run is the + data's ([#224]). +- Accepted, with no issue: where the chain names a section, an object that declares another type and gives a `/W` + before `MaxTrailerLength` cuts its dictionary is taken for no section, as a CIDFont's dictionary is read whole, no + stream data after it, and the section is looked for nearby; a stream of that type, whose stream data the cut hides, + would be read as the section whole ([#263]). ### Decoding @@ -369,14 +406,15 @@ its zlib patched is keeping part of the reader patched. | A decompression bomb | Every decoder writes through `PdfBoundedOutput`, or a buffer capped like it, at `MaxDecodedStreamLength` | `FilterTests.Keeps_exactly_what_the_bound_allows_and_reports_only_a_bound_it_met`; `HostileInputTests.Decodes_a_run_length_stream_no_further_than_the_bound` | Not at the bound | | A `/Filter` array of any length, each filter decoding to the bound | The filters before the last share one `MaxDecodedStreamLength` for what they pass along together, and the last has its own: a chain holds and fills two bounds at most, whatever its length. A `/Crypt` filter, which passes its data along as it is, is neither charged nor the last. A filter that decodes nothing ends the chain, so that a faulty Flate stream is read again once, not once an entry ([#166]) | `FilterTests.Shares_one_bound_among_the_filters_before_the_last`, `Bounds_a_filter_in_the_middle_of_a_chain_as_well_as_the_last_and_reports_the_first_that_stopped`, `Ends_a_chain_at_a_flate_step_that_decodes_nothing_and_returns_what_the_filters_before_it_decoded`, `A_crypt_filter_is_neither_charged_to_the_bound_the_filters_share_nor_the_last`; `HostileInputTests.Opens_a_cross_reference_stream_whose_run_length_filters_multiply_its_data_at_a_cost_its_number_of_filters_leaves_alone`, `Decodes_a_long_chain_of_sound_flate_layers_within_twice_the_bound`, `Decodes_data_no_flate_reading_can_decode_once_however_many_flate_filters_name_it`; `FilterDamageTests.A_chain_of_filters_decodes_to_its_data_or_to_the_start_of_it_with_the_guard_alone_reported` | Property | | A `/Filter` array of millions of filters, each costing decoding whatever its data | ISO 32000 sets no limit, and each filter costs a few microseconds, LZW a table of 42 KB, whatever its data; a filter whose data decodes to itself — `/RunLengthDecode` over `FF FF`, an identity `/Crypt` — never runs out of it. `MaxFilterChainLength` (64) bounds the filters a stream's data passes through: decoding stops before the first past it, and the elements past it are not read, by decoding, `HasImageFilter` or the cross-reference stream's check alike. Data a filter decodes to nothing ends decoding. What a chain keeps of what it read is what the file did not write plainly, within the entries the guard lets run: nothing for a name the array holds, the name an element written as a reference leads to, and only the values of parameters that differ from none at all; an array of forty million names, which an object stream's member holds in 350 KB of file, ran a decode out of memory when the chain kept 32 bytes for each | `HostileInputTests.Decodes_a_stream_naming_a_million_filters_at_the_cost_of_as_many_as_the_guard_lets_run`, `Decodes_a_stream_naming_a_million_filters_without_keeping_anything_for_the_names_its_array_holds`, `Decodes_nothing_further_once_a_filter_has_decoded_a_stream_to_nothing`; `FilterDamageTests.Gives_a_dictionary_of_parameters_beside_a_chain_to_its_flate_and_its_lzw_steps_alike`, `Reads_a_parameter_past_an_int_as_the_int_nearest_it_rather_than_its_low_bits`; `FilterTests.Reads_nothing_past_the_most_filters_the_guard_lets_a_stream_pass_through`, `Says_a_stream_is_an_image_only_when_its_image_filter_stands_within_the_guard`, `Reads_the_filters_a_chain_names_through_references_in_their_order_crypt_filters_among_them`; `CrossReferenceRuleTests.A_cross_reference_stream_s_filters_past_the_guard_are_not_followed`; `ReaderLimitsTests.Reaching_a_guard_keeps_what_fits_and_names_the_property_that_lifts_it`, `A_stream_the_reader_reads_whose_filters_pass_the_guard_is_cut_before_its_data` | No | -| A guard's cut reported as damage by the filters after it | Every filter after one the bound stopped, or after the parser's window cut the data, is given data the reader cut: no lost tail or missing checksum, no rows longer than the data; a hexadecimal digit or an ASCII85 group the cut parted is dropped, not completed, and no end marker is missing from it. A fault, or a checksum that disagrees, in what was read stays the file's ([#199], [#141]) | `FilterDamageTests.Says_nothing_of_the_end_of_data_a_filter_before_it_stopped_at_the_bound`, `Reports_a_fault_in_the_data_a_filter_kept_though_a_filter_before_it_stopped_at_the_bound`, `Drops_the_unit_a_guard_of_the_reader_parted_rather_than_complete_it`, `Completes_the_unit_its_end_marker_closes_however_the_reader_cut_the_data_after_it`, `Undoes_a_predictor_over_the_rows_of_a_bounded_lzw_output_and_the_part_of_the_last_it_holds`; `FilterTests.Undoes_a_predictor_over_the_rows_of_a_bounded_output_and_the_part_of_the_last_it_holds`, `Leaves_the_partial_last_row_of_data_the_reader_did_not_cut_as_the_predictor_left_it` | Property | +| A guard's cut reported as damage by the filters after it | Every filter after one the bound stopped, or after `MaxStreamLength` cut the data, is given data the reader cut: no lost tail or missing checksum, no rows longer than the data; a hexadecimal digit or an ASCII85 group the cut parted is dropped, not completed, and no end marker is missing from it. A fault, or a checksum that disagrees, in what was read stays the file's ([#199], [#141]) | `FilterDamageTests.Says_nothing_of_the_end_of_data_a_filter_before_it_stopped_at_the_bound`, `Reports_a_fault_in_the_data_a_filter_kept_though_a_filter_before_it_stopped_at_the_bound`, `Drops_the_unit_a_guard_of_the_reader_parted_rather_than_complete_it`, `Completes_the_unit_its_end_marker_closes_however_the_reader_cut_the_data_after_it`, `Undoes_a_predictor_over_the_rows_of_a_bounded_lzw_output_and_the_part_of_the_last_it_holds`; `FilterTests.Undoes_a_predictor_over_the_rows_of_a_bounded_output_and_the_part_of_the_last_it_holds`, `Leaves_the_partial_last_row_of_data_the_reader_did_not_cut_as_the_predictor_left_it` | Property | +| A `/Filter`, a `/DecodeParms` or a step's parameter that leads to an object a guard cut or left out of the index, read as no filter, or as the `/Flate` a cut parted of `/FlateDecode`: an object stream's header read as stray bytes, its catalog looked for elsewhere, a sound file under a lowered limit judged broken; a stream said to be an image whose data such a value leaves Flate-encoded | The chain of filters ends at such a value, through a chain of references too, as at the element past `MaxFilterChainLength` (`FilterChainEnd.Guard`): the steps before it run, the data is left encoded by the rest (`leftEncoded`), and nothing more is reported, the guard having been reported where it was reached. An object stream so left is hidden and its members are the guard's: a catalog among them is taken as read, and the validator reports `xref.checked-in-part`, not `xref.object-stream-broken`. Decoding says whether the stream's own data is cut — by `MaxStreamLength`, or by a filter stopped at `MaxDecodedStreamLength` —, which is what the readers of rows and members ask: a guard that loading a filter's value reaches is that value's, and hides none of the stream's own faults ([#263]). `HasImageFilter` reads what decoding reads before the image filter — the parameters of a Flate or LZW step before it, which it may load, decoding the object stream that holds one, as decoding would — and says no where such a value ends the chain first: an extractor that trusts it is not handed encoded data as the image | `GuardCutValueTests.An_object_stream_whose_filter_values_a_guard_cut_is_hidden_and_its_members_taken_as_read`, `A_content_stream_whose_filter_a_guard_cut_comes_back_as_the_filters_before_it_decode_it`, `A_guard_a_filter_value_reaches_as_it_loads_is_not_the_stream_s_own_cut`, `A_parameter_a_guard_cut_before_an_image_filter_leaves_no_encoded_image`; `FilterTests.Says_a_stream_is_an_image_only_when_decoding_it_stops_at_an_image_filter` | No | | An output buffer sized from the file | The first buffer is an estimate from the input, capped at the bound; it doubles and never grows past it | `FilterTests.A_filter_s_first_buffer_is_sized_by_its_bound_not_by_its_input`, `A_bounded_output_never_grows_past_its_bound_and_hands_back_a_full_buffer_as_it_is` | Yes | | Predictor parameters that overflow or size a row before a check | The row length is computed in 64 bits and compared with the data before anything is allocated | `FilterTests.Leaves_the_data_as_decoded_when_the_predictor_describes_rows_it_cannot_hold` | Seeds only | | An LZW code past the table | A fixed table of 4,096 codes. A code not yet defined stops decoding, keeps what came before and reports `filter.failed` | `FilterDamageTests.Decodes_an_lzw_code_up_to_the_next_one_to_define_and_stops_at_any_past_it`; `FilterTests.Stops_an_lzw_stream_at_a_code_it_has_not_defined_and_says_so` | Two seeds | | Data a filter cannot decode as its format says, read in silence | ASCII85, ASCIIHex and RunLength decode what breaks their format as other readers do and report it as `filter.failed`, each kind once a step, at the byte where the first lies: a byte that encodes nothing, passed over; an ASCII85 group past four bytes, a `~` no `>` follows, a last group of one digit, the `<~` PostScript writes. Data without its end marker — LZW's end-of-data code among them — decodes to its end: a repair when it ends after a whole unit, a warning when it ends inside one, an LZW code included. Counting costs nothing allocated; a filter stopped at the bound counts the faults of the part it read, and the end of data the reader cut or a filter stopped at the bound, a `~` whose `>` lies past the cut among it, is the guard's ([#141]) | `FilterDamageTests.Passes_over_the_bytes_that_encode_nothing_and_reports_where_the_first_lies`, `Reports_an_ascii85_group_whose_value_does_not_fit_in_four_bytes`, `Reads_an_ascii85_marker_out_of_place_as_other_readers_do_and_reports_it`, `Decodes_data_without_its_end_marker_to_its_end_and_says_whether_any_may_be_lost`, `Reports_no_missing_end_marker_where_a_filter_stopped_decoding_at_the_bound`, `Reports_no_missing_end_marker_where_a_filter_stopped_at_the_bound_completing_its_last_unit`, `Reports_no_missing_greater_than_sign_where_the_reader_cut_ascii85_data_after_its_tilde`, `Reports_no_end_marker_a_guard_of_the_reader_cut_off_and_a_fault_in_what_it_read`, `Decodes_an_lzw_stream_without_its_end_of_data_code_to_its_end_and_says_whether_a_code_was_cut`; `TextFilterRefereeTests` (qpdf) | No ([#176]) | | Damaged Flate data read again and again | A form is read again only after a fault, into the same output. The byte-at-a-time pass covers only the 8 KB piece the fault was met in (`FlateInput.ChunkLength`). A stream costs at most eight readings, three of them byte by byte, and a chain reads its input again only in the filter that met the fault: one that decodes nothing ends it. This is accepted: damaged streams read about 40 times slower, still linear in the file (#135, comment of 2026-09-30) | `FilterDamageTests.Reads_sound_data_once`, `Reads_again_a_byte_at_a_time_only_the_piece_of_input_the_fault_was_met_in`, `Reading_damaged_data_again_allocates_no_second_output_and_sound_data_nothing_to_read_it_again_with` | Yes | | The inflater's exceptions escaping | Only its complaints about the data (`InvalidDataException`, `IOException`) are faults; a preset-dictionary header is refused | `FilterDamageTests.Takes_only_the_inflater_s_complaints_about_its_data_as_faults`, `Reports_a_zlib_header_that_asks_for_a_preset_dictionary_instead_of_throwing` | Yes | -| Image codecs | DCT, JPX, JBIG2 and CCITT data is never decoded: the chain stops and the data comes back as it is. An unknown filter, or an element of `/Filter` that names none, is `filter.unsupported`, placed where the stream's data starts, and the chain stops there too: no filter after it reads data it cannot read ([#159], [#209]) | Structural; `FilterTests.Stops_a_chain_at_the_first_filter_it_does_not_know_and_reports_it_once`, `Ends_a_chain_at_an_element_that_names_no_filter_and_returns_what_the_filters_before_it_decoded` | — | +| Image codecs | DCT, JPX, JBIG2 and CCITT data is never decoded: the chain stops and the data comes back as it is, within `MaxStreamLength`. An unknown filter, or an element of `/Filter` that names none, is `filter.unsupported`, placed where the stream's data starts, and the chain stops there too: no filter after it reads data it cannot read ([#159], [#209]) | Structural; `FilterTests.Stops_a_chain_at_the_first_filter_it_does_not_know_and_reports_it_once`, `Ends_a_chain_at_an_element_that_names_no_filter_and_returns_what_the_filters_before_it_decoded` | — | Known gaps: @@ -452,8 +490,9 @@ Known gaps: ## What holds the defenses - **Hand-written hostile tests**, on every commit: `HostileInputTests`, `ReaderLimitsTests`, `ReaderTimingTests` - (a ratio, not a wall-clock budget), `WindowEdgeTests`, `StreamLengthTests`, `FilterTests`, `FilterDamageTests`, - `DiagnosticsTests`, and the validation rule tests. They hold the bounds a mutation of a small file cannot reach. + (a ratio, not a wall-clock budget), `WindowEdgeTests`, `StreamLengthTests`, `GuardCutValueTests`, `FilterTests`, + `FilterDamageTests`, `DiagnosticsTests`, and the validation rule tests. They hold the bounds a mutation of a small + file cannot reach. - **Budgets on real documents, and on a generated index.** `CorpusReadingTests` holds what indexing a 1,000-page document and walking its page tree allocate, and each operation on every corpus document — opening it, reading every object, walking its pages, validating a damaged one — to 20 s. `CorpusValidationTests.Validating_the_largest_document_stays_within_its_budget` @@ -711,6 +750,7 @@ The last column also names the questions they leave open, which that milestone s [#218]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/218 [#220]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/220 [#221]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/221 +[#224]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/224 [#229]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/229 [#230]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/230 [#237]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/237 @@ -720,8 +760,12 @@ The last column also names the questions they leave open, which that milestone s [#260]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/260 [#261]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/261 [#262]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/262 +[#263]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/263 [#273]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/273 [#274]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/274 [#275]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/275 [#276]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/276 [#277]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/277 +[#279]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/279 +[#280]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/280 +[#287]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/287 diff --git a/docs/website/docs/concepts/diagnostics.md b/docs/website/docs/concepts/diagnostics.md index 0e9dd1e6..1fefdc0d 100644 --- a/docs/website/docs/concepts/diagnostics.md +++ b/docs/website/docs/concepts/diagnostics.md @@ -26,46 +26,91 @@ not how wrong the file is. The reader parses an object through a window of the file — 8 KB to start with — and reads it again through a larger one when it runs past the edge. What the smaller window saw there, such as a string -without its end or a stream without its `endstream`, is dropped with that attempt, so the report says what -the object holds, not where a window happened to end. When `MaxObjectLength` allows no larger window, the -object is kept as far as it was read and the guard is reported: what the parse met before the edge — a stray -token, a key that is not a name — is reported beside it, and nothing of the token the edge cut, nor of what that token -leaves open, is. - -A stream's declared length is checked against the `endstream` that must follow it. When the data runs past -the window, the reader asks the file for the few bytes after the declared length rather than reading the -data, and they tell it whether `endobj` follows too. When no `endstream` is there, the first one after the -start of the data ends it, and where it is looked for depends on where the declared length ends: - -- **Inside the window, or nowhere** — a `/Length` that gives no length —, the first `endstream` after the - data is taken wherever it lies, past the header of the next object too: the reader looks in the window, - and in a larger one while none is found, as far as `MaxObjectLength` allows. -- **Past the window**, the reader searches the file from the start of the data up to the next object — - the one the file's index places as the file wrote it (as the reader rebuilt it when the document opened, if - the file wrote none), or the first object header, `N G obj`, the file's bytes hold, whichever comes first —, - or up to the end of the file. - Stopping at the next object keeps a later object's `endstream` from ending this stream, which is what a - stretch of zeros that erased the end of one and the objects after it would otherwise do, or an object whose - entry the index lost; when none lies before it, the declared length is kept, and the report says so. Text - in a damaged stream's data that reads as an object header stops the search as well, and the declared - length is kept: an object number and a generation the parser takes, however many zeros lead them, white - space between the three tokens, however much, and white space, a delimiter, or the start or the end of - the search around them. A comment between the tokens makes no header here, as it makes none for the scan - that rebuilds a damaged index; nor does a regular character glued before the number, though that scan - takes the digits after it for one. - -The search is not one of the reader's limits: a valid file's `endstream` follows its length, so only a -damaged file is searched, and once for each stream. Where it stops depends on the file's bytes and on its -index as the file wrote it, which the reader rebuilding or correcting its own index later does not change: -in the 401 documents of the project's test corpus measured, each stream takes the same length whatever was -read before it — though a damaged index can still serve another copy of an object, at another offset, -according to what was read first. A file crafted for it can still make what a stream holds depend on which -object was asked for first. What the searches of one document read together is bounded: none starts once -they have read four times the file's length. They read twice the file at most — 0.35 of it at most in the -test corpus — unless their stretches overlap: one object's header inside another's dictionary, or a header -the search does not take — a regular character glued before its number — that the index as the file wrote -it does not place. Past the bound, a stream keeps its declared length without a search, and the report says -so. +without its end, is dropped with that attempt, so the report says what the object holds, not where a window +happened to end. When `MaxObjectLength` allows no larger window, the object is kept as far as it was read and the +guard is reported: what the parse met before the edge — a stray token, a key that is not a name — is reported beside +it, and nothing of the token the edge cut, nor of what that token leaves open, nor of the next object's header the edge +cut short inside an array or a dictionary — `6 0` or `6 0 ob`, whose number and generation would otherwise read as keys +that are not names, or, for `6 0 ob`, which no reference can be, the number as a value —, is. A stream's data is no part of what a window must hold: where it +ends is found in the file, whatever window its dictionary was read through, and it never counts toward +`MaxObjectLength`. + +A stream's declared length is confirmed by the `endstream` that must follow it, after any white space and comments, +however many — a comment only once white space comes before it: ISO 32000-1 (7.2.3) counts a comment as white space +outside a stream's data, and a `%` right where the declared length ends is the data's, the length too short. Past the +window, the reader asks the file for the bytes after the declared length rather than reading the data, reading on as +far as white space and comments run, and they tell it whether `endobj` follows too. Any other +stream — its length unconfirmed, or a `/Length` that gives none — is searched once for the first `endstream` after the +start of its data, which ends it: in the bytes the parser holds, then in the file, up to the next object — the one the +file's index places as the file wrote it (as the reader rebuilt it when the document opened, if the file wrote none), +or the first object header, `N G obj`, the file's bytes hold, whichever comes first —, or up to the end of the file. + +Stopping at the next object keeps a later object's `endstream` from ending this stream, which is what a stretch of +zeros that erased the end of one and the objects after it would otherwise do, or an object whose entry the index lost. +Text in a damaged stream's data that reads as an object header stops the search as well: an object number and a +generation the parser takes, however many zeros lead them, white space between the three tokens, however much, and +white space, a delimiter, or the start or the end of the search around them. A comment between the tokens makes no +header here, as it makes none for the scan that rebuilds a damaged index; nor does a regular character glued before +the number, though that scan takes the digits after it for one. Neither `xref` nor `trailer` stops a search: the +stream of a classic file's last object, which no object follows, is searched through the table and the trailer to the +end of the file. + +With no `endstream` before the stop, the object's `endobj` plays its part. A declared length that ends before the stop, +and that the object's `endobj` follows as an `endstream` would, is kept: the stream lost only its `endstream`. Otherwise, +the object's own `endobj`, when only white space and comments separate it from the stop, ends the data, the end-of-line +before it left out. Both are reported as `stream.truncated`. Without one, a declared length that ends before the stop is +kept, and whether the object ends with its `endobj` is not judged; and the data otherwise runs to the stop, a declared +length past it cut there, the object judged to lack its `endobj` unless what is taken holds one. Both are reported as +`stream.length-invalid`, but data that runs to the end of the file as `stream.truncated`. + +A `/Length` that names an object one of the reader's limits cut is no length the file gave: what the cut left of it — +null, or `1` of a `15` it parted — is the guard's. An integer read before the cut is only a place to look for +`endstream` at, and the data is otherwise found by its `endstream`, as for a stream with no length; nothing is said of +the length, while a missing `endstream` is still the file's, and reported. One that names an object a guard left out of +the index is not judged either: its data is found the same way, and a missing `endstream` reported as the file's. Such a +stream may be valid, its length only hidden, and its data may spell an object header, as an embedded PDF file's does — +and the `endstream` of a stream the embedded file holds, which no search can tell from the stream's own. A header the +data spells that stops the search, with no `endobj` of the object's own before it, leaves the data up to it to the +guard, and so does an `endstream` the data spells — inside a string, say, an `endobj` after it or not —: the data is +marked cut by the guard, whose report the document holds, whether the object ends with `endobj` is not judged, and +nothing is said of the file. Only what follows the object's end — its `endobj`, or the `endstream` where none follows — +can prove an `endstream` the stream's own: the next object the index places, or a cross-reference stream the chain +read, or the end of the file, which data that goes on is never followed by, or the `xref` or `startxref` keyword, which the data may spell as well — data that spells +`endstream`, then one of them, and goes on is taken to end there, which only the hidden length could tell. The end is +then the file's, and so is an `endobj` the object lacks — but for white space or comments before the keyword, an +end-of-line above all, which only the hidden length says are the data's or not: its end is decoded as one the guard hides, and a +filter that runs out there, a Flate checksum short of a carriage return, says nothing of the file. A whole reference the cut left, `6 0 R` before the window's end, is no length +it hid: it leads on to the object it names, as read whole. + +A cross-reference stream whose `/Length` the chain cannot read yet, which a valid file may hold, is searched as well, up +to the next object the sections read so far place, or the end of the file, and its length read once the chain is. Its +rows may spell an object header and end in a carriage return: no header stops its search, and a carriage return before +the line feed that precedes `endstream` is kept. Data with no `endstream` before that stop ends nowhere the section +says, and its rows are not read — the file's fault, unless the `/Length` leads to an object one of the reader's limits +cut: the length that would have said where the data ends is then the guard's, and so is the section, the chain ending +there. Where such a `/Length` leaves the data at the first `endstream`, the rows it lacks are the guard's too, unless +what follows that `endstream` proves it the stream's own, as above: they are then the file's to lack. + +The search is not one of the reader's limits: a valid file's `endstream` follows its length, so only a damaged stream is +searched — and a cross-reference stream whose `/Length` the chain cannot read yet, and a stream whose `/Length` a guard +hid —, each once. Where it stops depends +on the file's bytes and on its index as the file wrote it, which the reader rebuilding or correcting its own index later +does not change: in the 401 documents of the project's test corpus measured, each stream takes the same length whatever +was read before it — though a damaged index can still serve another copy of an object, at another offset, according to +what was read first. A file crafted for it can still make what a stream holds depend on which object was asked for +first. What the searches of one document read together is bounded: none starts once they have read four times the file's +length. For a valid file, what they read — its cross-reference streams searched, its streams whose length a guard hid, +the white space after a declared length read on — comes to about twice the file, three times over white space that is +both the data a search takes and the run a declared length is read on through, and the proofs that an `endstream` +found under a hidden length is the stream's own, 48 or 64 bytes past each, once, and the white space they cross once more at +most, to about three times the file in all; the searches of damaged +streams read twice the file at most — 0.35 +of it at most, all searches together, in the test corpus — unless their stretches overlap: one object's header inside +another's dictionary, or a header the search does not take — a regular character glued before its number — that the +index as the file wrote it does not place. Past the bound, only the bytes the parser holds are searched: a declared +length that ends before the next object is kept, the data otherwise taken up to it or to the end of the file, and the +report says so — but for a stream whose length a guard hid, whose data up to there is the guard's, as an `endstream` +nothing proves is. ## Damaged stream data @@ -113,8 +158,8 @@ report, and reports nothing. A document's stream never decodes in silence. ## The reader's limits are not faults -Growing windows, and decoding, are bounded. The `limit.*` codes report the reader's own limits, not -faults of the file: a valid document can reach them — a large-format scan decodes past the 256 MB a stream +Growing windows, reading a stream's data, and decoding are bounded. The `limit.*` codes report the reader's own limits, +not faults of the file: a valid document can reach them — a large-format scan decodes past the 256 MB a stream may decode to by default. What fits within the limit is kept, and the limit replaces what the parser met where it stopped. [Reader limits](reader-limits.md) explains why they exist and why they are on by default. diff --git a/docs/website/docs/concepts/lazy-reading.md b/docs/website/docs/concepts/lazy-reading.md index f0e99192..80c0a565 100644 --- a/docs/website/docs/concepts/lazy-reading.md +++ b/docs/website/docs/concepts/lazy-reading.md @@ -16,8 +16,14 @@ Opening a document builds an index, finds the catalog, and stops there. 3. Builds a map from object number to a location: either a byte offset, or a position inside an object stream. Nothing is loaded meanwhile: a value a section refers to — a cross-reference stream's `/W` or `/Length` written as a reference, a `/Prev` — is read only where a section already read places it, so that - the map the chain gives is the file's own. A `/Length` no section read before places is read once the chain - is, and checked against the data the chain took up to its `endstream`. + the map the chain gives is the file's own. A `/Length` no section read before places, or that leads through + references to such an object, is read once the chain is, and checked against the data the chain took up to the + first `endstream` after the data's start, searched for in the file up to the next object the sections read so far + place, or its end — unless `MaxStreamLength` cut that data, which is the guard's; one that leads to an object `MaxObjectLength` cuts waits for the chain the same way, and is the + guard's: the guard is reached once the chain is read, and the length is not checked. Either way, when the section's + own row for that `/Length` — the row the reading of its rows keeps — lies among the rows the data holds, those + rows spelled `endstream`: the object is read where the row places it, without being kept, and the data is taken + again to the length it gives, or to the integer the guard's cut left of it, when an `endstream` confirms it. 4. Resolves the trailer's `/Root`, which reads the document catalog, and decodes the object stream that holds it when one does. When `/Root` leads to no catalog, it loads the objects the map holds, one after the other, until one is a catalog (`trailer.root-recovered`), and rebuilds the map when none is. @@ -51,16 +57,26 @@ can take 436 MB. An object the map does not hold is one the file does not define, and a reference to it reads as null, as the specification says — or one a limit left out of the map, which reads as null too, until the limit is raised: the -sections past `MaxXRefSectionCount`, the rows past `MaxXRefSectionLength` or past what `MaxDecodedStreamLength` -decodes of a cross-reference stream, the numbers past `MaxIndexEntries`. That is no damage, and the map is not rebuilt -for it. The map is rebuilt by scanning the whole file, once at most, and never while the chain is read: +sections past `MaxXRefSectionCount`, the rows past `MaxXRefSectionLength` or past what `MaxStreamLength` reads or +`MaxDecodedStreamLength` decodes of a cross-reference stream, a cross-reference stream whose `/W`, `/Index`, `/Filter` +or `/DecodeParms`, or `/Size` with no `/Index`, leads to an object `MaxObjectLength` cuts and the sections older than it — so too one whose +`/Length` leads to such an object, when no `endstream` ends its data before the next object or the end of the file, or +the rows past an `endstream` its rows spell, which only that length could tell from the end of its data —, the sections +a `/Prev` or an `/XRefStm` the reader follows that leads to such an object names — the older ones, or the cross-reference stream of a hybrid +file's revision, with the sections older than it, the chain ending with it —, the numbers past `MaxIndexEntries`. An +object the guard cut after a whole reference it holds hides none of them: the chain reads through it. That is no +damage, and the map is not rebuilt for it. The map is rebuilt by scanning the whole file, once at most, and never while +the chain is read: - at opening, when `startxref` cannot be found or read, when the section it names cannot be read, when the chain indexes nothing, or when `/Root` leads to no catalog and no object of the map is one; - when an object the map lacks is asked for, by opening or after it, while the map may have lost entries: a cross-reference section the chain names cannot be found, or a `/Prev` or `/XRefStm` is not an offset — one written as a reference no section read before it places included — (`xref.section-missing`); one is found and - cannot be read (`xref.section-unreadable`), a value its rows need written as such a reference among the causes; the chain loops back on itself (`xref.chain-cycle`); a row of a classic table cannot be read, which ends its subsection; a row gives an object + cannot be read (`xref.section-unreadable`), a value its rows need written as such a reference, or a `/Length` so + written, or that leads through references to an object no section read before places, whose data no `endstream` ends + before the next object or the end of the file, among the causes; the chain loops back on itself (`xref.chain-cycle`); + a row of a classic table cannot be read, which ends its subsection; a row gives an object in use what no entry can hold — a generation past 65,535, an offset of 2⁶³ or more, an object stream that is no object number, an index past 2,147,483,647, a field wider than 8 bytes whose leading bytes are not all zero —, which refuses that row alone, no older section's row standing for @@ -101,16 +117,44 @@ chain read for the numbers the scan did not find, rather than losing them. ## What it deliberately does not do Past the index and the catalog, nothing is parsed until something asks for it, and — short of a rebuild, which -scans the whole file — stream data is not even read from disk. A `PdfStream` holds the offset and length of its -bytes; the bytes arrive when you call `Decode()`, and not before. A test holds this on a document of about -500 KB, nearly all of it one page's content: opening it reads less than 200,000 bytes of the file, and reading -that content's bytes afterwards reads at least its 500,000 more. - -When it is decoded, a stream decodes under the [limits](reader-limits.md) of the document it came from, -however long after opening — so with `ThrowOnLimit` set, `Decode()` can throw long after `Open` returned. -What decoding meets — a stream that lost its tail, a limit reached — is reported when it is met, in the -diagnostics you pass to `Decode`, or in the document's own when you pass none: a damaged stream nobody -decodes is never reported, since nothing reads it. +scans the whole file, and of the search for where a damaged stream's data ends, below — stream data is not even read +from disk. A `PdfStream` holds the offset and length of its bytes; the bytes arrive when you call `GetRawBytes()` or +`Decode()`, and not before. A test holds this on a document of about 500 KB, nearly all of it one page's content: +opening it reads less than 200,000 bytes of the file, and reading that content's bytes afterwards reads at least its +500,000 more. + +Nor are the bytes kept once they have arrived: neither the stream nor the object cache holds them, and each call reads +them again. A document held in memory — opened from bytes or from a stream — lends them where they lie, without a copy: +what a call hands back is a view of that memory, which keeps it alive for as long as it is held, after the document is +disposed too, and changes as it does. One opened from a path reads them from the file into an array of their length. +Either is the caller's to keep or let go, so that memory follows the streams being read or decoded, not those read +before, and streams whose data share a stretch of the file — objects that overlap — do not each hold a copy of it. What +one call reads is bounded by `PdfReaderLimits.MaxStreamLength`, 256 MB by default: past it the data is cut, its filters +decode what was read as data the reader cut, and `limit.stream-length` is reported; `RawLength` stays the length the +file holds. + +Finding where a stream's data ends reads little of it. The reader parses an object through a window of the file, and +confirms the declared `/Length` by the `endstream` that must follow it, after white space and comments, however many — +a comment only once white space comes before it, since a `%` right where the declared length ends is the data's —: +past the window, it asks the file for the bytes after the declared length rather than reading the data, and reads on +as far as white space and comments run. A length nothing confirms, or a `/Length` that gives none, sends the reader on +one search for the first `endstream` after the data's start: in the bytes the parser holds, then in the file, up to the +nearer of the first object header the bytes hold and the next object the index as the file wrote it places, or the end +of the file. With no `endstream` before that stop, the object's `endobj` plays its part: the search follows the last one +through its own reads, the comments after it taken for white space, rather than read the bytes before the stop again +backward. The search is made once for each stream, however often the stream is parsed again, and whether the object's +`endobj` follows the declared length is read again only when a rebuild changes that length; no window grows to find an +`endstream`, so that a stream's data never counts toward `MaxObjectLength`; and the searches of a document read four +times the file at most — a valid file's, about three times, for the white space after a declared length, the +cross-reference streams whose `/Length` the chain cannot read yet, and the streams whose `/Length` a guard hid, with the +proof that their `endstream` is their own. What each case keeps, and reports, is in [Diagnostics](diagnostics.md#where-a-stream-ends). + +When it is read or decoded, a stream works under the [limits](reader-limits.md) of the document it came from, +however long after opening — so with `ThrowOnLimit` set, `GetRawBytes()` and `Decode()` can throw long after `Open` +returned. What decoding meets — a stream that lost its tail, a limit its filters reached — is reported when it is met, +in the diagnostics you pass to `Decode`, or in the document's own when you pass none; data cut at `MaxStreamLength` is +reported in the document's own once for each stream, as the reader reads it, and in those you pass to `Decode` each time +it decodes the cut data. A damaged stream nobody decodes is never reported, since nothing reads it. Holding a stream as an offset and a length is what will let a copy between documents — merging, assembling, stamping — move its **encoded** bytes as they are, with no decompress/recompress cycle in between, since nothing @@ -124,6 +168,10 @@ Beyond that bound, an object read again is read from the file again. That is the stays predictable, and the pathological case is slower rather than fatal. The bound is an option, and [Tune the reader's memory](../guides/tune-reader-memory.md) says when to move it. +A stream's data makes the same trade without a cache: no stream keeps it, cached or not, so that data asked for twice +is read twice, and, through `Decode()`, decoded twice. A caller that needs it more than once keeps what one call gave, +as [the same guide](../guides/tune-reader-memory.md#read-a-streams-data-once) shows. + The same options carry the [reader limits](reader-limits.md), which bound what one file may make the reader hold. diff --git a/docs/website/docs/concepts/reader-limits.md b/docs/website/docs/concepts/reader-limits.md index 4aaf1bfc..867a4922 100644 --- a/docs/website/docs/concepts/reader-limits.md +++ b/docs/website/docs/concepts/reader-limits.md @@ -37,7 +37,8 @@ it, and a file that kills the process is the worst outcome a document reader can ## Guards a valid file can reach Some of those bounds are reachable by a sound document: a large-format scan decodes past the 256 MB a stream may -decode to by default, a file saved incrementally a thousand times has a thousand cross-reference sections, and nothing +decode to by default, and stored without a filter its data would run past the 256 MB the reader reads of one stream at +once; a file saved incrementally a thousand times has a thousand cross-reference sections; and nothing in PDF 2.0 forbids more objects than the 8,388,607 the index holds by default. A bound like that is a **guard**. It is on by default, so an application that opens files it did not produce is protected without configuring anything; it is reported under its own `limit.*` code when a document reaches it; and its message names the option that lifts it, so that the document stays readable by whoever needs to read it. @@ -48,6 +49,26 @@ when it was the reader that stopped reading it. The bounds live on the options r two documents opened side by side can be read under different ones, and a stream decodes under the limits of the document it came from, however long after opening. +A value read through a reference to an object a guard cut, or left out of the index, is the guard's too. What the cut +left of it may be the start of what the file holds — `1` of `15`, `/Flate` of `/FlateDecode` —, so it is never judged +as a value the file gave; a whole reference it left, though, leads on to the object it names, as read whole. The data +of a stream whose `/Length` is so named is found by its `endstream`, and its length is not judged, though an `endstream` +the data lacks is still the file's to lack — unless an object header the data spells stops the search, or an +`endstream` it spells, which only the hidden length could tell from the end of the data: the data is then the guard's +too. An `endstream` is taken for the stream's own only where what follows its object's end is the next object the +index places, a cross-reference stream the chain read, or the end of the file, which data that goes on is never followed +by, or a cross-reference section's keyword, `xref` or `startxref`, which the data may spell as well: data that spells +`endstream`, then one of them, and goes on is taken to end there, which only the hidden length could tell. The end is +then the file's, though white space or comments before the keyword, which only the length says are the data's or not, +leave where it ends to the guard as the data decodes: a filter that runs out there says nothing of the file. Decoding stops at a filter so named, or one whose parameters are, and leaves +the rest of the data encoded, as past the most filters the reader runs; and a `/Prev` or an `/XRefStm` the reader +follows that is so named, or a cross-reference stream whose values lead to an object a guard cut, ends the chain as a +section a guard cut does, wherever near its named offset the section is found — unless what the values read say makes +the section unreadable whatever the cut hides, or data whose length the file gives, its filters written direct, lacks +the rows they declare whatever it hides: a fault of the file, as read whole, such a section read as read whole, lacking +rows. The rows such data may still hold are read with the value the guard cut, and are the guard's: the chain ends at +that section, as where a guard cut its rows, since an older section's rows would stand for them. + What a guard leaves out of the cross-reference index — the oldest sections of a long chain, the rows past a long table's bound, the objects past the most the index holds — is no damage either. Those objects read as null, unknown rather than missing, until you raise the limit; the reader does not rebuild the index for them, as it would for a file @@ -80,6 +101,15 @@ reader keeps, 32 MB together, the one it decoded last, whatever its size, and th more than its output: a Flate filter allocates up to about five times what it decodes to, and holds two to three times that while it decodes. +Before its first filter runs, a stream's encoded data is read, and that read is bounded too. ISO 32000 sets no limit on +a stream's length, and data no filter decodes — none named, an image filter, one the library does not know, one that +could not decode it — comes back as it is, out of reach of `MaxDecodedStreamLength`, which bounds what filters produce. +`MaxStreamLength` bounds what is read of one stream's data at once, 256 MB by default; past it the data is cut, and its +filters decode what was read as data the reader cut. No stream keeps its data: it is read each time it is asked for, +lent where it lies when the document is held in memory, read into an array of its length from a file. A stream costs +its data only while it is read or decoded, and for as long as you keep what that gave you, and streams whose data share +a stretch of the file do not each hold a copy of it. + `PdfReaderLimits.Unbounded` takes every limit to the most the implementation can hold. It is for documents your own application produced, never for uploads: under it, a file can make the reader hold whatever it asks for, up to those maxima. @@ -90,7 +120,16 @@ Other bounds protect the reader itself — its stack, the chains of references i documents producers write: real documents nest a handful of levels, not 128, and producers write an object rather than a reference to a reference. An object stream's header lists its objects by increasing offset, so that a sound one's objects each lie before the next one starts: the reader reads each no further, and a header that breaks that -order ends where it breaks it. Those stay internal constants, since lifting them would let nothing more be read, and +order ends where it breaks it. A sound stream's `endstream` follows its declared length; where it does not, or the +`/Length` gives none, the reader searches the file for it past the bytes it parsed the object through, rather than parse +the object through a larger window, so that a stream's data — and the end-of-line after its `stream` keyword — never +counts toward its object's length. A document's +searches read four times the file at most: a valid file is read past those bytes only for the data of a cross-reference +stream whose `/Length` the chain cannot read yet, for the data of a stream whose `/Length` a guard cut or left out of the +index, and for white space and comments after a declared length, about twice the file — three times over the white +space a cross-reference stream's search takes as data that its length, once read, is read on through —, and for the +proof that an `endstream` found under a hidden `/Length` is the stream's own, about three times the file in all. Those stay internal +constants, since lifting them would let nothing more be read, and each says where it is declared why no valid file reaches it. A new bound is classified when it is added ([ADR 34](/project/adr/every-valid-pdf-is-readable-and-the-readers-guards-are)): diff --git a/docs/website/docs/concepts/validation.md b/docs/website/docs/concepts/validation.md index c99fef21..0a06c88f 100644 --- a/docs/website/docs/concepts/validation.md +++ b/docs/website/docs/concepts/validation.md @@ -154,8 +154,17 @@ before the first rule —, one rule also looks at every object it holds, to name The document's [reader limits](reader-limits.md) apply to validation too. An object the reader cut at a limit is not a fault of the file, so a rule meeting one judges nothing of what it lacks or holds, or says, as information, that it could not check it whole, rather than calling the file broken; opening the document with a raised -limit lets it check the rest. The page tree rules take a node so cut to give what its pages inherit, and leave its -`/Count`, the value of those above it and the pages the tree may leave out unjudged; the reader takes a catalog so cut for the -catalog, as it read it, rather than look for another, and does not look for one either when a limit left the object -`/Root` names out of the index. Validation -throws only for the caller's own mistakes, and for a limit when the document was opened to throw on one. +limit lets it check the rest. The same holds of a value that leads to such an object: a cross-reference stream whose +`/W`, `/Index`, `/Filter` or `/DecodeParms`, or `/Size` with no `/Index`, leads to one is said not checked rather than +malformed — so is one whose `/Length`, which the chain could not read, leads to one where only that length could say +where its data ends —, unless what the values read say makes it unreadable whatever the guard hid, or data whose length the file gives, its +filters written direct, lacks the rows they declare whatever it hid — the file's, such a section read as read whole, +lacking rows, the rows it may still hold said not checked, nor the `/Size` they bear on; +a `/Prev` or an `/XRefStm` the reader follows that leads to one ends the chain there rather than naming a section not found; and an +object stream whose `/Filter` or `/DecodeParms` leads to one, or to an object a limit left out of the index, is left +encoded, the objects it holds said not checked rather than lost +([#263](https://github.com/AdCodicem/AdCodicem.Pdf/issues/263)). The page tree rules take a node so cut to give what its +pages inherit, and leave its `/Count`, the value of those above it and the pages the tree may leave out unjudged; the +reader takes a catalog so cut for the catalog, as it read it — one such an object stream holds included —, rather than +look for another, and does not look for one either when a limit left the object `/Root` names out of the index. +Validation throws only for the caller's own mistakes, and for a limit when the document was opened to throw on one. diff --git a/docs/website/docs/guides/find-what-the-reader-repaired.md b/docs/website/docs/guides/find-what-the-reader-repaired.md index b988002a..75b6c464 100644 --- a/docs/website/docs/guides/find-what-the-reader-repaired.md +++ b/docs/website/docs/guides/find-what-the-reader-repaired.md @@ -93,6 +93,10 @@ if (document.GetObject(new PdfObjectId(12)) is PdfStream stream) } ``` +The list also receives, each time you decode the stream, `limit.stream-length` when its data runs past what the reader +reads of one stream; the document's `Diagnostics` hold that report too, once, since the reader makes it as it reads the +data. + ## Decide whether to accept the file Diagnostics say what the reader did, not whether the file is sound. To decide whether to accept it, validate it too: diff --git a/docs/website/docs/guides/handle-reader-limits.md b/docs/website/docs/guides/handle-reader-limits.md index 8f9d836d..878377fd 100644 --- a/docs/website/docs/guides/handle-reader-limits.md +++ b/docs/website/docs/guides/handle-reader-limits.md @@ -29,9 +29,9 @@ foreach (var entry in document.Diagnostics) } ``` -Reading is lazy: a stream reaches `MaxDecodedStreamLength` when it is decoded, so look after decoding, not only after -opening. And an object a limit left out of the cross-reference index — the oldest sections of a long chain, the objects -past `MaxIndexEntries` — reads as null, as an object the file lacks does: the `limit.*` entry is what tells them apart. +Reading is lazy: a stream reaches `MaxStreamLength` when its data is read, and `MaxDecodedStreamLength` when it is +decoded, so look after reading streams, not only after opening. And an object a limit left out of the cross-reference +index — the oldest sections of a long chain, the objects past `MaxIndexEntries` — reads as null, as an object the file lacks does: the `limit.*` entry is what tells them apart. ## Raise the limit the report names @@ -50,6 +50,17 @@ using var document = PdfDocument.Open("topographic-map.pdf", options); Every stream of that document decodes under its limits, however long after opening you decode it. +The two bounds on a stream are not interchangeable. A large image stored without a filter reaches `MaxStreamLength`, +the bound on what is read of one stream's data, and not `MaxDecodedStreamLength`, which bounds what filters produce. +Raise `MaxStreamLength` for it: + +```csharp +var options = new PdfReaderOptions +{ + Limits = PdfReaderLimits.Default with { MaxStreamLength = 512 * 1024 * 1024 }, +}; +``` + ## Read documents you trust without limits For documents your own application produced, lift every limit at once: @@ -88,7 +99,8 @@ catch (PdfLimitExceededException exception) Put the `try` around everything that uses the document, not only around `Open`: the exception comes from whichever operation reaches the limit — `Open` while it indexes the file, `GetObject` or resolving a reference when an object is -too long or lies past what a full index holds, `Decode` when a stream decodes too far, `Validate` as it reads. +too long or lies past what a full index holds, `GetRawBytes` or `Decode` when a stream's data runs past +`MaxStreamLength`, `Decode` when a stream decodes too far, `Validate` as it reads. ## See also diff --git a/docs/website/docs/guides/tune-reader-memory.md b/docs/website/docs/guides/tune-reader-memory.md index 41114b1b..e7a551c9 100644 --- a/docs/website/docs/guides/tune-reader-memory.md +++ b/docs/website/docs/guides/tune-reader-memory.md @@ -1,6 +1,6 @@ --- title: Tune the reader's memory -description: Trade memory for fewer re-reads with the object cache, keep more diagnostics, and open documents without copying them. +description: Trade memory for fewer re-reads with the object cache, read a stream's data once, keep more diagnostics, and open documents without copying them. --- # Tune the reader's memory @@ -25,7 +25,34 @@ var options = new PdfReaderOptions using var document = PdfDocument.Open("catalog.pdf", options); ``` -Measure before and after: the cache only pays off when the same objects are read again. +Measure before and after: the cache only pays off when the same objects are read again. A stream in the cache holds +its dictionary and where its data lies, not the data: its size does not weigh on the cache. + +## Read a stream's data once + +A stream keeps none of its data: each call to `GetRawBytes` or `Decode` reads it again. Opened from a path, each call +reads the data from the file into an array of its length; opened from a stream or from bytes, which the document holds +in memory, `GetRawBytes` lends the bytes where they lie, without a copy, and so does `Decode` with data no filter +decodes. What is lent is a view of the document's memory: for as long as you hold it, it keeps that memory alive, after +`Dispose` too, and it changes if that memory does. Copy it before you reuse a buffer the document read in place. + +When you use the same data more than once, keep what one call gave you rather than call again: + +```csharp +using AdCodicem.Pdf.Objects; + +if (document.GetObject(new PdfObjectId(12)) is PdfStream stream) +{ + var data = stream.Decode(); // read and decoded once + + Index(data); + Render(data); +} +``` + +What one call reads is bounded by `PdfReaderLimits.MaxStreamLength`, 256 MB by default: past it the data is cut, and +`limit.stream-length` reported. [Read or refuse a document that reaches a limit](handle-reader-limits.md) shows how to +raise it. ## Keep more diagnostics diff --git a/docs/website/docs/reference/diagnostics.md b/docs/website/docs/reference/diagnostics.md index 69c87fde..96c74df6 100644 --- a/docs/website/docs/reference/diagnostics.md +++ b/docs/website/docs/reference/diagnostics.md @@ -74,11 +74,11 @@ The list `PdfDocument.Diagnostics` holds, in the order the entries were recorded | `Decode(stream, diagnostics)` | Where the reports go | |---|---| -| `diagnostics` given | Into it, and nowhere else | +| `diagnostics` given | Into it, each time the stream is decoded; the document's own `Diagnostics` also hold, once, the `limit.stream-length` the reader reports as it reads data `MaxStreamLength` cuts. What loading a value the filters read meets — the `/Filter` entry, a filter's parameters — is the document's alone, a guard that cut it included, which ends decoding there | | none, a stream read from a document | The document's own `Diagnostics` | | none, a stream built in memory | Nowhere: nothing is reported | -A stream that is never decoded is never reported: nothing reads it. +A stream that is never decoded reports no fault of its decoding: nothing decodes it. ## Severities @@ -101,18 +101,18 @@ Codes are stable: they are part of the public contract, because callers filter o | `header.missing` | The file has no `%PDF-` header in its first 4,096 bytes; it was read all the same, its offsets counted from its first byte | | `xref.rebuilt` | The index was rebuilt by scanning the whole file | | `xref.offset-adjusted` | An object, or a cross-reference section the chain names, was not where the file said, and was found nearby | -| `xref.section-missing` | A cross-reference section `/Prev` or `/XRefStm` names holds nothing that reads as a section, nor does any place nearby, lies outside the file, or is named by a value that is not an offset; what only it indexed is found by rebuilding the index when it is asked for. Reported at the offset the section was named at, or, when that offset lies outside the file or the value is not an offset, at the section whose trailer named it | -| `xref.section-unreadable` | A cross-reference section `/Prev` or `/XRefStm` names is there and cannot be read — a stray token among a table's rows or in place of its trailer, a subsection numbering objects past 2,147,483,647, a stream whose `/W`, `/Index`, `/Size` or data cannot give rows, or whose `/W`, `/Index`, `/Size`, `/Filter` or `/DecodeParms` is a reference no section read before it places where it can be read without loading it — and nothing nearby can be read in its place; the rows read before the fault are kept, and what only the rest indexed is found by rebuilding the index when it is asked for. Reported at the section, the fault in the message | +| `xref.section-missing` | A cross-reference section `/Prev` or `/XRefStm` names holds nothing that reads as a section, nor does any place nearby, lies outside the file, or is named by a value that is not an offset; what only it indexed is found by rebuilding the index when it is asked for. Reported at the offset the section was named at, or, when that offset lies outside the file or the value is not an offset, at the section whose trailer named it. A `/Prev` or `/XRefStm` the reader follows — not a hybrid file's cross-reference stream's own `/Prev`, nor the links of a table an `/XRefStm` names, which it leaves unread —, written as a reference that leads to an object one of the reader's limits cut, names what the guard hides: no section is reported missing, nothing is rebuilt, the chain ends there, and the guard is reported once the chain is read | +| `xref.section-unreadable` | A cross-reference section `/Prev` or `/XRefStm` names is there and cannot be read — a stray token among a table's rows or in place of its trailer, a subsection numbering objects past 2,147,483,647, a stream whose `/W`, `/Index`, data or — with no `/Index` — `/Size` cannot give rows, or whose `/W`, `/Index`, `/Filter` or `/DecodeParms`, or `/Size` with no `/Index`, is a reference no section read before it places where it can be read without loading it, or whose `/Length` is such a reference, or leads to one through others, and no `endstream` follows its data before the next object the sections read so far place, or before the end of the file, or none was looked for once the document's searches for `endstream` read as much of the file as they may — and nothing nearby can be read in its place; the rows read before the fault are kept, and what only the rest indexed is found by rebuilding the index when it is asked for. Reported at the section, the fault in the message. A value that names an object one of the reader's limits cut, directly or through a chain of references — a `/Length` whose data no `endstream` ends before that stop among them —, is the guard's, not the file's: the section is not reported, nor looked for nearby, the chain ends there, nothing is rebuilt, what only that section and the older ones index reads as null, and the guard is reported once the chain is read — but for what makes the section unreadable whatever the value holds, which is the file's and reported here, as read whole: a value past it no section read so far places, a `/W` that gives no three widths or none past 0, or rows numbered wrongly as far as the values read say — a subsection wrong by the element read whatever the cut one holds among them —; data whose length the file gives, its filters written direct, decoded whole, that holds fewer rows than the counts read declare however few bytes the widths read leave a row is read as read whole instead, lacking rows, its trailer merged and the chain going on, where it was moved to too | | `xref.chain-cycle` | The chain of previous sections looped: a `/Prev` or `/XRefStm` names an offset the chain has already named, whether it read a section there or not. Reported at that offset, or, when it lies outside the file, at the section whose trailer named it | | `xref.entry-out-of-range` | An object's entry places it outside the file — past its end, before its start, or past what a `long` holds once the header's offset is added —; reported with no position, the offset in the message, exactly. A section named outside the file is `xref.section-missing`'s, or the rebuild's when `startxref` names it | -| `stream.length-invalid` | A stream's `/Length` is not where its data ends: its `endstream` lies elsewhere and ends the data, or the `/Length` is no length — absent, not a non-negative integer, or naming an object the file lacks or that could not be read — and the `endstream` ends the data, or no `endstream` follows the declared length — none before the next object or the end of the file, or none looked for once the document's searches read as much as they may —, and that length is kept. A cross-reference stream whose `/Length` the chain could not read, its data taken up to its first `endstream`, has it read once the chain is: a `/Length` that is no length, or that no `endstream` confirms, is reported as any stream's, and one an `endstream` past the one the chain stopped at confirms is reported with the rows the chain left unread | -| `stream.truncated` | A stream has no `endstream` before the end of the file, or before the `endobj` that follows its data, and its data runs to the end of the file | +| `stream.length-invalid` | A stream's `/Length` is not where its data ends: its `endstream` lies elsewhere and ends the data; or the `/Length` is no length — absent, not a non-negative integer, or naming an object the file lacks or that could not be read — and the `endstream` ends the data; or no `endstream` lies before the next object or the end of the file, no `endobj` of its object ends the data, and the data is the declared length, when it ends before that stop, or runs up to the next object; or the document's searches for `endstream` have read as much of the file as they may, and the data is taken without a search — but where a guard hid the `/Length`, whose data is then the guard's. A `/Length` that names an object a guard cut, or left out of the index, is not judged, though a missing `endstream` is reported all the same. [Streams whose length is wrong](#streams-whose-length-is-wrong), below, gives each case. A cross-reference stream whose `/Length` the chain could not read, its data taken up to its first `endstream`, has it read once the chain is, unless a guard cut that data before every row the section declares — data that holds them all is checked whatever a guard did —: a `/Length` that is no length, or that no `endstream` confirms, is reported as any stream's, and one an `endstream` past the one the chain stopped at confirms is reported with the rows the chain left unread. Where the section's own row for that `/Length` lies among the rows the data holds — in a predicted row the spelled `endstream` parts too, the data decoded once more knowing its end hidden —, and gives a length an `endstream` confirms, the data is taken again to it before the rows are read, and nothing is reported; where that decoding reaches `MaxDecodedStreamLength` short of the row, the rows past the bound are the guard's, reported under `limit.decoded-stream`; nor is anything of a `/Length` a guard cut or left out of the index | +| `stream.truncated` | A stream has no `endstream` before the next object or the end of the file, and its object's `endobj` ends its data — following the declared length, which is kept, or ending the object before that stop —; or neither, no declared length ends before the end of the file, and its data runs to it. Data taken without a search, the document's searches having read as much of the file as they may, is `stream.length-invalid`'s, to the end of the file too. [Streams whose length is wrong](#streams-whose-length-is-wrong), below, gives each case | | `stream.self-reference` | An object stream's dictionary names an object the stream holds — as its `/Type`, `/Length`, `/N`, `/First`, a filter or a parameter —: that object reads as null while the stream is decoded, and the stream is decoded without it | | `object-stream.member-moved` | An object is in the object stream its entry names, at another index than the entry gives, and was read where the stream's header lists it. Reported where the stream's data starts, the object, the stream and both indexes in the message | -| `object-stream.unreadable` | What an object stream — a stream an entry of the index names as one — says of itself cannot be believed: it is not of `/Type /ObjStm`, its `/N` is absent or no non-negative integer, or, when it declares objects, its `/First` is, or lies past its decoded data, or its header ends or breaks before listing as many objects as `/N` declares — a pair that is not two integers, that gives a number no object can have or an offset no member can start at, whose offset is not past the one before it, or whose member would start at or past the end of the decoded data, unless a guard cut it. The header is read as far as it goes, whatever its `/N` claims: the objects listed before the fault are read, the others cannot be read from it. Reported where the stream's data starts, the stream and its fault in the message | +| `object-stream.unreadable` | What an object stream — a stream an entry of the index names as one — says of itself cannot be believed: it is not of `/Type /ObjStm`, its `/N` is absent or no non-negative integer, or, when it declares objects, its `/First` is, or lies past its decoded data, or its header ends or breaks before listing as many objects as `/N` declares — a pair that is not two integers, that gives a number no object can have or an offset no member can start at, whose offset is not past the one before it, or whose member would start at or past the end of the decoded data, unless a guard cut it. The header is read as far as it goes, whatever its `/N` claims: the objects listed before the fault are read, the others cannot be read from it. Reported where the stream's data starts, the stream and its fault in the message. An object stream whose `/Filter`, `/DecodeParms` or a value of a step's parameters names an object a guard cut, or left out of the index, is left encoded and not reported: its objects are the guard's, unknown until it is lifted | | `object-stream.member-stream` | An object an object stream holds is written as a stream, or holds one, which ISO 32000-1 (7.5.7) forbids: it reads as null, and nothing of it is decoded. Reported once for each object and each definition of its object stream, where the stream's data starts, the object, the stream and the byte of its decoded data where the object starts in the message | | `syntax.unexpected-token` | A token stood where the syntax does not allow it: where a value or a dictionary key was expected, after a key that has no value, or closing an array with a dictionary's end; it was read as null, skipped, or it ended what it stood in | -| `syntax.truncated-object` | An object ended before it was whole: the file, or an object stream's decoded data, ended in the middle of it — a value missing, or an array, a dictionary or a string it opened never closed —, or an `endobj` stood where a value or a key should be, inside an array or a dictionary it opened. What was read is kept: a string takes the bytes to the end of the data, and an `endobj` ends every container it finds open — the object's own, or, when the object lost it too, the next object's, the container then taking that object. Reported once, for the innermost construct left open, where it opens — where the value would start, when none is open; past the depth the reader follows, the container it skipped, which holds what it did not follow —, the constructs around it counted in the message; inside an object an object stream holds, where the stream's data starts, the member and the byte in the message. A member is read to the next member's start, the last to the end of the decoded data: running into either, it is cut short there, and the message says which. The edge of a window the reader grows is no end of the data, and a cut a guard made is the guard's — a member read past `MaxObjectLength` is `limit.object`'s | +| `syntax.truncated-object` | An object ended before it was whole: the file, or an object stream's decoded data, ended in the middle of it — a value missing, or an array, a dictionary or a string it opened never closed —, or an `endobj`, or the next object's header, stood where an element, a key or a value should be, inside an array or a dictionary it opened. What was read is kept: a string takes the bytes to the end of the data, and an `endobj` ends every container it finds open, as does the next object's header, which an object that lost its `endobj` too meets first, and which leaves the object without its `endobj`. A header is `N G obj` as the search for a stream's `endstream` stops at one: digits alone, N from 1 to 2,147,483,647 and G from 0 to 65,535, white space alone between them — no comment —, and `obj` a whole keyword; tokens that read as one inside a container that closes after them end it there all the same, and the object's `endobj` is judged missing. An object stream's member, which no header follows, and a value standing alone, which is no container, never end at one. A trailer, which is no object, ends at one all the same, and the message says the header came inside what it left open: `An object header came inside a dictionary, which was never closed.` Reported once, for the innermost construct left open, where it opens — where the value would start, when none is open; past the depth the reader follows, the container it skipped, which holds what it did not follow —, the constructs around it counted in the message; inside an object an object stream holds, where the stream's data starts, the member and the byte in the message. A member is read to the next member's start, the last to the end of the decoded data: running into either, it is cut short there, and the message says which. The edge of a window the reader grows is no end of the data, and a cut a guard made is the guard's — a member read past `MaxObjectLength` is `limit.object`'s | | `syntax.depth-exceeded` | Nesting went deeper than the reader will follow | | `syntax.number-out-of-range` | A number is beyond what a real can hold — its magnitude rounds past the largest a double holds, about 1.8 × 10³⁰⁸ —: it was read as null, so a dictionary holds no entry for its key, and an array holds a null in its place. Reported where the number starts — where its object stream's data starts, for a number inside an object one holds —, quoted in the message. An integer past the range of a `long` is no fault of its own: it reads as the real nearest to it | | `syntax.hex-string-invalid` | A hexadecimal string holds bytes that are neither hexadecimal digits nor white space, the only bytes ISO 32000-1 (7.3.4.3) lets it hold — a `%` among them, which is no comment there —: they were skipped, as pdf.js and pdfium skip them, where PDFBox stops at the first, MuPDF and poppler read each as the digit 0, and qpdf and pypdf lose the string. Reported once for each string, where its first such byte lies, how many there are in the message; inside an object an object stream holds, where the stream's data starts, the member and the byte in the message | @@ -122,16 +122,17 @@ Codes are stable: they are part of the public contract, because callers filter o | `trailer.root-recovered` | The trailer's `/Root` does not lead to a document catalog, and the catalog was found among the file's objects — those its index holds when the index is sound, a rebuilt index's otherwise | | `filter.failed` | A filter's data is damaged: Flate data of which nothing decoded before the byte the fault lies in is left as the filters before it decoded it, and the filters after it are not tried; otherwise what decoded is kept, up to that byte or to the end — nothing, for LZW data whose first code is undefined, and the filters after it decode that —, and a repair when nothing was lost. ASCII85, ASCIIHex, RunLength and LZW data that breaks its format — a byte that encodes nothing, an end marker missing or out of place — is decoded as other readers decode it (*Damaged stream data*, below) | | `filter.checksum-mismatch` | A Flate stream decoded to its end, but its zlib checksum disagrees with what it decoded to: all of it was kept, and some of it may be wrong | -| `filter.unsupported` | The file names a filter the library does not implement, or an element of `/Filter` names no filter at all, or `/Filter` is neither a name nor an array; decoding stopped there, with what the filters before it decoded. A `/Filter` that is null is one the stream does not have, and is not reported | +| `filter.unsupported` | The file names a filter the library does not implement, or an element of `/Filter` names no filter at all, or `/Filter` is neither a name nor an array; decoding stopped there, with what the filters before it decoded. A `/Filter` that is null is one the stream does not have, and is not reported. Nor is a `/Filter`, an element of it, a `/DecodeParms` or a value of a step's parameters that names an object a guard cut, or left out of the index — directly or through a chain of references —: what the cut left of it, such as a name it parted, is the guard's, and decoding ends there, as it does at the element past `MaxFilterChainLength`, with what the filters before it decoded; nothing more is reported, the guard having been reported where it was reached | | `limit.decoded-stream` | A filter decodes to more than `MaxDecodedStreamLength`, or the filters before the last pass along more than it together; decoding stopped there, and the stream comes back as the filters after the cut decode what was kept within the bound | +| `limit.stream-length` | A stream's encoded data, as the file holds it, is longer than `MaxStreamLength`, which bounds what is read of it at once: the data within the bound was read, decoded as data the reader cut — what the filters lose past the cut is no damage — or handed back as it is where decoding hands data back undecoded, as with no filter, an image filter or a filter the library does not know. `Data.Length` and `RawLength` stay the length the file holds. Reported where the stream's data starts: once in the document's diagnostics, when its data is first read — by `GetRawBytes` or by decoding —, and in diagnostics of your own passed to `Decode` with each decoding, as `limit.decoded-stream` is | | `limit.object` | An object is longer than `MaxObjectLength`, its stream data aside; the part within it was parsed. An object an object stream holds is bounded alike, and reported once for each definition of its object stream, where that stream's data starts, the object, the stream and the byte of its decoded data where the object starts in the message | | `limit.xref-section-length` | A classic cross-reference table is longer than `MaxXRefSectionLength`; the entries within it were read, and its trailer past them; the objects the rows past them place read as null, and the chain is not followed further | | `limit.xref-section-count` | The chain of cross-reference sections is longer than `MaxXRefSectionCount`; the newest were read, and what only the older ones index reads as null. Reported where the chain goes on, or, when that lies outside the file, at the section whose `/Prev` names it, the offset in the message | -| `limit.trailer` | A trailer, or a cross-reference stream's dictionary, is longer than `MaxTrailerLength`; the part within it was parsed, and the chain ends there: an older section, an `/XRefStm` or a `/Root` past the cut is unknown, what only the older sections index reading as null | +| `limit.trailer` | A trailer, or a cross-reference stream's dictionary — never that stream's data —, is longer than `MaxTrailerLength`; the part within it was parsed, and the chain ends there: an older section, an `/XRefStm` or a `/Root` past the cut is unknown, what only the older sections index reading as null. Where the chain names a section and what lies there declares another type before the cut — a CIDFont, whose `/W` gives widths —, the guard cut no section: it is reported all the same, as the dictionary of the object read as a cross-reference stream, and the section is looked for nearby, the chain going on, as for a CIDFont's dictionary read whole, which no stream data follows | | `limit.index-entries` | The index holds `MaxIndexEntries` object numbers — those in use, free or refused — and met another: the numbers it holds were indexed, and an object past them reads as null. Reported once a document, where the first number it had no room for was met: the section that gave it, the object header or object stream a rebuild read, or the chain's newest section for an entry the chain gave that a rebuild could not keep | | `limit.filter-chain` | A stream's `/Filter` array names more filters than `MaxFilterChainLength`, `/Crypt` among them; decoding stopped before the first past them, with what the filters within the bound decoded, and the elements past them were not read. A cross-reference stream or an object stream it ends is read as cut before its data: its rows or its objects are unknown | -The seven `limit.*` codes report the reader's own limits, not faults of the file; each message names the property of +The eight `limit.*` codes report the reader's own limits, not faults of the file; each message names the property of `PdfReaderLimits` that lifts it. They are listed with their defaults in [Reader limits](reader-limits.md). ### Faults of the syntax @@ -146,6 +147,7 @@ start, so no fault is two objects'. ```text Warning syntax.truncated-object at 328: The file ended inside a literal string, which takes the 6,575 bytes from where it opens to that end; the 2 arrays or dictionaries around it were never closed either. Warning syntax.truncated-object at 381: An endobj ended the object inside an array, which was never closed. +Warning syntax.truncated-object at 507: An object header ended the object inside a dictionary, which was never closed, before the value of its last key. Warning syntax.hex-string-invalid at 1204: A hexadecimal string holds 5 bytes that are neither hexadecimal digits nor white space, the first here; they were skipped. Warning syntax.name-escape-invalid at 96: A name holds a number sign that two hexadecimal digits do not follow, kept as the byte it is: the name reads as /A#23zz. Warning syntax.key-repeated at 412: The dictionary gives the key /Filter more than once, written here with #xx escapes; the last value given is kept. @@ -153,17 +155,51 @@ Warning syntax.key-repeated at 412: The dictionary gives the key /Filter more th ### Streams whose length is wrong -A stream whose declared length the file cannot hold is reported as `stream.truncated` when the file ends inside its -data, and as `stream.length-invalid` when its `endstream` comes first. Each stream is reported once, however often it -is parsed again — after the cache let it go, or the index was rebuilt. How the reader finds where such a stream ends -is explained in [Diagnostics](../concepts/diagnostics.md#where-a-stream-ends). +A declared length is confirmed by an `endstream` after it, white space and comments between, however many — a comment +only once white space comes before it: a `%` right where the declared length ends is the data's, and leaves the length +unconfirmed. Any other stream — its length unconfirmed, or none to confirm — is searched once for the first `endstream` +after the start of its data, up to the next object: the first object header the file's bytes hold after that start, or +the next object the file's index places, whichever is nearer. Each case is reported where the data starts, and each +stream once, however often it is parsed again — after the cache let it go, or the index was rebuilt. How the reader +finds where such a stream ends, and why, is explained in [Diagnostics](../concepts/diagnostics.md#where-a-stream-ends). + +| What follows the data | Code | The data taken | +|---|---|---| +| An `endstream` after the declared length | None | The declared length | +| No such `endstream`, or no length to confirm; an `endstream` before the next object or the end of the file | `stream.length-invalid` | Up to that `endstream`, the end-of-line before it left out | +| No `endstream` before that stop; the object's `endobj` after the declared length, which ends before the stop, white space and comments between, as before an `endstream` | `stream.truncated` | The declared length | +| No `endstream` before the stop; the object's own `endobj` ends the object there, only white space and comments between them | `stream.truncated` | Up to that `endobj`, the end-of-line before it left out | +| None of the above; a declared length that ends before the stop | `stream.length-invalid` | The declared length; whether the object ends with its `endobj` is not judged | +| None of the above; no declared length that ends before the stop | `stream.length-invalid` at the next object, `stream.truncated` at the end of the file | Up to the stop; the object lacks its `endobj`, unless what is taken holds one | +| Not searched: the document's searches for `endstream` have read as much of the file as they may, and the bytes read with the object hold neither an `endstream` nor an object header | `stream.length-invalid`, the message saying so | The declared length, when it ends before the stop; otherwise up to the stop, the file past those bytes not searched for a header | + +A file that ends on the `stream` keyword, or on the carriage return after it, holds a stream whose data starts where the +file ends and holds nothing: its missing `endstream` is reported as after a line feed, as the table gives it. + +A `/Length` that names an object one of the reader's limits cut — directly or through a chain of references — gives no +length the file wrote: what the cut left of it, null or an integer it parted, is the guard's, which reported it. An +integer read before the cut is only a place to look for `endstream` at — the data ends there when one follows it —, +never kept as a length; the data is otherwise found by its `endstream`, as for a stream with no length, and nothing is +said of the length. A missing `endstream` is still the file's, and is reported, the message naming the object the guard +cut. A `/Length` that names an object a guard left out of the index, directly or through a chain of references, is not +judged either: the data is found by its `endstream` the same way, and a missing one is the file's, the message naming the +object the guard left out — `The stream's /Length names object 21 0, which one of the reader's limits left out of the +index, and no endstream follows its data`, then where the search stopped and what was taken. For either, an object +header the data spells, as an embedded PDF file's does, that stops the search — no `endobj` of the object's own before +it —, or an `endstream` the data spells, an `endobj` after it or not, leaves the data up to it to the guard: nothing is +reported of the stream, whose data is cut by the guard, and whether its object ends with `endobj` is not judged. An +`endstream` is taken for the stream's own, its end the file's, only where what follows its object's end — its +`endobj`, or the `endstream` where none follows — is the next object the index places, a cross-reference stream the +chain read, the end of the file, or the `xref` or `startxref` keyword. White space or comments before the keyword, an +end-of-line above all, which only the hidden length says are the data's or not, leave where the data ends to the guard +as it decodes, as [Damaged stream data](#damaged-stream-data) says. ```text Warning stream.length-invalid at 5501: The stream declared 19954 bytes but ended after 19952. Warning stream.length-invalid at 19901349: The stream declared 202154 bytes, and no endstream follows them before the next object, at 20103524; the declared length is kept. Warning stream.length-invalid at 316: The stream's /Length is a real number, 61.5, not a non-negative integer; its data ends after 68 bytes. Warning stream.length-invalid at 542577: The stream's /Length names object 18 0, which holds a stream, not a non-negative integer; its data ends after 5555 bytes. -Warning stream.truncated at 315: The stream has no endstream before the endobj that follows its data; the 310 bytes to the end of the file are taken as its data. +Warning stream.truncated at 315: The stream has no endstream: its object's endobj follows the 68 bytes it declared, which are taken as its data. ``` ### Object streams @@ -203,6 +239,8 @@ Warning syntax.unexpected-token at 125: A token was found where a value was expe | Flate, nothing decodes before the fault | `filter.failed`, Warning | The data as the filters before it left it — still encoded, when it is the first —, and the filters after it are not tried | | A filter decodes past `MaxDecodedStreamLength`, or the filters before the last pass along more than it together | `limit.decoded-stream`, Warning | What the filters after the cut decode of the data up to the bound | | Names more filters than `MaxFilterChainLength` | `limit.filter-chain`, Warning | The data as the filters within the bound decoded it | +| Data longer than `MaxStreamLength` | `limit.stream-length`, Warning | What the filters decode of the data up to the bound, as data the reader cut; that data as it is, when no filter decodes it | +| A `/Filter`, an element of it, a `/DecodeParms` or a value of a step's parameters names an object a guard cut, or left out of the index | None of its own: the guard's, reported where it was reached | The data as the filters before that value decoded it | The byte counts in these reports are the reporting filter's own. The bytes kept are counted before any predictor (`/DecodeParms` `/Predictor`) is applied, so `Decode` may return another length — a PNG predictor takes a byte off each @@ -214,11 +252,20 @@ diagnostics hold what decoding a stream met once, however often the stream is de or the document's, by the reader as it serves an object stream's objects again after its budget let them go or as it rebuilds the index, or by the validator —, as long as they have room for it; diagnostics you pass of your own hold every decoding's reports. A stream the reader itself cut at one of its limits is reported as that limit, not as a lost tail; a -fault in the part it did read is the file's, and is reported. That holds whoever cut it: the window the parser reads an -object through, or a filter before this one that stopped at `MaxDecodedStreamLength`. Of data the reader cut, no lost -tail, missing checksum or missing end marker is reported, a hexadecimal digit without its pair or an ASCII85 group short -of five is dropped rather than completed, and a predictor undoes the part of its last row the data holds rather than -report rows the parameters describe as longer than the data. +fault in the part it did read is the file's, and is reported. That holds whoever cut it: the reader, reading the stream's +data within `MaxStreamLength`, or its object within `MaxObjectLength`, or a filter before this one that stopped at +`MaxDecodedStreamLength`. Of data the reader cut, no lost tail, missing checksum or missing end marker is reported, a +hexadecimal digit without its pair or an ASCII85 group short of five is dropped rather than completed, and a predictor +undoes the part of its last row the data holds rather than report rows the parameters describe as longer than the data. +Nor is the end of data whose end a guard hides — its `endstream` proven its own past white space or comments that only +a `/Length` a guard cut or left out of the index says are the data's or not — reported. Its first filter decodes it up +to where that run starts — a comment past white space within the data, but in ASCII85 data, where a `%` is a digit —: +what it meets before is the file's and reported, its running out there — a lost tail, a checksum short, a missing end +marker — is not, and where its output may lack what the run holds — Flate that lost its tail, LZW or RunLength data +without its end marker — the output's end is cut, as above, for the predictor, the filters after it and an object +stream's members. Flate data whose deflate body ends in the data as taken decodes whole, what it reports as read whole +but a checksum the run cuts short, or that disagrees only where it reaches into the run. The filters after a first whose output is whole +decode, and report, as read whole. ```text Warning filter.failed at 59534: A Flate stream ends before its data does; what decoded before the end was kept. @@ -227,6 +274,7 @@ Warning filter.failed at 2102: A Flate stream is corrupt at byte 889 of its 2062 Warning filter.checksum-mismatch at 3278: A Flate stream's checksum disagrees with the 174803 bytes its data decoded to; all were kept, and some may be wrong. Warning filter.failed at 4336336: An ASCII85 stream ends without its ~> end marker, after a group of a single digit, which encodes no byte; it was dropped, and data may have been lost. Warning filter.failed at 812: An ASCIIHex stream holds a byte that encodes nothing, at byte 5 of its 12; it was passed over. +Warning limit.stream-length at 9318: A stream's data runs past 256 MB; only what lies within it was read. Raise PdfReaderLimits.MaxStreamLength to read past it. ``` ## Exceptions diff --git a/docs/website/docs/reference/object-model.md b/docs/website/docs/reference/object-model.md index b839650e..50494c13 100644 --- a/docs/website/docs/reference/object-model.md +++ b/docs/website/docs/reference/object-model.md @@ -21,7 +21,7 @@ references they hold. Every one derives from `PdfObject`. How a document reads t | `PdfString` | A string's bytes, text or binary | `Bytes`; `ToText()` for its text | Its text between double quotes, escaped: `"Annual report"` | | `PdfArray` | An ordered list of objects | Its items, by index | `[array of 3]` | | `PdfDictionary` | Entries, each keyed by a name | Its entries, by key | `<>`, or `<>` when its `/Type` is no name | -| `PdfStream` | A dictionary and its encoded data | `Dictionary`, `GetRawBytes()`; `Decode()` for the decoded data | `<>`, by its `/Subtype`, else its `/Type` | +| `PdfStream` | A dictionary and its encoded data, which a stream read from a document [reads each time it is asked for](#a-streams-data) | `Dictionary`, `GetRawBytes()`; `Decode()` for the decoded data | `<>`, by its `/Subtype`, else its `/Type` | | `PdfReference` | A reference to an object, such as `12 0 R` | `Id`; `Resolve()` for the object | `12 0 R` | ## What `ToString()` gives @@ -84,9 +84,38 @@ writes past what a `long` holds is read as a real, and so described with its poi The longest description is 327 characters, that of -4.9E-324, the negative of the smallest positive double. +## A stream's data + +A `PdfStream` holds its dictionary and a `PdfStreamData`. For a stream read from a document, that says where the +encoded bytes lie and how many there are, not what they are: they are read each time something asks for them, and kept +by nothing — neither the stream nor the document's object cache —, so that a stream costs its data only while it is +read or decoded, and for as long as the caller keeps what that gave. + +| Member | What it gives | What it costs | +|---|---|---| +| `Dictionary` | The stream's dictionary | Nothing: it was read with the stream | +| `RawLength`, `Data.Length` | The count of the stream's encoded bytes in the file: its `/Length` where an `endstream` confirms it, otherwise as far as the reader takes the data to run ([Where a stream ends](../concepts/diagnostics.md#where-a-stream-ends)). Data longer than `MaxStreamLength` is counted whole | Nothing: found when the stream is parsed | +| `Data.Position` | The offset in the file the data starts at | Nothing | +| `GetRawBytes()`, `Data.GetBytes()` | The encoded bytes, as the file holds them, up to `MaxStreamLength`: past it, the bytes within the bound, and `limit.stream-length` reported once for the stream, in the document's diagnostics | A read, each call: from a document opened from bytes or from a stream, the bytes lent where they lie, without a copy — a view of that memory, which keeps it alive for as long as you hold it, and changes as it does; from one opened from a path, read from the file into an array of their length | +| `Decode()` | The data the stream's filters decode, as far as they go: decoding stops at an image filter, at a filter the library does not know, and where the [reader limits](reader-limits.md) end it — a `/Filter`, a `/DecodeParms` or a parameter that leads to an object a limit cut or left out of the index among them. Data cut at `MaxStreamLength` is decoded as far as it was read, and `limit.stream-length` reported in the diagnostics you pass, each time | A read, as `GetRawBytes()` makes, then what each filter produces; data no filter decodes comes back as it was read | +| `HasImageFilter()` | Whether `Decode()` hands back an encoded image: whether `/Filter` names an image filter before anything that ends decoding whatever the data holds — a filter the library does not know, an element that names no filter, a filter past `MaxFilterChainLength`, a value or parameter of the filters' that leads to an object a limit cut or left out of the index | None of the data: what decoding reads before the image filter — the filters' names, and the parameters of a Flate or LZW step before it —, which can load an object, and decode the object stream that holds it, as decoding would | + +Opened with `ThrowOnLimit`, a document's streams throw a `PdfLimitExceededException` from `GetRawBytes()` and +`Decode()` instead, each time their data runs past `MaxStreamLength`; `Decode()` throws at the other limits it reaches +too, and `HasImageFilter()` when loading a value the filters read reaches one. No call's result is kept for the next: +a caller that uses the data twice keeps what one call gave, rather than call again +([Tune the reader's memory](../guides/tune-reader-memory.md#read-a-streams-data-once)). + +Data built with `PdfStreamData.FromMemory` is held by the stream, whatever its length, and handed back as it was given; +its `Position` is -1. + ## Objects from a disposed document An object keeps what it holds once its document is disposed: a dictionary's direct values, an array's items, a stream's `RawLength`, every object's `ToString()`. What reaches back into the document throws `ObjectDisposedException`, however the document was opened and whether or not it was read before: `Resolve()` on a reference, and every accessor that -resolves one, such as `GetDictionary`; `GetRawBytes()` and `Decode()` on a stream read from the file. +resolves one, such as `GetDictionary`; `GetRawBytes()` and `Decode()` on a stream read from the file, and +`HasImageFilter()` when it has a value to load. Raw bytes taken before from a document held in memory stay a view of +that memory, and so does what `Decode()` gave back when no filter changed the data — with no filter, or where decoding +stopped at the first: an image filter, one the library does not know or could not decode the data with, the end of the +chain —: they keep it alive for as long as they are held, and change as it does. diff --git a/docs/website/docs/reference/reader-limits.md b/docs/website/docs/reference/reader-limits.md index 4322e8da..97ccbd1b 100644 --- a/docs/website/docs/reference/reader-limits.md +++ b/docs/website/docs/reference/reader-limits.md @@ -14,10 +14,11 @@ document instead, in [Read or refuse a document that reaches a limit](../guides/ | Property of `PdfReaderLimits` | Default | Reached by | What is kept | Code | |---|---|---|---|---| | `MaxDecodedStreamLength` | 256 MB | A decompression bomb, or a large-format scan: a 9,600 × 11,410 RGB map decodes to 313 MB. Of a chain of filters, the filters before the last share the bound for what they pass along together, and the last has its own; a `/Crypt` filter, which passes its data along as it is, is neither charged nor the last | The first bytes, up to the bound | `limit.decoded-stream` | +| `MaxStreamLength` | 256 MB | A stream read from a file whose encoded data is that long, read in one piece each time `GetRawBytes` or decoding asks for it: that map's image stored without a filter would be 313 MB. It bounds what is read, where `MaxDecodedStreamLength` bounds what filters produce: data decoding hands back as it is — no filter, an image filter, a filter the library does not know, one that could not decode it — is bounded by it alone | The data's first bytes, up to the bound; `Data.Length` and `RawLength` stay the length the file holds | `limit.stream-length` | | `MaxObjectLength` | 16 MB | An object that long, the data of a stream aside, written in the file or held by an object stream: an array of a million references | The object as far as the bound | `limit.object` | | `MaxXRefSectionLength` | 64 MB | A classic cross-reference table of more than about 3.3 million entries | The entries within the bound | `limit.xref-section-length` | | `MaxXRefSectionCount` | 1,024 | A document saved incrementally more than a thousand times | The newest sections | `limit.xref-section-count` | -| `MaxTrailerLength` | 64 KB | A trailer that long, or a cross-reference stream's dictionary: its `/Index` grows with every scattered update | The trailer as far as the bound | `limit.trailer` | +| `MaxTrailerLength` | 64 KB | A trailer that long, or a cross-reference stream's dictionary — its `/Index` grows with every scattered update —, never that stream's data | The trailer as far as the bound | `limit.trailer` | | `MaxIndexEntries` | 8,388,607 | A document of more objects than ISO 32000-1's Annex C advises; or a cross-reference stream of one-byte rows, of which 256 MB decode from a few hundred bytes | The object numbers indexed first: the newest sections', a rebuild's first | `limit.index-entries` | | `MaxFilterChainLength` | 64 | A `/Filter` array naming more filters, `/Crypt` among them: real streams name one to three, and a hostile array of millions costs decoding a few microseconds a filter, and some filters a table of tens of kilobytes, whatever their data | The data as the filters within the bound decoded it; the elements past it are not read | `limit.filter-chain` | @@ -32,7 +33,8 @@ bytes allocated an object, so a document near the bound takes about 24 s and 3 G validates uploads can lower it to the documents it accepts. A value of zero or less is refused with an `ArgumentOutOfRangeException`. A length above `Array.MaxLength` -is taken as `Array.MaxLength`: a decoded stream is held in one piece, and no runtime allocates a larger one. +is taken as `Array.MaxLength`: a decoded stream, and a stream's data read at once, are held in one piece, and no +runtime allocates a larger one. `PdfReaderOptions.Limits` refuses null with an `ArgumentNullException`. ## When a limit is reached @@ -48,7 +50,8 @@ there. Raise PdfReaderLimits.MaxDecodedStreamLength to read past it. - An object, a cross-reference section or a trailer is reported once, however often it is read again — an object an object stream holds, once for each object and each definition of its object stream —; a full index, once a document. - What a limit leaves out of the index — the sections past `MaxXRefSectionCount`, the rows past - `MaxXRefSectionLength` or past what `MaxDecodedStreamLength` decodes of a cross-reference stream, the object + `MaxXRefSectionLength`, or past what `MaxStreamLength` reads or `MaxDecodedStreamLength` decodes of a + cross-reference stream, the object numbers past `MaxIndexEntries` — reads as null until you raise the limit. The reader does not rebuild the index for it, since the file may be sound, and the validator reports it as unknown, under `xref.checked-in-part`, rather than missing. When the trailer's `/Root` names an object so left out, the catalog reads as null too: the reader does not @@ -62,6 +65,19 @@ there. Raise PdfReaderLimits.MaxDecodedStreamLength to read past it. object so cut lacks or holds, and takes a page tree node so cut to give what its pages inherit. An object header that runs past the bound, where a cross-reference entry places it, is where a read reaches the limit, and the validator says that entry was not checked. +- A stream whose data runs past `MaxStreamLength` is read up to the bound each time its data is asked for, by + `GetRawBytes` or by decoding, and its filters decode what was read as data the reader cut: that it runs out is not + reported as damage. Data that decoding hands back as it is comes back as far as the bound. The guard is reported at the + offset the stream's data starts at: once in the document's own diagnostics, as the reader reads the data, and, as the + bound on decoded data is, in the diagnostics you pass to `Decode` each time it decodes the stream. `GetRawBytes` takes + no diagnostics: the document's hold its report. With `ThrowOnLimit` each read throws. In an object stream so cut, an + object that lies past the cut is the guard's, unknown until you raise the limit, as what any limit leaves out is: + + ```text + Warning limit.stream-length at 1904: A stream's data runs past 256 MB; only what lies within it was read. Raise + PdfReaderLimits.MaxStreamLength to read past it. + ``` + - A stream decoded past the bound is reported by the first of its filters that stopped there: each time it is decoded, in the diagnostics you pass to `Decode`; once, in the document's own — when you pass none, or the document's —, however often you, the reader or the validator decode it. The guard is counted each time all the same, and with `ThrowOnLimit` @@ -90,12 +106,71 @@ there. Raise PdfReaderLimits.MaxDecodedStreamLength to read past it. cross-reference stream or an object stream the reader meets so is read as one a guard cut before its data: its rows, or its objects, are unknown until you raise the limit, as what any limit leaves out is. +- A value a stream is read with that leads to an object a guard cut, or left out of the index — its `/Length`, + `/Filter` or `/DecodeParms`, a value of a filter's parameters, directly or through a chain of references — is the + guard's, and nothing is reported of it but the guard. The data of a stream whose `/Length` is so named is found by + the search for its `endstream`, and its length is not judged: an integer the cut left, which may be the first digits + of a longer one, is only a place to look for `endstream` at. No `endstream` after the data of a stream whose + `/Length` names an object a guard cut, or left out of the index, is still the file's fault, and reported; the report + names the object at the end of the chain of references. The search may stop short of it, though, at an object header + the data spells, as an embedded PDF file's does, or at an `endstream` the data spells, as a string may, an `endobj` + after it or not: the length that tells those from the end of the data is the guard's. An `endstream` ends data that + is the file's only where what follows its object's end — its `endobj`, or the `endstream` itself where none follows — + proves it the stream's own: the next object the index places, a cross-reference stream the chain read, or the end of + the file, which no data that goes on is followed by, or the `xref` or `startxref` keyword — which data may spell too: data that spells `endstream`, then + `xref` or `startxref`, and goes on is taken to end there as the file's, which only the hidden length could tell; white + space before the keyword, which only the length says is the data's or not, leaves the data's end to the guard as it + decodes, so that a filter that runs out there says nothing of the file. Elsewhere the data up to there comes back cut by the guard, + `CutByGuard`, whether the object ends with `endobj` is not judged, and nothing is said of the file. A reference the cut + left whole, the cut falling after `6 0 R`, hides nothing: it is read through to the + object it names, as it is read whole, while the chain is read as afterward. Decoding stops at a filter so named, or + at the filter whose + parameters are: the filters before it run, and the stream comes back as they decoded it, still encoded by the rest, + as past `MaxFilterChainLength`. `HasImageFilter` reads what decoding reads before the image filter, the parameters of + a Flate or LZW filter that comes before it included, and says false when such a value ends the chain there; under + `ThrowOnLimit`, loading one throws where decoding would. An object stream the reader meets so + is read as one a guard cut before its data: its objects are unknown until you raise the limit, and when the catalog + is among them the reader does not look for another. While the chain is read, a `/Prev` or an `/XRefStm` the reader + follows that leads to an object `MaxObjectLength` cuts — not a hybrid file's cross-reference stream's own `/Prev`, + nor the links of a table an `/XRefStm` names in a stream's place, which it leaves unread —, a cross-reference stream + whose `/W`, `/Index`, `/Filter` or `/DecodeParms` does, or whose `/Size` does with no `/Index`, and one whose + `/Length` does when no `endstream` ends its data, are the guard's, as a section it cut is: the chain ends there, the + sections it names are not read — the older ones, or a hybrid file's cross-reference stream —, the index is not + rebuilt, and the guard is reported once the chain is read, or thrown by `Open`. So does a section found near the + offset it is named at, which such a value keeps from being read, or which declared itself one — a table's `xref` + keyword, `/Type /XRef` — before a guard cut it; a place there that declares another type — a CIDFont, whose `/W` gives + widths —, or whose dictionary a guard cut before it said what it is, is not taken for the section, and the next is + tried. A guard that cut, where the chain names a section, the dictionary of what declares another type cut no section — + a CIDFont's, which read whole no stream data follows —, while one that cut a value of a stream there leaves it the + section the guard cut, whatever type it gives, as read whole it is the section. What makes a section unreadable whatever + a value a guard cut holds is the file's fault, as read whole: a value past the cut that no section read so far places, a + `/W` that gives no three widths or none past 0, rows numbered wrongly — an `/Index` that holds no pairs, a + subsection whose elements are no integers, whose first number or count is below 0, or that numbers past the object + numbers, by both its elements or by the one read whatever the cut one holds, a `/Size` with no `/Index` that gives no + count —, as far as + the values read, written direct or through references, say. Data whose length the file gives, its filters written + direct, decoded whole, that holds fewer rows than the counts read declare however few bytes the widths read leave a + row lacks them whatever the cut hides: the section is read as read whole, lacking rows, its trailer merged and the + index rebuilt, and the object the guard cut is loaded once the chain is read, which reaches the guard. The rows the + data may hold at the fewest bytes a row may take, read with the cut value, are the guard's: the chain ends at the + section, as where rows a guard cut, and the validator says it unchecked; the chain goes on only past data that holds + none. The rows a cross-reference stream's data lacks, + where its `/Length` so leads and its data was taken to the first `endstream`, are the guard's too, unless what follows + that `endstream` proves it the stream's own, as above: they are then the file's to lack, and the index is rebuilt. An + object the chain read through a reference the cut left whole is loaded once the chain is read only for a value it + keeps, or for what the offset a section is named at holds, which is read as a load reads it, its reports kept: not + for a place near there it guessed the section was moved to and dropped as none. - A stream read from a document decodes under **that document's** limits, however long after opening you decode it. A stream you build in memory decodes under `PdfReaderLimits.Default`. - A classic trailer that ends within the window its cross-reference table was read through is read whole, whatever - its length. `MaxTrailerLength` bounds how far the reader follows one past that window. + its length. `MaxTrailerLength` bounds how far the reader follows one past that window, and a cross-reference stream's + dictionary, not its data: the data of one whose `/Length` the reader cannot read while it reads the chain is searched + for its `endstream` in the file, up to the next object the sections read so far place, or the end of the file. Data + no `endstream` ends before there makes its section malformed, a fault of the file and not of a guard — unless its + `/Length` leads to an object `MaxObjectLength` cuts, which makes the section the guard's, as above. - Until the reader decodes streams a piece at a time, a stream that decodes past about 2 GB is cut there, and - reported, even under `Unbounded`. + reported, even under `Unbounded`. A stream's data is read in one piece, and data past about 2 GB is read up to + there, and reported under `limit.stream-length`, even under `Unbounded`. - The reader keeps the object streams it decoded within a budget of 32 MB — their decoded data, and their headers' tables —, and always keeps the one it decoded last, whatever its size. Raising `MaxDecodedStreamLength` raises how large that one can be. @@ -117,8 +192,10 @@ The exception comes from whichever operation reaches the limit: | Operation | When | |---|---| | `PdfDocument.Open` | While it indexes the file and reads its catalog | -| `GetObject`, or resolving a reference | When an object is too long, or the object stream that holds it decodes too far or names too many filters; when the read rebuilds the index, at any limit the rebuild reaches, once the rebuild is finished — the first reached: a limit the rebuild's scan reaches before the read's own, the read's own before one finishing the rebuild reaches; and each time it looks up an object a full index left out | -| `Decode` | When a stream decodes too far, or its decoding reaches the filter past `MaxFilterChainLength` | +| `GetObject`, or resolving a reference | When an object is too long, or the object stream that holds it runs past `MaxStreamLength`, decodes too far or names too many filters; when the read rebuilds the index, at any limit the rebuild reaches, once the rebuild is finished — the first reached: a limit the rebuild's scan reaches before the read's own, the read's own before one finishing the rebuild reaches; and each time it looks up an object a full index left out | +| `GetRawBytes` | When the stream's data runs past `MaxStreamLength`, each time it is read | +| `Decode` | When a stream's data runs past `MaxStreamLength`, the stream decodes too far, its decoding reaches the filter past `MaxFilterChainLength`, or loading a value the filters read — the `/Filter` entry, a filter's parameters — reaches a limit, as resolving a reference does | +| `HasImageFilter` | When loading a value decoding reads before the image filter — the `/Filter` entry, a filter's parameters — reaches a limit, as resolving a reference does | | `PdfValidator.Validate` | As it reads, like any other caller; and before its first rule, at any limit the rebuild it may run then reaches, once that rebuild is finished | A document whose index had to be rebuilt finishes the rebuild before throwing, so it stays usable afterwards. @@ -132,6 +209,7 @@ the documents producers write. They are not options: |---|---|---| | Nesting of arrays and dictionaries | 128 levels | Real documents nest a handful of levels; the bound keeps a hostile file from exhausting the stack | | A reference that resolves to another reference | followed 32 times | Producers write the object itself; the bound breaks a chain that loops | +| Searches for a stream's `endstream` | four times the file's length, for a document's searches together | A valid stream's `endstream` follows its declared length: the file is read past the bytes an object is parsed through only for the data of a cross-reference stream whose `/Length` the chain cannot read, for the data of a stream whose `/Length` a guard cut or left out of the index, and for white space and comments after a declared length that run past a first read — about twice the file, three times over the white space a cross-reference stream's search takes as data that its length, once read, is read on through —, and the proof that an `endstream` found under a hidden `/Length` is the stream's own reads 48 or 64 bytes past each, once, and the white space it crosses once more at most, about three times the file in all. Past the bound, a stream whose `endstream` the bytes read with its object do not show is not searched for further, and its report says so — or, where a guard hid its `/Length`, its data up to the next object is the guard's, `CutByGuard`, nothing reported | | Objects loaded while loading another | 64 | An indirect `/Length` leads to an integer, and a stream resolves its filters and the parameters they read before decoding any of its data: no level holds decoded data while the next loads | | Object numbers | up to 2,147,483,647 | An object's number is an `int`, and ISO 32000-1's Annex C advises at most 8,388,607 indirect objects. A number past it is refused where it is read — a header, a reference, a subsection, a row, an object stream's header —, never narrowed to one that would name another object. An object is numbered from 1; 0 names the head of the free list, where a reference, a subsection or a row may give it | | An object stream's members | each read to the next one's start, the last to the end of the decoded data | ISO 32000-1 (7.5.7) has a header list its objects by increasing offset, so that each lies before the next one starts: a pair whose offset is not past the one before it, or whose object would start past the data, ends the header there, as a stray token does, and the objects listed before it are read. A member past `MaxObjectLength` is that limit's | diff --git a/docs/website/docs/reference/validation-rules.md b/docs/website/docs/reference/validation-rules.md index 0226ba58..281c9baf 100644 --- a/docs/website/docs/reference/validation-rules.md +++ b/docs/website/docs/reference/validation-rules.md @@ -43,27 +43,27 @@ References are to ISO 32000-1:2008. | `file.startxref-wrong` | Error | The offset `startxref` gives holds no cross-reference section: the index is out of reach, and is rebuilt by scanning. White space before the section is `xref.offset-imprecise`'s; a section that is there and cannot be read is `xref.section-malformed`'s. | The first section of the chain | | `file.trailer-missing` | Error | A cross-reference table's rows are not followed by the `trailer` keyword: they run into a dictionary, or the file ends. qpdf, as the reader, then rebuilds the index. | Each classic table of the chain | | `file.trailer-malformed` | Error | A trailer is not a well-formed dictionary: the `trailer` keyword is not followed by one, or the reader read it despite syntax errors — a key that is not a name, a dictionary or a string left unclosed, a hexadecimal string holding bytes that are neither digits nor white space, a name whose number sign is no escape, a key given twice —, so an entry may have been lost with the syntax, or read otherwise than its writer meant. A cross-reference stream's dictionary is held to the same. | Each trailer of the chain, as the reader parsed it | -| `file.root-invalid` | Error | The trailer's `/Root` is missing or does not lead to a document catalog (Table 15), a value that is not a reference included — unless it is a catalog written in the trailer itself, which the reader takes as it is ([#111]). The reader then looks for an object of `/Type /Catalog` among those the index holds — and rebuilds the index only when none is one —: the catalog it finds is its choice, not the file's. Not judged when the trailer could not be read at all, nor when the index was rebuilt as the document opened, nor when one of the reader's limits left the catalog out of the index as the document opened, which `xref.checked-in-part` says, or cut the object `/Root` names — through a chain of references too, and in an object stream whose cut leaves it unread, past the cut data or in a stream the guard made unreadable ([#259]) — or the trailer a catalog written there was read from, which the reader then takes for the catalog as it read it, looking for no other, and its `limit.*` report names ([#175]) — whatever a rebuild found since —, nor a `/Root` the trailers lack where one of the reader's limits kept a trailer from being read whole — cut, or older than where the chain stopped, at the section count or at a section whose rows a limit cut or filled the index with —: it may lie there, and the reader rebuilds nothing for it. Rows a limit cut that run on to no trailer lack one: the file's, which a rebuild answers. The catalog the finding names is the one the reader took last, in the rebuild run before the first rule if there was one. | The trailer the chain gave, merged as readers merge it, and `/Root` before the reader looked for a catalog | -| `file.size-wrong` | Warning | A section's `/Size` is missing, or is not one more than the highest object number it and the sections it updates use (Tables 15 and 17, 7.5.6, 7.5.8.4). A hybrid file's cross-reference stream meets one of Table 17's two readings or the other — one more than the highest number it and the sections it updates use, or the `/Size` of the table that names it —, and either is accepted. A `/Size` that leaves in-use objects numbered above it is `xref.object-past-size`'s. A `/Size` written as a reference, or as a real, is said to be. Not judged when the chain lost a section, or one of the reader's limits cut it. | Each section's trailer, and the highest number each section indexes | -| `file.trailer-value-wrong` | Warning | A value a section's trailer gives the reader to read the chain is written as the specification does not allow, and read all the same, as other readers read it: a reference where it must be direct, read where a section read before it places the object; a real with no fractional part where it must be an integer, read as the integer it equals ([#182], [#215]). A cross-reference stream's `/Type`, `/Index`, `/Prev` and `/W`, the elements of `/Index` and `/W`, and its `/Filter` and `/DecodeParms` are direct (7.5.8.2); in a file that declares PDF 2.0, so are its `/Length` and the elements and entries of its `/Filter` and `/DecodeParms` (an erratum of ISO 32000-2, pdf-issues #246). A classic trailer's `/Prev` is direct (Table 15); an `/XRefStm` may be indirect (Table 19). `/W`, `/Index`, `/Prev`, `/XRefStm`, a stream's `/Length` and its `/Predictor`, `/Colors`, `/BitsPerComponent`, `/Columns` and `/EarlyChange` are integers. One finding per value, which says both faults of a reference that leads to a real. A value the rows are read with that the reader could not read so is the section's fault — `xref.section-malformed`, `xref.section-not-found` —, and `/Size` is `file.size-wrong`'s; a `/Type`, or a `/Length` read once the chain is, is reported here whether or not it could be read. | Each section's trailer, as the file wrote it, and the objects its references name, through the index the reader settled on before the first rule | +| `file.root-invalid` | Error | The trailer's `/Root` is missing or does not lead to a document catalog (Table 15), a value that is not a reference included — unless it is a catalog written in the trailer itself, which the reader takes as it is ([#111]). The reader then looks for an object of `/Type /Catalog` among those the index holds — and rebuilds the index only when none is one —: the catalog it finds is its choice, not the file's. Not judged when the trailer could not be read at all, nor when the index was rebuilt as the document opened, nor when one of the reader's limits left the catalog out of the index as the document opened, which `xref.checked-in-part` says, or cut the object `/Root` names — through a chain of references too, and in an object stream whose cut leaves it unread, past the cut data or in a stream the guard made unreadable ([#259]), or in one whose filters end at a `/Filter`, a `/DecodeParms` or a parameter that leads to an object a guard cut or left out of the index, which leaves its data encoded ([#263]) — or the trailer a catalog written there was read from, which the reader then takes for the catalog as it read it, looking for no other, and its `limit.*` report names ([#175]) — whatever a rebuild found since —, nor a `/Root` the trailers lack where one of the reader's limits kept a trailer from being read whole — cut, or older than where the chain stopped, at the section count or at a section whose rows a limit cut or filled the index with —: it may lie there, and the reader rebuilds nothing for it. Rows a limit cut that run on to no trailer lack one: the file's, which a rebuild answers. The catalog the finding names is the one the reader took last, in the rebuild run before the first rule if there was one. | The trailer the chain gave, merged as readers merge it, and `/Root` before the reader looked for a catalog | +| `file.size-wrong` | Warning | A section's `/Size` is missing, or is not one more than the highest object number it and the sections it updates use (Tables 15 and 17, 7.5.6, 7.5.8.4). A hybrid file's cross-reference stream meets one of Table 17's two readings or the other — one more than the highest number it and the sections it updates use, or the `/Size` of the table that names it —, and either is accepted. A `/Size` that leaves in-use objects numbered above it is `xref.object-past-size`'s. A `/Size` written as a reference, or as a real, is said to be. Not judged when the chain lost a section, or one of the reader's limits cut it or ended it before the sections a `/Prev` or an `/XRefStm` names ([#263]). | Each section's trailer, and the highest number each section indexes | +| `file.trailer-value-wrong` | Warning | A value a section's trailer gives the reader to read the chain is written as the specification does not allow, and read all the same, as other readers read it: a reference where it must be direct, read where a section read before it places the object; a real with no fractional part where it must be an integer, read as the integer it equals ([#182], [#215]). A cross-reference stream's `/Type`, `/Index`, `/Prev` and `/W`, the elements of `/Index` and `/W`, and its `/Filter` and `/DecodeParms` are direct (7.5.8.2); in a file that declares PDF 2.0, so are its `/Length` and the elements and entries of its `/Filter` and `/DecodeParms` (an erratum of ISO 32000-2, pdf-issues #246). A classic trailer's `/Prev` is direct (Table 15); an `/XRefStm` may be indirect (Table 19). A value the reader does not read is judged as the file wrote it, a reference it holds not followed, which would read what the reading never did: a hybrid file's cross-reference stream's own `/Prev`, which the chain does not follow, going on through the table's; the `/Prev` and `/XRefStm` of a table an `/XRefStm` names in a stream's place; the elements of a `/W` past the third, the widths of three fields being all the reader reads; and what of a cross-reference stream's `/Filter` and `/DecodeParms` decoding does not read — the parameters of a filter that reads none, `/Colors`, `/BitsPerComponent` and `/Columns` where no predictor is, `/EarlyChange` but for LZW, or what lies past the element that ends the chain, under the reader's `MaxFilterChainLength` ([#263]). What decoding reads is judged as it read it, whatever the object cache let go since. `/W`, `/Index`, `/Prev`, `/XRefStm`, a stream's `/Length` and its `/Predictor`, `/Colors`, `/BitsPerComponent`, `/Columns` and `/EarlyChange` are integers. One finding per value, which says both faults of a reference that leads to a real where the reader follows it. A value the chain is read with that the reader could not read so is the section's fault — `xref.section-malformed`, `xref.section-not-found` —, unless it leads to an object one of the reader's limits cut, where the chain ends for the limit, a section whose rows need it left to `xref.checked-in-part` ([#263]), and `/Size` is `file.size-wrong`'s; a `/Type`, or a `/Length` read once the chain is, is reported here whether or not it could be read. | Each section's trailer, as the file wrote it, and the objects the references the reader follows name, through the index the reader settled on before the first rule | ### Cross-references | Rule | Severity | Meaning | Reads | |---|---|---|---| -| `xref.section-malformed` | Error | A cross-reference section the chain names is there — an `xref` keyword, or a cross-reference stream — and cannot be read, or holds fewer rows than it declares: a subsection header that is not two integers or that numbers objects past 2,147,483,647, a row that is not an offset, a generation and `n` or `f`, a `/W` that is not three field widths that are not negative and not all zero, an `/Index` that is not pairs numbering objects from 0 to 2,147,483,647, a `/Size` that is no count of objects where there is no `/Index`, or a value the rows are read with — `/W`, `/Index`, `/Filter`, `/DecodeParms` and the parameters a Flate or LZW step reads, a `/Size` where there is no `/Index` — written as a reference no section read before it places where it can be read without loading it, or as references that lead from one to the next to no value ([#182]). A row that gives an object in use what no entry can hold — a generation past 65,535, an offset of 2⁶³ or more, an object stream that is no object number, an index past 2,147,483,647, or a field wider than 8 bytes whose leading bytes are not all zero — is refused alone, and makes the section malformed too; a free row is read whatever its generation, and a row of a type past 2 is skipped ([#221]). The objects it leaves out are found, if at all, by scanning the file. A section one of the reader's limits cut is not malformed for the cut (ADR 34), but what the reader read of it is the file's: a row it refused before the cut, what it met past the rows reading on to a table's trailer. | Each section of the chain | -| `xref.section-not-found` | Error | A section a `/Prev` or an `/XRefStm` names is neither there nor within 512 bytes of it, or what names it is not an offset — tiff2pdf writes `/Prev 576066 0 R`. The objects only it indexes are missing from the index. The section `startxref` names is `file.startxref-wrong`'s. | Each section of the chain | +| `xref.section-malformed` | Error | A cross-reference section the chain names is there — an `xref` keyword, or a cross-reference stream — and cannot be read, or holds fewer rows than it declares: a subsection header that is not two integers or that numbers objects past 2,147,483,647, a row that is not an offset, a generation and `n` or `f`, a `/W` that is not three field widths that are not negative and not all zero, an `/Index` that is not pairs numbering objects from 0 to 2,147,483,647, a `/Size` that is no count of objects where there is no `/Index`, or a value the rows are read with — `/W`, `/Index`, `/Filter`, `/DecodeParms` and the parameters a Flate or LZW step reads, a `/Size` where there is no `/Index` — written as a reference no section read before it places where it can be read without loading it, or as references that lead from one to the next to no value ([#182]); or a cross-reference stream whose `/Length` is such a reference, or leads through references to an object no section read before it places, and whose data no `endstream` ends before the next object the sections read so far place, or the end of the file — or was not searched for one, the document's searches for `endstream` having read as much of the file as they may ([#220]). Rows that spell `endstream` end such a stream's data there, and the section lacks the rows past them, unless the section's own row for that `/Length` — the row the reading of its rows keeps, one of a reserved type ignored —, among the rows before them, places an object that gives a longer length an `endstream` confirms — in a predicted row the spelled one parts too, undone as far as it goes —: the data is then taken to it. A row that gives an object in use what no entry can hold — a generation past 65,535, an offset of 2⁶³ or more, an object stream that is no object number, an index past 2,147,483,647, or a field wider than 8 bytes whose leading bytes are not all zero — is refused alone, and makes the section malformed too; a free row is read whatever its generation, and a row of a type past 2 is skipped ([#221]). The objects it leaves out are found, if at all, by scanning the file. A section one of the reader's limits cut is not malformed for the cut (ADR 34), but what the reader read of it is the file's: a row it refused before the cut, what it met past the rows reading on to a table's trailer. Nor is a cross-reference stream whose value the rows are read with leads to an object a limit cut: the section counts as cut there, the chain ends at it, nothing is rebuilt, and `xref.checked-in-part` says so ([#263]) — but for what makes it unreadable whatever the value holds, as read whole: a value past it no section read so far places, a `/W` that gives no three widths or none past 0, or rows numbered wrongly as far as the values read say — a subsection wrong by the element read whatever the cut one holds among them —, or data whose length the file gives, its filters written direct, decoded whole, that holds fewer rows than the counts read declare however few bytes the widths read leave a row — read then as read whole, lacking rows —, whatever type the stream gives. Nor is one whose `/Length`, which the chain could not read, leads to an object `MaxObjectLength` cuts, where only that length could say where the data ends: with no `endstream` before the next object or the end of the file, the section counts as cut, as above; when its rows spell one, the integer the cut left of that object — whether the section's own row places it, or a newer section does — is where an `endstream` is looked for, the data taken to it when one confirms it, and otherwise the rows past the spelled one the limit's, the section counting as cut there, as above ([#224]). | Each section of the chain | +| `xref.section-not-found` | Error | A section a `/Prev` or an `/XRefStm` names is neither there nor within 512 bytes of it, or what names it is not an offset — tiff2pdf writes `/Prev 576066 0 R`. The objects only it indexes are missing from the index. The section `startxref` names is `file.startxref-wrong`'s. A `/Prev` or an `/XRefStm` the reader follows, written as a reference that leads to an object `MaxObjectLength` cuts, names no section missing: the chain ends there for the limit, what it names unread, which `xref.checked-in-part` says ([#263]). | Each section of the chain | | `xref.section-shifted` | Warning | A section a `/Prev` or an `/XRefStm` names starts a few bytes from where it is named, and the reader found it there. | Each section of the chain | | `xref.chain-loop` | Error | The chain names a section it has already read, or an offset it has already named where it read none, in the file or outside it: the reader stops there, and the section the chain should have gone on to is unknown, what only it indexed found, if at all, by rebuilding the index — as for `xref.section-not-found`. | The chain | | `xref.entry-broken` | Error | An in-use entry places its object neither at its offset nor within 512 bytes of it, in an object stream whose header does not list it, or in one where it is written as a stream or holds one, which an object stream cannot hold (7.5.7), and which reads as null — the index the reader rebuilt included, where it places such an object and the file's own index places nothing, or the part of it the reader read where one of its limits stopped the chain short ([#161]) — as a load decides it: the header read at the offset after white space and comments, however long, a whole token no header begins with ruling one out at once, and the one found near it confirmed by what a load parses there ([#175], [#256]). An offset outside the file — past its end, before its start, or past what a `long` holds once the header's offset is added — is looked for nearby as a load looks: an object found there is `xref.entry-shifted`'s. One finding per entry, located at the object and the offset the entry gives, or at the object alone when that offset lies outside the file. | Each object's header, as a load reads it, a few dozen bytes in a sound file; each object stream's header, and each member whose bytes hold `stream`, parsed as a load parses it | | `xref.entry-shifted` | Warning | An in-use entry places its object a few bytes from where it is — the object's own header confirming it —, or at another index of its object stream than the one its header gives. One finding per entry, located at the object alone when the entry's offset lies outside the file, a load having found the object near it ([#175]). | As `xref.entry-broken` | | `xref.generation-mismatch` | Warning | An in-use entry gives a generation other than the one its object is written with. The references and the object's header agree against the entry, and the reader reads the object; qpdf reads a reference to it as a reference to nothing. | Each object's header | | `xref.object-stream-circular` | Error | An object stream needs, to be read, an object only reading it can give: its `/Length`, `/Type`, `/Filter`, `/DecodeParms`, `/N` or `/First` names an object it holds — directly, through objects written in the file, or through another object stream whose own keys need it in turn (7.5.7) —, a name or a number through a chain of references alone, as far as a reference resolves — a `/Length` one object further, which the parse reads before it resolves it —, and a filter and its parameters whole, every element and entry of their arrays and dictionaries and of the objects they name — more than decoding may read, and nothing of a parameter that is a stream, which [#277] will align with what decoding reads. The reader resolves a stream's `/Length` as it parses it, then its `/Type`, and nothing else of a stream not of `/Type /ObjStm` ([#274]): of such a stream, its `/Type` alone is followed, a loop through it being why the stream is not taken for an object stream; a loop through its `/Length` leaves its objects lost to its type all the same, `xref.object-stream-broken`'s. The reader decodes the stream without that value, as qpdf reports a loop, and cannot vouch for what it decoded ([#51]). One finding per stream; `xref.object-stream-broken` says nothing more of it. | Each object stream's dictionary and the objects its decoding keys name; nothing is decoded | -| `xref.object-stream-broken` | Error | An object stream the index places objects in is not in the index, is not an object stream — not of `/Type /ObjStm`, which the reader reads no object from either ([#274]) —, or does not decode to the header its `/N` and `/First` describe: every object placed in it is lost. A header breaks at a pair that is not two integers, that names a number no object can have or an offset no object can start at, whose offset is not past the one before it, or whose object would start at or past the end of the decoded data, unless a guard cut it (7.5.7); when it breaks after listing some objects, the reader reads those, and the objects lost are those the index places in the stream and the header does not list before the break, which the message counts — none, and no finding, when the header lists every one ([#275]). One finding per stream. An object stream whose own entry is broken is `xref.entry-broken`'s, and one the chain's index lacks once a reader limit stopped it short `xref.checked-in-part`'s, though a rebuild found the stream since. | Each object stream once, without keeping it — once more when the reader rebuilt its index, the chain's index places some of the objects in it, and only the rebuilt index others | +| `xref.object-stream-broken` | Error | An object stream the index places objects in is not in the index, is not an object stream — not of `/Type /ObjStm`, which the reader reads no object from either ([#274]) —, or does not decode to the header its `/N` and `/First` describe: every object placed in it is lost. A header breaks at a pair that is not two integers, that names a number no object can have or an offset no object can start at, whose offset is not past the one before it, or whose object would start at or past the end of the decoded data, unless a guard cut it (7.5.7); when it breaks after listing some objects, the reader reads those, and the objects lost are those the index places in the stream and the header does not list before the break, which the message counts — none, and no finding, when the header lists every one ([#275]). One finding per stream. An object stream whose own entry is broken is `xref.entry-broken`'s, and one the chain's index lacks once a reader limit stopped it short `xref.checked-in-part`'s, though a rebuild found the stream since — as is one whose decoding a guard ended, its data left encoded, at the filter past `MaxFilterChainLength` or at a `/Filter`, a `/DecodeParms` or a parameter that leads to an object a guard cut or left out of the index ([#263]). | Each object stream once, without keeping it — once more when the reader rebuilt its index, the chain's index places some of the objects in it, and only the rebuilt index others | | `xref.object-stream-value-wrong` | Warning | An object stream gives its `/N`, `/First` or `/Length` as a real number with no fractional part, where the specification asks for an integer (Tables 5 and 16). The reader reads it as the integer it equals, as MuPDF, PDFBox and pdf.js do; qpdf and poppler lose the stream's objects ([#215]). One finding per stream and key. A value with a fractional part is another fault: a `/N` or a `/First` leaves the stream unreadable, `xref.object-stream-broken`'s, and a `/Length` is recovered through the stream's `endstream`, which the reader reports (`stream.length-invalid`). | As `xref.object-stream-circular` | | `xref.offset-imprecise` | Warning | Offsets name the white space before what they designate rather than its first byte (7.5.4, 7.5.5): Microsoft Print to PDF names the line feed before each object, and some tools the one before `xref`. Every reader skips the white space. One finding for the file, with how many offsets and the first of them. | The sections of the chain, and each object's header | | `xref.object-past-size` | Warning | In-use objects are numbered above the trailer's `/Size`, which Table 15 makes a conforming reader ignore ("shall be ignored and defined to be missing"). The entries, the objects' headers and the references agree against `/Size`, and the reader reads those objects, as qpdf does while warning. One finding for the file, with how many objects and the first of them; `file.size-wrong` then says nothing more of that `/Size`. | The index and the trailer's `/Size` | -| `xref.checked-in-part` | Information | Part of the index was not checked: one of the reader's limits stopped it reading a section, a trailer or an object stream whole, the chain past `MaxXRefSectionCount` sections, or the index past the `MaxIndexEntries` object numbers it holds, said once, where it filled — raising the limit lets the rest be checked —; an entry whose header, where it places its object or near it, runs past `MaxObjectLength`; the entries the probe left undecided once reading their headers had consumed twice the file's length — the bytes crossed up to each decision, past the first 64 of each, and every byte read past a run an entry skips, which a valid file's headers, distinct bytes, never reach —, said once ([#175]); or the document is encrypted and its object streams are readable only once decrypted (M16). A chain cut where it goes on outside the file is located at the section whose `/Prev` names where. | What the reader and the probe of the index recorded | +| `xref.checked-in-part` | Information | Part of the index was not checked: one of the reader's limits stopped it reading a section, a trailer or an object stream whole, the chain past `MaxXRefSectionCount` sections, or the index past the `MaxIndexEntries` object numbers it holds, said once, where it filled — raising the limit lets the rest be checked —; an entry whose header, where it places its object or near it, runs past `MaxObjectLength`; the entries the probe left undecided once reading their headers had consumed twice the file's length — the bytes crossed up to each decision, past the first 64 of each, and every byte read past a run an entry skips, which a valid file's headers, distinct bytes, never reach —, said once ([#175]); the sections older than one whose `/Prev`, which the reader follows, leads to an object `MaxObjectLength` cuts — or the cross-reference stream of its revision, and any section older than it, when its `/XRefStm` does —, which end the chain there, said at that section with the reference that names them ([#263]); or the document is encrypted and its object streams are readable only once decrypted (M16). A chain cut where it goes on outside the file is located at the section whose `/Prev` names where. | What the reader and the probe of the index recorded | The entries the `xref.entry-*`, `xref.generation-mismatch`, `xref.object-stream-*`, `xref.offset-imprecise` and `xref.object-past-size` rules check are those of the index the file's chain gave, as the reader first read @@ -87,7 +87,7 @@ it is checked: the exception above judges the index the reader rebuilt, and the | Rule | Severity | Meaning | Reads | |---|---|---|---| | `object.reference-missing` | Warning | An object reachable from the trailer refers to an object the file does not define — none in its index, a free entry, or object 0, which heads the free list and is never in use (7.5.4) —, which is null (7.3.10): the reader reads it so, rebuilding nothing, and what the reference was to give is lost. One finding per object that holds such references, naming the first. An object one of the reader's limits left out of the index is unknown rather than missing, and is `xref.checked-in-part`'s. An object the index holds and the reader cannot produce is `xref.entry-broken`'s or `xref.object-stream-broken`'s — but one a rebuild serves from another object stream than the chain's index places it in, or from another definition of that stream, which does not list it or holds it written as a stream, and so loses, which no finding names yet ([#276]) —, a kid of the page tree `page-tree.kid-invalid`'s — one in a node's `/Kids`, an array of kids of its own or a root written in the catalog the walk read whole, not what an array written among the kids holds, which the walk does not read —, and a trailer's `/Root` that names nothing `file.root-invalid`'s. | Every object reachable from the trailer, once each | -| `object.endobj-missing` | Warning | An object reachable from the trailer does not end with `endobj` (7.3.10): another token, or the end of the file, follows its value. The reader reads the value as far as it goes, as qpdf does while reporting it. An empty object, `2 0 obj endobj`, has its `endobj`. After a stream whose data runs past the 8 KB window the reader first reads through, the file is asked for the bytes after its `endstream` since [#55]. More than a few dozen bytes of white space or comment before the next token leave what follows unseen, and a stream whose declared length the reader kept for want of an `endstream`, or whose data an `endobj` follows without one, leaves where the object ends unknown: neither is judged. | What follows each reachable object's value, as the reader read it | +| `object.endobj-missing` | Warning | An object reachable from the trailer does not end with `endobj` (7.3.10): another token, or the end of the file, follows its value. The reader reads the value as far as it goes, as qpdf does while reporting it. An empty object, `2 0 obj endobj`, has its `endobj`. An object whose array or dictionary was never closed lacks it where the next object's header — `N G obj` in digits, white space alone between them, an object number from 1 to 2,147,483,647 and a generation up to 65,535 —, met where an element, a key or a value is expected, ends every container left open ([#230]); an array or a dictionary that holds such a header and is closed after it ends there all the same, its `endobj` judged missing, which none of the corpus's 344,361 objects does. What follows a stream's `endstream` is read through white space and comments however long, and the file asked for it when it lies past the 8 KB window the reader first reads through — after a stream whose data runs past that window ([#55]), or an `endstream` the window's edge comes too soon after ([#174]). A stream with no `endstream` has its `endobj` when the object's `endobj` follows the length it declared — after white space and comments, a comment only once white space comes before it —, or ends its data with nothing but white space and comments after it before the next object or the end of the file; one whose data runs on to the next object, or to the end of the file, lacks it. Where the object ends is unknown, and is not judged, after a stream whose declared length the reader kept with neither an `endstream` nor an `endobj` after it, one whose data, run on to the next object or the end of the file, holds an `endobj` that may be its object's own, and one whose end lies past what the reader reads first once the document's searches for `endstream` have read as much of the file as they may. So is a stream whose `/Length` one of the reader's limits cut or left out of the index, when an object header its data spells, or an `endstream` it spells, ends its data: only that length could tell either from the stream's own end. An `endstream` is the stream's own, and an `endobj` its object lacks judged missing, only where what follows the object's end — its `endobj`, or the `endstream` where none follows — is the next object the index places, a cross-reference stream the chain read, the end of the file, or the `xref` or `startxref` keyword ([#224]). | What follows each reachable object's value, as the reader read it | | `object.name-null-character` | Warning | An object reachable from the trailer holds a name with a null character, written `#00`, which a name cannot contain (7.3.5). The reader keeps the name; qpdf refuses it and reads dictionaries without the keys so named. One finding per object, naming the first. | Every name of every reachable object | | `object.key-missing` | Warning | An object reachable from the trailer lacks a key the [Arlington PDF Model](#object-shapes-from-the-arlington-pdf-model) requires of its type — absent, or `null`, which in a dictionary is the same (7.3.7): a key the model requires once the version the file declares has it, or one it requires in some versions only, such as a form XObject's `/Name` in PDF 1.0. An inheritable key is looked for up the `/Parent` chain, and a field's `/DA` in the interactive form's too; a key only an extension of ISO 32000 defines is never required. An array shorter than the elements it requires lacks each required element it does not hold, one finding per element. [One finding per type and key](#object-shapes-from-the-arlington-pdf-model), at the first object, with how many lack it. | Every object reachable from the trailer, typed along the model's links, once each | | `object.value-type-wrong` | Warning | A value is of a type the Arlington PDF Model does not allow for its key, judged by the value's class: a real, even `400.0`, where an integer belongs; a string where a name does; a stream where a dictionary does, or the reverse, which is reported on the object that holds it and not walked into. A `null` written in an array is judged like any other value; one reached through a reference is not judged. One finding per type and key. | As `object.key-missing` | @@ -110,7 +110,7 @@ object nothing reachable refers to is not judged by them. | `page-tree.parent-wrong` | Warning | A node or a page has no `/Parent`, or one that is not the node listing it; or the root has one (Tables 29 and 30). The reader, as qpdf, walks the tree through `/Kids` and inherits along that path. | The page tree | | `page-tree.mediabox-invalid` | Warning | A page has no `/MediaBox`, its own or inherited, or one that is not four numbers enclosing an area (Table 30, 7.9.5): its size is unknown, and qpdf gives it a US Letter sheet's. Any two opposite corners make a rectangle, so a box written from its upper corner is sound. Missing, reported for each page; malformed, where it is written. | The page tree, and each media box | | `page-tree.resources-missing` | Warning | A page has no `/Resources`, its own or inherited (Table 30), where an empty dictionary says it uses none. | The page tree | -| `page-tree.page-orphaned` | Information | An object of `/Type /Page`, with a `/Parent` or a `/Contents`, that the page tree does not list: no reader shows it. Nothing forbids one — an edit leaves them —; a marked-content property list typed `/Page`, as DocuSign writes, is not a page. Judged only where the file's own index is sound and whole and the tree was read whole — no part of it left unread or cut by one of the reader's limits —, since elsewhere loading an object may bring back what an update deleted, and a page the tree seems not to list may lie past a cut. | Every object the file's index holds in use, once each, without its content | +| `page-tree.page-orphaned` | Information | An object of `/Type /Page`, with a `/Parent` or a `/Contents`, that the page tree does not list: no reader shows it. Nothing forbids one — an edit leaves them —; a marked-content property list typed `/Page`, as DocuSign writes, is not a page. Judged only where the file's own index is sound and whole — no section lost, cut, or whose `/Prev` or `/XRefStm` one of the reader's limits hid, with the sections it names ([#263]) — and the tree was read whole — no part of it left unread or cut by one of the reader's limits —, since elsewhere loading an object may bring back what an update deleted, and a page the tree seems not to list may lie past a cut. | Every object the file's index holds in use, once each, without its content | The page tree is walked once from the catalog's `/Pages`, through `/Kids`, as qpdf walks it: a node is a dictionary with a `/Kids` array or of `/Type /Pages`, any other dictionary the tree lists is a page, and pages are counted as the @@ -191,8 +191,8 @@ element — weighing most, and **a tie leaves the object unchecked** rather than not decode, or could not decrypt before M16, or left out of the index by a reader limit —, is present, and is not checked: that is `object.reference-missing`'s or the cross-reference rules' business. An object a reader limit cut — at `MaxObjectLength`, or a member of an object - stream whose data `MaxDecodedStreamLength` cut — is not judged either, nor what is written inside it: what it lacks - may lie past the limit. An ancestor so cut, or one a guard left out of the index, gives any key an object would inherit + stream whose data `MaxStreamLength` or `MaxDecodedStreamLength` cut, or whose decoding a guard ended ([#263]) — is not + judged either, nor what is written inside it: what it lacks may lie past the limit. An ancestor so cut, or one a guard left out of the index, gives any key an object would inherit from it. - Keys are taken in ordinal order and elements in order, so that two validations give the same findings in the same order. @@ -304,9 +304,10 @@ Some faults the structural profile does not report, each for a reason: from a stray dictionary typed `/Page`. - **An array, a dictionary or a string still open at the end of the data** — the reader takes it as it stands and reports it as `syntax.truncated-object`, in the document's diagnostics ([#119]); `object.endobj-missing` reports the - object that runs to the end of the file, as it reports any other object without its `endobj` — but for one that lost - its container's closing too, whose container the next object's `endobj` ends: the reader reads that `endobj` as the - object's, and only `syntax.truncated-object` tells ([#230]). + object that runs to the end of the file, as it reports any other object without its `endobj`, and the one whose + containers the next object's header ended ([#230]). A container an `endobj` ends is `syntax.truncated-object`'s alone, + that `endobj` being the object's — or the next object's, when the object lost its own and that object's header holds + a comment, which the header rule does not take for one: the container then runs on to that `endobj`. The object-shape rules read part of the Arlington PDF Model, and leave the rest silent until a rule of its own is worth its noise: @@ -357,15 +358,19 @@ PDF/UA profiles of the `AdCodicem.Pdf.Conformance` package (M20) take families o [#128]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/128 [#129]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/129 [#161]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/161 +[#174]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/174 [#175]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/175 [#182]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/182 [#215]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/215 [#219]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/219 +[#220]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/220 [#221]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/221 +[#224]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/224 [#230]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/230 [#237]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/237 [#256]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/256 [#259]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/259 +[#263]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/263 [#274]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/274 [#275]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/275 [#276]: https://github.com/AdCodicem/AdCodicem.Pdf/issues/276 diff --git a/src/AdCodicem.Pdf/Diagnostics/PdfDiagnosticCodes.cs b/src/AdCodicem.Pdf/Diagnostics/PdfDiagnosticCodes.cs index a83b6148..0f53d5ff 100644 --- a/src/AdCodicem.Pdf/Diagnostics/PdfDiagnosticCodes.cs +++ b/src/AdCodicem.Pdf/Diagnostics/PdfDiagnosticCodes.cs @@ -35,7 +35,9 @@ public static class PdfDiagnosticCodes /// A cross-reference section the chain names, through /Prev or /XRefStm, is where it is named but /// cannot be read — a stray token among a table's rows or in place of its trailer, a subsection numbering objects /// past 2,147,483,647, a cross-reference stream whose /W, /Index, /Size or data cannot give - /// rows —, and nothing near it can be read in its place. The rows read before the + /// rows, or whose /Length no section read before it places and whose data no endstream ends before the + /// next object or the end of the file —, and nothing near it can be read in its place. A value that leads to an object + /// one of the reader's limits cut is the limit's: the chain ends there, and nothing is reported of the section. The rows read before the /// fault, if any, were kept; the objects only the rest indexes are missing from the index, which is rebuilt by /// scanning the file when one of them is asked for. The report is placed at the section, the fault in its message. /// @@ -59,13 +61,17 @@ public static class PdfDiagnosticCodes /// A stream's /Length is not where its data ends: its endstream lies elsewhere and ends the data, the /// /Length gives no length the reader can take and the endstream ends the data, or no endstream /// follows the declared length — none before the next object or the end of the file, or none looked for once the - /// document's searches read as much of the file as they may — and that length is kept. + /// document's searches read as much of the file as they may — and that length is kept, or the data runs to the next + /// object, or, taken without a search, to the end of the file. A /Length that leads to an object one of the reader's limits cut, or left out of the index, is not + /// judged: the limit's report says why. /// public const string StreamLengthInvalid = "stream.length-invalid"; /// - /// A stream has no endstream before the end of the file, or before the endobj that follows its data: - /// its data runs to the end of the file. + /// A stream has no endstream before the next object or the end of the file: its object's endobj ends its + /// data — after its declared length, or where the object ends —, or its data runs to the end of the file. Data taken + /// without a search, the document's searches having read as much of the file as they may, is + /// 's, whose endstream no search said is missing. /// public const string StreamTruncated = "stream.truncated"; @@ -218,6 +224,15 @@ public static class PdfDiagnosticCodes /// public const string LimitDecodedStream = "limit.decoded-stream"; + /// + /// A stream's encoded data runs past , and was read up to it: + /// what lies within it is decoded as data the reader cut, and handed back as it is where decoding hands its data back + /// undecoded. Reported once for each stream in the document's diagnostics, when its data is first read, and in + /// diagnostics of your own each time you decode it. Like every limit.* code, the guard is the reader's, not + /// damage in the file. + /// + public const string LimitStreamLength = "limit.stream-length"; + /// /// An object is longer than , and only the part /// within it was parsed. An object an object stream holds is bounded alike, and reported once for each definition of diff --git a/src/AdCodicem.Pdf/Diagnostics/PdfDiagnostics.cs b/src/AdCodicem.Pdf/Diagnostics/PdfDiagnostics.cs index 974e0899..afbd8952 100644 --- a/src/AdCodicem.Pdf/Diagnostics/PdfDiagnostics.cs +++ b/src/AdCodicem.Pdf/Diagnostics/PdfDiagnostics.cs @@ -129,6 +129,45 @@ internal void RollBack(PdfDiagnosticsMark mark) } } + /// + /// Drops every entry recorded since but the guards reached, their limit.* reports and + /// their count standing, and those the capacity left out counted as left out still: what a reading met beside a guard + /// it is to make again is met again, and the guard stands. The reports decodings moved in that the mark recorded are + /// held again, with the guards' kept, so that a decoding that meets one again does not report it twice. + /// + internal void RollBackBesideLimits(PdfDiagnosticsMark mark) + { + var kept = mark.Count; + var held = mark.DecodingReports is { } recorded ? new Dictionary(recorded) : null; + + for (var i = mark.Count; i < _entries.Count; i++) + { + var entry = _entries[i]; + + if (!entry.Code.StartsWith("limit.", StringComparison.Ordinal)) + { + continue; + } + + _entries[kept++] = entry; + + if (held is not null && _decodingReports?.ContainsKey(entry) == true) + { + held[entry] = held.GetValueOrDefault(entry) + 1; + } + } + + // Every guard reached since the mark was reported once, kept as an entry or left out by the capacity. + var limitsLeftOut = Math.Max(0, _limits - mark.Limits - (kept - mark.Count)); + _entries.RemoveRange(kept, _entries.Count - kept); + _suppressed = mark.Suppressed + limitsLeftOut; + + if (held is not null) + { + _decodingReports = held; + } + } + /// /// Moves every entry recorded since into , which applies /// its own capacity, and drops them from this instance. diff --git a/src/AdCodicem.Pdf/Documents/PdfDocument.cs b/src/AdCodicem.Pdf/Documents/PdfDocument.cs index 4f56f3b0..0e7e95b6 100644 --- a/src/AdCodicem.Pdf/Documents/PdfDocument.cs +++ b/src/AdCodicem.Pdf/Documents/PdfDocument.cs @@ -27,7 +27,11 @@ namespace AdCodicem.Pdf.Documents; /// accessor that resolves, a stream's data asked for —, however it was opened and whether or not it was read before. /// What was handed out stays the caller's: a dictionary's direct values, a stream's , /// every object's ToString. Disposing lets go of what the document kept: the objects it read, its index, and the -/// file, closed when it was opened from a path, and held no longer when it was opened from memory or a stream. +/// file, closed when it was opened from a path, and held no longer when it was opened from memory or a stream. A stream's +/// raw bytes taken before from a document held in memory are a view of that memory rather than a copy +/// (), and so is what decoding gives back when no filter changes the data +/// (): for as long as they are held, they keep it alive, and they change +/// as it does. /// /// public sealed class PdfDocument : IDisposable diff --git a/src/AdCodicem.Pdf/Documents/PdfReaderLimits.cs b/src/AdCodicem.Pdf/Documents/PdfReaderLimits.cs index afb699bb..e6293fe9 100644 --- a/src/AdCodicem.Pdf/Documents/PdfReaderLimits.cs +++ b/src/AdCodicem.Pdf/Documents/PdfReaderLimits.cs @@ -33,6 +33,7 @@ public sealed record PdfReaderLimits public static PdfReaderLimits Unbounded { get; } = new() { MaxDecodedStreamLength = int.MaxValue, + MaxStreamLength = int.MaxValue, MaxObjectLength = int.MaxValue, MaxXRefSectionLength = int.MaxValue, MaxXRefSectionCount = int.MaxValue, @@ -61,12 +62,38 @@ public sealed record PdfReaderLimits /// The value is zero or less. public int MaxDecodedStreamLength { get; init => field = Length(value); } = 256 * 1024 * 1024; + /// + /// Gets the most of one stream's encoded data the reader reads at once, in bytes. Defaults to 256 MB: a /Length + /// of a few digits can claim gigabytes, and a scanned map can hold an uncompressed image of 300 MB. + /// + /// + /// + /// The encoded data is read in one piece each time it is asked for — , or + /// decoding it —, so a value above is taken as that length, and a /Length past it + /// is cut there. Data past the bound is cut at it: what lies within it is read, and decoded as data the reader cut, + /// whose lost tail is no damage. Reaching it reports , + /// once for each stream in the document's diagnostics, and in diagnostics of your own each time you decode the stream + /// (). stays the + /// length the file holds. + /// + /// + /// Data that decoding hands back as it is — no filter, an image filter, a filter the library does not know, one that + /// could not decode it — is bounded by this guard, not by , which bounds what the + /// filters produce: a large uncompressed image needs this bound raised alone (#170). + /// + /// + /// The value is zero or less. + public int MaxStreamLength { get; init => field = Length(value); } = 256 * 1024 * 1024; + /// /// Gets the longest object the reader parses, in bytes, the data of a stream aside, whether the file writes it /// directly or an object stream holds it. Defaults to 16 MB, which a direct array of a million references would need. /// /// - /// A value above is taken as that length. Reaching it reports + /// A stream's data never counts toward it, however its end is found, nor the end-of-line after its stream + /// keyword: a /Length that is wrong, missing or past the file has the data searched for its endstream past + /// the bytes the object was parsed through, not read through a larger window (#174); what the reader reads of the data at once is + /// 's. A value above is taken as that length. Reaching it reports /// , once for each object, and for each object an object /// stream holds, once for each definition of that stream. /// @@ -109,7 +136,9 @@ public int MaxXRefSectionCount /// /// /// A classic trailer that ends within the window its cross-reference table was read through is read whole, - /// whatever its length: the guard bounds how far the reader follows one past that window. A value above + /// whatever its length: the guard bounds how far the reader follows one past that window. A cross-reference stream's + /// data is not a trailer: one whose /Length the reader cannot read while it reads the chain is searched for its + /// endstream in the file, up to the next object or the end of the file. A value above /// is taken as that length. Reaching it reports /// , and ends the cross-reference chain at the section cut: an /// older section, or a key, past the cut is unknown, and what only older sections index reads as null until the diff --git a/src/AdCodicem.Pdf/IO/EndObjTail.cs b/src/AdCodicem.Pdf/IO/EndObjTail.cs new file mode 100644 index 00000000..acba3ac6 --- /dev/null +++ b/src/AdCodicem.Pdf/IO/EndObjTail.cs @@ -0,0 +1,250 @@ +namespace AdCodicem.Pdf.IO; + +/// +/// Follows, through the reads of a search for endstream in their order, where the data of a stream that has no +/// endstream ends when its object's own endobj ends it: at the last endobj, a whole keyword, after +/// which only white space and comments run to where the search stops, the end-of-line before it left out (#174). +/// +/// +/// +/// A comment after the keyword is white space, as ISO 32000-1 (7.2.3) makes one outside a stream's data: it runs to the end +/// of its line, and an endobj it holds is no keyword. Each byte is read once, however much the reads repeat of the +/// ones before; the last few bytes read are kept, so that a keyword that runs from one read into the next is found, and +/// what comes before it known. +/// +/// +/// Where the search stops is known only once the read that holds the header stopping it is searched, and that header may +/// start in a read before it, which was followed past it. What lies between is the header's runs of digits and white +/// space, which make no keyword: the place where what follows the last endobj stopped being white space and +/// comments says, alone, whether that run reached the header. +/// +/// +internal struct EndObjTail +{ + /// + /// The bytes a read keeps of the reads before it: the part of an endobj they hold, five bytes at most, and the two + /// before it that say whether it starts a token and where its end-of-line starts — seven, rounded up. + /// + private const int Kept = 8; + + private readonly long _dataStart; + private long _upTo; + private Mode _mode; + private long _keywordEnd; + private long _dataEnd; + private long _brokenAt; + private ulong _recent; + private int _recentCount; + + /// Initializes a new instance of the struct, for data that starts at . + public EndObjTail(long dataStart) + { + _dataStart = dataStart; + _upTo = dataStart; + _keywordEnd = -1; + _dataEnd = -1; + _brokenAt = long.MaxValue; + } + + private enum Mode : byte + { + /// No endobj that may end the data is being followed. + Data, + + /// An endobj was read whose next byte says whether it is a whole keyword. + AfterKeyword, + + /// White space follows the last whole endobj. + Tail, + + /// A comment after the last whole endobj runs on. + Comment, + } + + /// + /// Reads the bytes a read of the search holds, from where the reads before stopped to . + /// + /// The bytes of the read. + /// Where they start in the file. + /// Where the search stops in them, or where they end. + public void Read(ReadOnlySpan bytes, long at, long until) + { + var start = (int)Math.Max(0, _upTo - at); + var end = (int)(Math.Min(until, at + bytes.Length) - at); + + if (start >= end) + { + return; + } + + var i = _mode == Mode.Data ? Straddling(bytes, at, start, end) : start; + + while (i < end) + { + switch (_mode) + { + case Mode.AfterKeyword: + // "endobjx" is no keyword; anything else that follows it may be the tail, and is read as such. + if (PdfCharacters.IsRegular(bytes[i])) + { + _brokenAt = at + i; + _mode = Mode.Data; + } + else + { + _mode = Mode.Tail; + } + + break; + + case Mode.Tail: + var next = bytes[i..end].IndexOfAnyExcept(PdfCharacters.Whitespace); + + if (next < 0) + { + i = end; + } + else if (bytes[i + next] == (byte)'%') + { + i += next + 1; + _mode = Mode.Comment; + } + else + { + i += next; + _brokenAt = at + i; + _mode = Mode.Data; + } + + break; + + case Mode.Comment: + // A comment runs to the end of the line; both line endings occur in the wild. + var lineEnd = bytes[i..end].IndexOfAny((byte)'\r', (byte)'\n'); + i = lineEnd < 0 ? end : i + lineEnd + 1; + _mode = lineEnd < 0 ? Mode.Comment : Mode.Tail; + break; + + default: + var hit = bytes[i..end].IndexOf(StreamEnds.EndObjKeyword); + + if (hit < 0) + { + i = end; + } + else + { + i = Found(bytes, at, start, at + i + hit) ? i + hit + StreamEnds.EndObjKeyword.Length : i + hit + 1; + } + + break; + } + } + + Keep(bytes[start..end]); + _upTo = at + end; + } + + /// + /// Gives where the data ends when the search stops at , at an endobj that only white space + /// and comments separate from it; -1 when none does, or the reads did not reach it. + /// + public readonly long DataEnd(long stop) => + stop <= _upTo && _keywordEnd >= 0 && _keywordEnd <= stop && stop <= _brokenAt ? _dataEnd : -1; + + /// + /// Finds an endobj that starts in the bytes the reads before kept and ends in these, and gives where to go on + /// reading these from. + /// + private int Straddling(ReadOnlySpan bytes, long at, int start, int end) + { + var keyword = StreamEnds.EndObjKeyword; + + // The keyword holds no part of itself again: one way at most of parting it fits. + for (var held = 1; held < keyword.Length && held <= _recentCount; held++) + { + var rest = keyword.Length - held; + + if (start + rest > end || !bytes.Slice(start, rest).SequenceEqual(keyword[held..])) + { + continue; + } + + var matches = true; + + for (var k = 0; k < held && matches; k++) + { + matches = Recent(held - 1 - k) == keyword[k]; + } + + if (matches) + { + return Found(bytes, at, start, at + start - held) ? start + rest : start; + } + } + + return start; + } + + /// + /// Takes the endobj that starts at , unless the byte before it makes it part of a longer + /// token, as the one the data may end at; whether the byte after it does is read next. + /// + /// + /// A regular character right before it makes it part of the token that character is in; a solidus makes it a name, + /// /endobj, and a percent sign a comment that starts there, %endobj. Only that byte is read: the bytes + /// before the keyword are the data's, which no lexing reads, so a comment that starts further back on the line does not + /// hide it — any other delimiter ends a token or stands alone, wherever a string, a dictionary or a comment it may open + /// would end. + /// + /// Whether the keyword starts a token. + private bool Found(ReadOnlySpan bytes, long at, int start, long keyword) + { + if (ByteAt(bytes, at, start, keyword - 1) is var before and >= 0 && + (PdfCharacters.IsRegular((byte)before) || before is '/' or '%')) + { + return false; + } + + var dataEnd = keyword; + + if (ByteAt(bytes, at, start, dataEnd - 1) == '\n') + { + dataEnd--; + } + + if (ByteAt(bytes, at, start, dataEnd - 1) == '\r') + { + dataEnd--; + } + + _keywordEnd = keyword + StreamEnds.EndObjKeyword.Length; + _dataEnd = dataEnd; + _brokenAt = long.MaxValue; + _mode = Mode.AfterKeyword; + return true; + } + + /// + /// Gives the byte at in the file: in these bytes from on, among those + /// kept before it; -1 before the start of the data. + /// + private readonly int ByteAt(ReadOnlySpan bytes, long at, int start, long position) => + position < _dataStart ? -1 + : position >= at + start ? bytes[(int)(position - at)] + : Recent((int)(at + start - 1 - position)); + + /// Gives a byte kept, bytes before the last one read. + private readonly byte Recent(int back) => (byte)(_recent >> (8 * back)); + + /// Keeps the last bytes read, the latest in the lowest byte. + private void Keep(ReadOnlySpan read) + { + foreach (var value in read[Math.Max(0, read.Length - Kept)..]) + { + _recent = (_recent << 8) | value; + } + + _recentCount = (int)Math.Min(Kept, _recentCount + (long)read.Length); + } +} diff --git a/src/AdCodicem.Pdf/IO/EndStreamState.cs b/src/AdCodicem.Pdf/IO/EndStreamState.cs index 098b9f24..78086071 100644 --- a/src/AdCodicem.Pdf/IO/EndStreamState.cs +++ b/src/AdCodicem.Pdf/IO/EndStreamState.cs @@ -8,23 +8,28 @@ internal enum EndStreamState : byte /// /// Nowhere between the start of the data and the next object after it — the one an index places, or the first header - /// the file's bytes hold, whichever is nearer —: the declared length is kept. + /// the file's bytes hold, whichever is nearer —, and no endobj of its object ends the data: the declared length + /// is kept when it ends before that object, and the data runs up to it otherwise. /// MissingBeforeNextObject, /// - /// Nowhere between the start of the data and the end of the file: the declared length is kept when the file can - /// hold it, and the data runs to the end of the file otherwise. + /// Nowhere between the start of the data and the end of the file, and no endobj of its object ends the data: + /// the declared length is kept when the file can hold it, and the data runs to the end of the file otherwise. /// MissingBeforeEndOfFile, - /// Nowhere before the endobj that follows the data, which is taken to run to the end of the file. + /// + /// Nowhere before the next object or the end of the file, and its object's endobj plays its part: the data is + /// the declared length the endobj follows, or ends where that endobj ends the object. + /// MissingBeforeEndObj, /// - /// Not looked for: the document's searches for endstream read as much of the file as they may, which only - /// searches that share a stretch of it make them do — objects that overlap, or headers the search does not take that - /// no index as written places. The declared length is kept. + /// Not looked for past the bytes the parser held: the document's searches for endstream read as much of the + /// file as they may, which only searches that share a stretch of it make them do — objects that overlap, or headers + /// the search does not take that no index as written places. The declared length is kept when it ends before the + /// next object, and the data runs up to it otherwise. /// NotSearched, } diff --git a/src/AdCodicem.Pdf/IO/Filters/Ascii85Filter.cs b/src/AdCodicem.Pdf/IO/Filters/Ascii85Filter.cs index 06fa3629..5f05bb85 100644 --- a/src/AdCodicem.Pdf/IO/Filters/Ascii85Filter.cs +++ b/src/AdCodicem.Pdf/IO/Filters/Ascii85Filter.cs @@ -18,8 +18,10 @@ internal static class Ascii85Filter /// What the data holds that the format does not allow. /// The most the data may decode to. /// - /// Whether the reader cut the data short at one of its limits: a group left partial at its end, before any - /// ~, is one the cut parted, and is dropped rather than completed. + /// Whether the data ends where the file's may not — the reader cut it short at one of its limits, or the caller says a + /// guard hides where it ends —: a group left partial at its end, before any ~, is one the cut parted, and is dropped rather + /// than completed, though the bytes it encodes count against , which + /// then says, and counts them. /// public static byte[] Decode( ReadOnlySpan data, out bool limited, out DataFaults faults, int maxLength, bool cut = false) @@ -142,6 +144,13 @@ public static byte[] Decode( limited = !WriteGroup(output, (uint)tuple, count - 1, group); } + else if (count > 1) + { + // Data that ends where the file's may not leaves a group it parts, whose bytes the missing digits decide: none is + // written. The whole data holds at least the count-1 bytes it encodes, which the bound counts all the same. + limited = count - 1 > maxLength - output.Count; + faults.Dropped = count - 1; + } if (!ended) { diff --git a/src/AdCodicem.Pdf/IO/Filters/AsciiHexFilter.cs b/src/AdCodicem.Pdf/IO/Filters/AsciiHexFilter.cs index 84b126f6..d8781c70 100644 --- a/src/AdCodicem.Pdf/IO/Filters/AsciiHexFilter.cs +++ b/src/AdCodicem.Pdf/IO/Filters/AsciiHexFilter.cs @@ -18,8 +18,10 @@ internal static class AsciiHexFilter /// What the data holds that the format does not allow. /// The most the data may decode to. /// - /// Whether the reader cut the data short at one of its limits: a digit left alone at its end, before any - /// >, is the first of a pair the cut parted, and is dropped rather than completed with a zero. + /// Whether the data ends where the file's may not — the reader cut it short at one of its limits, or the caller says a + /// guard hides where it ends —: a digit left alone at its end, before any >, is the first of a pair the cut parted, and is + /// dropped rather than completed with a zero, though the byte counts against , which + /// then says, and counts it. /// public static byte[] Decode( ReadOnlySpan data, out bool limited, out DataFaults faults, int maxLength, bool cut = false) @@ -86,6 +88,13 @@ public static byte[] Decode( limited = true; } } + else if (high >= 0) + { + // Data that ends where the file's may not leaves its last byte half read, which the next digit decides: it is + // not written, and the whole data, which holds it, counts it against the bound all the same. + limited = length >= maxLength; + faults.Dropped = 1; + } return buffer.AsSpan(0, length).ToArray(); } diff --git a/src/AdCodicem.Pdf/IO/Filters/DataFaults.cs b/src/AdCodicem.Pdf/IO/Filters/DataFaults.cs index 60fa20fa..94110013 100644 --- a/src/AdCodicem.Pdf/IO/Filters/DataFaults.cs +++ b/src/AdCodicem.Pdf/IO/Filters/DataFaults.cs @@ -46,6 +46,12 @@ internal struct DataFaults /// For data that ends inside a RunLength run: how many of its bytes it holds. public int UnitHeld; + /// + /// For data the reader cut, or whose end the caller says a guard hides, that ends inside an ASCII85 group or an + /// ASCIIHex pair: how many bytes the unit would decode to, which were not written. + /// + public int Dropped; + /// Counts a byte passed over at , counted from 0. public void Pass(int index) { diff --git a/src/AdCodicem.Pdf/IO/Filters/FilterChain.cs b/src/AdCodicem.Pdf/IO/Filters/FilterChain.cs index a571e5a4..a1dd0f31 100644 --- a/src/AdCodicem.Pdf/IO/Filters/FilterChain.cs +++ b/src/AdCodicem.Pdf/IO/Filters/FilterChain.cs @@ -32,6 +32,13 @@ namespace AdCodicem.Pdf.IO.Filters; /// towards it, though decoding runs no step for it. /// /// +/// It ends, as well, at a value that leads to an object one of the reader's guards cut, or left out of the index: the +/// /Filter entry or an element of it, /DecodeParms or a value of a step's parameters. What the cut left of +/// it — null, a name it parted, a value it may yet go on past — is the guard's, and no filter the file gave: the step it +/// is read for, and those after it, do not run, and the data is left as the steps before them decode it, the guard +/// reported where it was reached (#263). +/// +/// /// What was resolved is kept only where the file did not write it plainly. The /Filter array is read again as the /// steps run, so a name it holds costs nothing more; the name an element written as a reference led to is kept, a slot /// each. A dictionary that serves every step is read once for Flate and once for LZW, and of an array of parameters, only @@ -107,23 +114,19 @@ private FilterChain(in Reading reading) /// The most filters the stream's data may pass through: the chain ends at the element past them. public static FilterChain Resolve(PdfDictionary dictionary, int maxFilters) { - var reader = default(Resolving); - TryRead(dictionary, maxFilters, ref reader, readParameters: true, keep: true, out var reading); - return reading.Filters is null ? NoFilter : new FilterChain(reading); + var reading = Read(dictionary, maxFilters, keep: true); + return reading.Filters is null && reading.End == FilterChainEnd.None ? NoFilter : new FilterChain(reading); } /// - /// Resolves what ends the chain of filters the stream describes, reading none of their - /// parameters and keeping nothing. + /// Resolves what ends the chain of filters the stream describes, as decoding resolves it, + /// and keeps nothing: the parameters decoding reads are read too, since a value among them that leads to an object a + /// guard cut, or left out of the index, ends the chain where it is read (#263). /// /// The stream's dictionary. /// The most filters the stream's data may pass through: the chain ends at the element past them. - public static FilterChainEnd EndOf(PdfDictionary dictionary, int maxFilters) - { - var reader = default(Resolving); - TryRead(dictionary, maxFilters, ref reader, readParameters: false, keep: false, out var reading); - return reading.End; - } + public static FilterChainEnd EndOf(PdfDictionary dictionary, int maxFilters) => + Read(dictionary, maxFilters, keep: false).End; /// /// Follows, through , every value decoding the stream describes @@ -135,7 +138,7 @@ public static FilterChainEnd EndOf(PdfDictionary dictionary, int maxFilters) /// False when ended the reading. public static bool TryFollow(PdfDictionary dictionary, int maxFilters, ref TReader reader) where TReader : struct, IFilterValueReader => - TryRead(dictionary, maxFilters, ref reader, readParameters: true, keep: false, out _); + TryRead(dictionary, maxFilters, ref reader, keep: false, out _); /// Gets the steps decoding runs, in order. public Steps GetEnumerator() => new(this); @@ -195,12 +198,11 @@ public bool MoveNext() /// The stream's dictionary. /// The most filters the stream's data may pass through: the chain ends at the element past them. /// Follows each value to what it leads to, and can end the reading. - /// Whether the parameters are read, or only the names of the filters. /// Whether what decoding needs again is kept, or only what ends the chain. /// What was read. /// False when ended the reading. private static bool TryRead( - PdfDictionary dictionary, int maxFilters, ref TReader reader, bool readParameters, bool keep, out Reading reading) + PdfDictionary dictionary, int maxFilters, ref TReader reader, bool keep, out Reading reading) where TReader : struct, IFilterValueReader { reading = new Reading(); @@ -274,7 +276,7 @@ PdfName name when PdfFilterPipeline.IsImageFilter(name) => FilterChainEnd.Image, Append(ref reading.Names, ref reading.NameCount, filter); } - if (readParameters && (filter == PdfName.FlateDecode || filter == PdfName.LZWDecode)) + if (filter == PdfName.FlateDecode || filter == PdfName.LZWDecode) { if (!parametersRead) { @@ -327,6 +329,27 @@ PdfName name when PdfFilterPipeline.IsImageFilter(name) => FilterChainEnd.Image, return true; } + /// + /// Resolves the chain of filters the stream describes, as decoding reads it: a value that + /// leads to an object one of the reader's guards cut, or left out of the index, ends it where it is read, as the + /// element past the guard on filters does — the steps before it run, and what they decode is left encoded by the rest + /// (#263). + /// + private static Reading Read(PdfDictionary dictionary, int maxFilters, bool keep) + { + var reader = default(Resolving); + + if (!TryRead(dictionary, maxFilters, ref reader, keep, out var reading)) + { + // The steps read whole before the value stand: the chain ends at the element that value was read for. + reading.End = FilterChainEnd.Guard; + reading.EndIndex = reading.Length; + reading.EndValue = null; + } + + return reading; + } + /// Adds to , which grows as a list does. private static void Append(ref T[]? items, ref int count, T item) { @@ -426,12 +449,22 @@ private struct Reading() public FilterParameters Lzw = new(); } - /// Follows each value as does, and never ends the reading. + /// + /// Follows each value as does, and ends the reading at one that leads to an object + /// one of the reader's guards cut, or left out of the index (). + /// private struct Resolving : IFilterValueReader { public readonly bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? value) { value = written?.Resolve(); + + if (written is PdfReference reference && reference.Source is PdfFileReader reader && reader.LeftByGuard(reference)) + { + value = null; + return false; + } + return true; } } diff --git a/src/AdCodicem.Pdf/IO/Filters/FilterChainEnd.cs b/src/AdCodicem.Pdf/IO/Filters/FilterChainEnd.cs index dfa82ee5..e01902d0 100644 --- a/src/AdCodicem.Pdf/IO/Filters/FilterChainEnd.cs +++ b/src/AdCodicem.Pdf/IO/Filters/FilterChainEnd.cs @@ -20,4 +20,11 @@ internal enum FilterChainEnd /// An element past the most filters the reader's guard lets a stream's data pass through. Limit, + + /// + /// A value decoding reads — the /Filter entry or an element of it, /DecodeParms or a value of a step's + /// parameters — that leads to an object one of the reader's guards cut, or left out of the index: what it holds is + /// the guard's, and the step it is read for, and those after it, do not run (#263). + /// + Guard, } diff --git a/src/AdCodicem.Pdf/IO/Filters/PdfFilterPipeline.cs b/src/AdCodicem.Pdf/IO/Filters/PdfFilterPipeline.cs index d91de075..76456305 100644 --- a/src/AdCodicem.Pdf/IO/Filters/PdfFilterPipeline.cs +++ b/src/AdCodicem.Pdf/IO/Filters/PdfFilterPipeline.cs @@ -33,23 +33,74 @@ public static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? diag /// /// Bounds what each filter's output may be and how many filters run, and says what reaching either does. public static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? diagnostics, PdfLimitGuard guard) => - Decode(stream, diagnostics, guard, out _); + Decode(stream, diagnostics, guard, endHidden: false, reachedBefore: false, out _, out _); /// - /// Decodes a stream's data under its document's guards, and says whether the most filters the reader lets data pass - /// through ended decoding: what comes back is then the data still encoded by the filters past them, no part of what - /// the stream decodes to, as a reader of its rows or its objects must know. + /// Decodes a stream's data under its document's guards, and says whether one of them ended decoding — the most filters + /// the reader lets data pass through, or a value the filters read that a guard cut or left out of the index: what comes + /// back is then the data still encoded by the filters past it, no part of what the stream decodes to —, and whether one + /// cut the data, as a reader of its rows or its objects must know. Data a proven endstream ended past white space + /// and comments that only a hidden length says are the data's or not () decodes up + /// to where that run starts, the first filter's running out there saying nothing of the file, and the data cut by a + /// guard where its output may lack what the run holds; Flate data whose deflate body ends in the data as taken is + /// whole. /// /// The stream to decode. /// Receives what decoding met; when none are supplied, the stream's document does. - /// Whether ended decoding. + /// + /// Whether ended decoding, or a value the filters read that + /// a guard cut or left out of the index did (#263). + /// + /// + /// Whether the data decoded is cut by a guard: the stream's own data, which the reader read in part, or what a filter + /// stopped at , or what a first filter that ran out where + /// a guard hides the data's end decoded to. A guard that loading a value the filters read reaches is that value's, and + /// does not cut the data. + /// /// /// Decoding reaches one of its document's guards, and the document was opened to throw when it does. /// - public static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? diagnostics, out bool leftEncoded) => - Decode(stream, diagnostics, stream.Data.LimitGuard, out leftEncoded); + public static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? diagnostics, out bool leftEncoded, out bool cutByGuard) => + Decode(stream, diagnostics, stream.Data.LimitGuard, endHidden: false, reachedBefore: false, out leftEncoded, out cutByGuard); - private static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? diagnostics, PdfLimitGuard guard, out bool leftEncoded) + /// + /// Decodes a stream's data as does, its end, with + /// , one a guard hides: a filter that runs out there says nothing of the file, as for data a + /// guard cut, while a fault in what it read is the file's still. + /// + /// The stream to decode. + /// Receives what decoding met; when none are supplied, the stream's document does. + /// + /// Whether where the data ends is the guard's to hide: data a search took up to an endstream nothing confirmed, + /// where the length that would say is one a guard cut, or a guard on decoding kept the data from being taken again to + /// it. The data decodes as data a guard cut does — a last group a filter holds in part, which the hidden end may have + /// parted, is dropped, and a predictor's last row undone as far as it goes —, though it is not counted as cut by it: a + /// group dropped so counts against all the same, and + /// against what the filters before the last share, the whole data holding at least as much. A filter after it is given + /// less, and may stay within the bound where the whole data would not: a decoding of data whose end is hidden that a + /// decoding of it reached the bound before keeps that verdict, with . + /// + /// + /// Whether a decoding of the same data stopped at before, + /// and reported it: the data comes back cut by the guard, and no filter reports it again. + /// + /// As the other overload gives it. + /// As the other overload gives it. + /// + /// Decoding reaches one of its document's guards, and the document was opened to throw when it does. + /// + public static ReadOnlyMemory Decode( + PdfStream stream, PdfDiagnostics? diagnostics, bool endHidden, bool reachedBefore, out bool leftEncoded, out bool cutByGuard) => + Decode(stream, diagnostics, stream.Data.LimitGuard, endHidden, reachedBefore, out leftEncoded, out cutByGuard); + + private static ReadOnlyMemory Decode( + PdfStream stream, + PdfDiagnostics? diagnostics, + PdfLimitGuard guard, + bool endHidden, + bool reachedBefore, + out bool leftEncoded, + out bool cutByGuard) { diagnostics ??= guard.DocumentDiagnostics; @@ -66,7 +117,14 @@ private static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? dia // its own reports in. if (diagnostics is null || !ReferenceEquals(diagnostics, guard.DocumentDiagnostics)) { - return DecodeChain(stream, chain, data, diagnostics, guard, out leftEncoded); + // Data MaxStreamLength cut is reported once in the document's diagnostics, as the reader reads it; diagnostics of + // a caller's own receive it with each decoding, as they receive what decoding meets. + if (diagnostics is not null && stream.Data.CutByGuard && stream.Data.Length > guard.Bound(PdfLimit.StreamLength)) + { + guard.Reach(PdfLimit.StreamLength, diagnostics, guard.DescribeStreamLengthCut(), stream.Data.Position); + } + + return DecodeChain(stream, chain, data, diagnostics, guard, endHidden, reachedBefore, out leftEncoded, out cutByGuard); } var decodings = diagnostics.Decodings; @@ -74,7 +132,7 @@ private static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? dia try { - return DecodeChain(stream, chain, data, decodings, guard, out leftEncoded); + return DecodeChain(stream, chain, data, decodings, guard, endHidden, reachedBefore, out leftEncoded, out cutByGuard); } finally { @@ -85,15 +143,39 @@ private static ReadOnlyMemory Decode(PdfStream stream, PdfDiagnostics? dia /// Decodes through the filters of , reporting into . private static ReadOnlyMemory DecodeChain( - PdfStream stream, FilterChain chain, ReadOnlyMemory data, PdfDiagnostics? diagnostics, PdfLimitGuard guard, out bool leftEncoded) + PdfStream stream, + FilterChain chain, + ReadOnlyMemory data, + PdfDiagnostics? diagnostics, + PdfLimitGuard guard, + bool endHidden, + bool reachedBefore, + out bool leftEncoded, + out bool cutByGuard) { leftEncoded = false; - // Whether the data a filter is given is data the reader cut at one of its limits: the parser's window cut the - // stream, or a filter before it stopped at MaxDecodedStreamLength. Every filter after a cut is given cut data, - // and the cut is reported once, by the guard that made it. - var cut = stream.Data.CutByGuard; - var limitReported = false; + // Whether the data a filter is given ends where the file's may not: data the reader cut at one of its limits — + // MaxStreamLength, or the window a guard stopped at, cut the stream's data, or a filter before it stopped at + // MaxDecodedStreamLength —, or data whose end the caller says a guard hides, or what a first filter that ran out + // where a guard hides the data's end decoded. Every filter after it is given such data, and a cut is reported once, + // by the guard that made it. A filter drops a last group such data holds in part, and counts it against the bound + // all the same (Ascii85Filter, AsciiHexFilter), and against what the filters before the last share. + var cut = stream.Data.CutByGuard || endHidden; + + // Data whose endstream is proven its own past white space and comments only a hidden length says are the data's or + // not (#224): the first filter decodes it up to where that run starts. What it meets before is the file's, its + // running out there says nothing of the file, and where its output may lack what the run holds — Flate that lost + // its tail, LZW or RunLength data without its end marker —, the output's end is the guard's, as of data a guard + // cut, for the filters after it and the reader of what the stream decodes to. Flate data whose deflate body ends + // in the data as the search took it is whole however the run reads, its checksum alone reaching into it. White + // space decodes to nothing in ASCII85 or ASCIIHex data, whose output is whole, and so does a comment in ASCIIHex, + // which no valid hexadecimal data holds; in ASCII85 a % is a digit, and no comment ends the data. + var hidden = stream.Data.EndHidden && !cut; + var endCut = false; + + // A guard a decoding of the same data reached before was reported then: its verdict stands, whatever this one meets. + var limitReported = reachedBefore; // The filters before the last share one bound for what they pass along together, and the last keeps its own: a // stream of one or two filters decodes as each filter alone would, and a chain however long holds, and fills, no @@ -113,6 +195,42 @@ private static ReadOnlyMemory DecodeChain( var last = step.Index == chain.LastIndex; var maxLength = last ? bound : (int)Math.Clamp(shared, 0, bound); + var given = data; + var silenced = false; + var checksumHidden = false; + + if (hidden) + { + hidden = false; + silenced = true; + var run = HiddenRunStart(data.Span, comments: step.Name != PdfName.ASCII85Decode); + + if (step.Name == PdfName.FlateDecode && FlateBodyEnds(data, maxLength, out var whole, out var clean)) + { + if (clean) + { + data = ApplyPredictor(whole, step, diagnostics, stream.Data.Position, cut: false); + shared -= data.Length; + continue; + } + + // A checksum that disagrees only where it reaches into the run says nothing of the file. + checksumHidden = run < data.Length && !ChecksumDisagrees(data[..run], maxLength); + } + else + { + data = data[..run]; + + // Data that is all the run decodes to nothing, which may lack all the run holds, but in ASCII85 or + // ASCIIHex data. + if (data.IsEmpty) + { + endCut = step.Name != PdfName.ASCII85Decode && step.Name != PdfName.ASCIIHexDecode; + break; + } + } + } + data = ApplyOne( step, data, @@ -121,24 +239,109 @@ private static ReadOnlyMemory DecodeChain( stream.Data.Position, guard, ref cut, + silenced, + checksumHidden, ref limitReported, - out var failed); + out var dropped, + out var failed, + out var ranOut); // Data a filter could not decode at all comes back as the filters before it left it, as it does at a filter the // library does not know: the filters after it would read what it encoded, and each fail on it again. if (failed) { - return data; + cutByGuard = stream.Data.CutByGuard || limitReported || endCut; + return given; } - shared -= data.Length; + if (silenced && ranOut) + { + cut = true; + endCut = true; + } + + // A group the filter dropped at the cut is data the whole would decode to, and leaves the filters after it no + // more room than the whole data would. + shared -= data.Length + dropped; } ReportEnd(chain, diagnostics, stream.Data.Position, guard); - leftEncoded = chain.End == FilterChainEnd.Limit; + + // A value a guard cut or left out of the index was reported where the guard was reached: the chain ends there in + // silence. + leftEncoded = chain.End is FilterChainEnd.Limit or FilterChainEnd.Guard; + cutByGuard = stream.Data.CutByGuard || limitReported || endCut; return data; } + /// + /// Gives where the run of white space, and with of comments, that ends + /// starts: the bytes only a hidden length says are the data's or not, as the gap between a + /// length and its endstream may hold them (#174), a comment there starting past white space within the data, at + /// the first % its line holds so — one the data starts with is data. + /// + /// Each line is read twice at most: the run is found in time linear in the data. + private static int HiddenRunStart(ReadOnlySpan data, bool comments) + { + var start = data.Length; + + while (true) + { + while (start > 0 && PdfCharacters.IsWhitespace(data[start - 1])) + { + start--; + } + + if (!comments) + { + return start; + } + + // A comment runs to the end of its line, which the white space passed ends, or the data's end does: it starts at + // the first % its line holds past white space, the end-of-line before the line's start among it. + var line = start; + + while (line > 0 && data[line - 1] is not ((byte)'\r' or (byte)'\n')) + { + line--; + } + + var comment = -1; + + for (var i = line; i < start && comment < 0; i++) + { + if (data[i] == (byte)'%' && i > 0 && PdfCharacters.IsWhitespace(data[i - 1])) + { + comment = i; + } + } + + if (comment < 0) + { + return start; + } + + start = comment; + } + } + + /// + /// Decodes Flate data whose end a guard hides as the search took it, and determines whether its deflate body ends in it + /// — its checksum whole, disagreeing or short —, the bound not reached: what it decoded to is then whole, and + /// says whether it is zlib data whose checksum agrees, with nothing to report. + /// + private static bool FlateBodyEnds(ReadOnlyMemory data, int maxLength, out byte[] decoded, out bool clean) + { + var ended = FlateFilter.TryDecode(data, out decoded, out var repaired, out var ending, out _, out var limited, maxLength) && + !limited && ending is FlateEnding.Whole or FlateEnding.ChecksumMismatch or FlateEnding.ChecksumMissing; + clean = ended && !repaired && ending == FlateEnding.Whole; + return ended; + } + + /// Determines whether the checksum of Flate data disagrees with what it decodes to. + private static bool ChecksumDisagrees(ReadOnlyMemory data, int maxLength) => + FlateFilter.TryDecode(data, out _, out _, out var ending, out _, out _, maxLength) && ending == FlateEnding.ChecksumMismatch; + /// /// Determines whether a filter produces an image format that should be left encoded. /// @@ -156,8 +359,24 @@ public static bool IsImageFilter(PdfName name) => /// /// Whether is data the reader cut; set when this filter stops at the bound too. /// + /// + /// Whether where the data ends says nothing of the file, though it is no cut — data whose end a guard hides, up to the + /// white space and comments only the hidden length says are the data's or not —: a filter that runs out there reports + /// nothing, drops no group the end parts, and undoes a predictor's last row as far as it goes only where it ran out. + /// + /// + /// Whether a Flate checksum that disagrees does so only where it reaches into what a hidden length says is the data's + /// or not, which says nothing of the file. + /// /// Whether the stream reported a filter that stopped at the bound already. + /// How many bytes a last group the cut parted would decode to, which the filter did not write. /// Whether the filter could decode nothing, which ends the chain. + /// + /// Whether the filter ran out at the end of its data where its output may lack what more data would decode to: Flate + /// that lost its tail, LZW or RunLength data without its end marker. White space, which decodes to nothing in ASCII85 + /// and ASCIIHex data, is no more data for them, nor a comment, which no ASCIIHex data holds, and which ends no ASCII85 + /// data. + /// private static ReadOnlyMemory ApplyOne( in FilterStep step, ReadOnlyMemory data, @@ -166,13 +385,20 @@ private static ReadOnlyMemory ApplyOne( long position, PdfLimitGuard guard, ref bool cut, + bool silenced, + bool checksumHidden, ref bool limitReported, - out bool failed) + out int dropped, + out bool failed, + out bool ranOut) { var name = step.Name; var maxLength = bounds.MaxLength; var given = cut; + var quiet = given || silenced; + dropped = 0; failed = false; + ranOut = false; if (name == PdfName.FlateDecode) { @@ -189,18 +415,18 @@ private static ReadOnlyMemory ApplyOne( } // Data a guard of the reader's cut ends early because the reader stopped reading it, or a filter before - // this one stopped decoding it, which the guard reported: that it ran out says nothing of the file. A fault, - // or a checksum that disagrees, lies in bytes the reader did read, and the whole data would meet it at the - // same point: it is the file's. - ReportEnding( - given && ending is FlateEnding.TailLost or FlateEnding.ChecksumMissing ? FlateEnding.Whole : ending, - decoded.Length, - faultAt, - data.Length, - diagnostics, - position); + // this one stopped decoding it, which the guard reported, and data whose end a guard hides ends where the guard + // may: that it ran out says nothing of the file. A fault, or a checksum that disagrees, lies in bytes the reader + // did read, and the whole data would meet it at the same point: it is the file's. + var endHidden = (quiet && ending is FlateEnding.TailLost or FlateEnding.ChecksumMissing) || + (checksumHidden && ending == FlateEnding.ChecksumMismatch); + ReportEnding(endHidden ? FlateEnding.Whole : ending, decoded.Length, faultAt, data.Length, diagnostics, position); Reached(limited, name, bounds.Shared, diagnostics, position, guard, ref cut, ref limitReported); - return ApplyPredictor(decoded, step, diagnostics, position, cut); + + // Data that ran out where a guard hides its end may have lost the end of its last row there: it is undone as far + // as it goes, as of data a guard cut. + ranOut = ending == FlateEnding.TailLost; + return ApplyPredictor(decoded, step, diagnostics, position, cut || (silenced && ranOut)); } if (name == PdfName.LZWDecode) @@ -217,16 +443,16 @@ private static ReadOnlyMemory ApplyOne( $"An LZW stream uses code {undefinedCode}, which it has not defined; what decoded before it was kept."), position); } - else if (ending == DataEnding.Unmarked && !given) + else if (ending == DataEnding.Unmarked && !quiet) { - // Data the reader cut ends early because the reader stopped reading it: that no code ends it says nothing - // of the file. + // Data the reader cut ends early because the reader stopped reading it, and data whose end a guard hides + // ends where the guard may: that no code ends it says nothing of the file. diagnostics?.Repair( PdfDiagnosticCodes.FilterFailed, "An LZW stream ends without its end-of-data code; it decoded to the end of its data.", position); } - else if (ending == DataEnding.UnmarkedInsideCode && !given) + else if (ending == DataEnding.UnmarkedInsideCode && !quiet) { diagnostics?.Warn( PdfDiagnosticCodes.FilterFailed, @@ -235,7 +461,8 @@ private static ReadOnlyMemory ApplyOne( } Reached(limited, name, bounds.Shared, diagnostics, position, guard, ref cut, ref limitReported); - return ApplyPredictor(decoded, step, diagnostics, position, cut); + ranOut = undefinedCode < 0 && ending is DataEnding.Unmarked or DataEnding.UnmarkedInsideCode; + return ApplyPredictor(decoded, step, diagnostics, position, cut || (silenced && ranOut)); } byte[] text; @@ -256,10 +483,12 @@ private static ReadOnlyMemory ApplyOne( text = RunLengthFilter.Decode(data.Span, out textLimited, out faults, maxLength); } - // The end of data the reader cut, or of data the filter stopped decoding at the bound, is the reader's: only what - // the filter read is the file's. - ReportFaults(faults, name, data.Length, textLimited, ended: !given && !textLimited, diagnostics, position); + // The end of data the reader cut, or whose end a guard hides, or of data the filter stopped decoding at the bound, is + // the reader's: only what the filter read is the file's. + ReportFaults(faults, name, data.Length, textLimited, ended: !quiet && !textLimited, diagnostics, position); Reached(textLimited, name, bounds.Shared, diagnostics, position, guard, ref cut, ref limitReported); + dropped = faults.Dropped; + ranOut = name == PdfName.RunLengthDecode && faults.Ending != DataEnding.Marked; return text; } @@ -274,7 +503,8 @@ private static ReadOnlyMemory ApplyOne( /// Whether the filter stopped at the bound, the faults it counts being those of the part it read. /// /// Whether the filter read its data to the end the file gave it, which its end then says something of: an end marker - /// missing, or a ~ no > follows, where the reader cut the data or the filter stopped is the cut's. + /// missing, or a ~ no > follows, where the reader cut the data, a guard hides its end, or the filter + /// stopped is the cut's. /// /// Receives the reports. /// Where the stream's data starts. diff --git a/src/AdCodicem.Pdf/IO/HeaderText.cs b/src/AdCodicem.Pdf/IO/HeaderText.cs new file mode 100644 index 00000000..f3665ede --- /dev/null +++ b/src/AdCodicem.Pdf/IO/HeaderText.cs @@ -0,0 +1,231 @@ +using System.Globalization; +using AdCodicem.Pdf.Objects; + +namespace AdCodicem.Pdf.IO; + +/// +/// One read of a search for endstream as the rule for object headers sees it (): its +/// bytes, after a few that stand for what the search read before them. +/// +/// +/// Whether obj ends an object header depends on the runs of white space and of digits before it, which may run +/// back past the start of a read, as far as the start of the data. What the reads before hold is carried into each one +/// as a few bytes that decide every header as they would: the runs of white space or of digits that end where the read +/// starts, four at most — each run of white space as one space, each run of digits as the digits of its value, or of +/// 2,147,483,648 for any larger one, which no number nor generation reaches however many digits follow —, and the byte +/// before them. A header holds four such runs, and asks of the byte before them only whether it ends a token: no run +/// reads that byte. The start of the data is carried as a space, which ends a token as white space does and holds no +/// digit. The bytes carried have negative indexes, and each records where the run it stands for starts in the file. +/// None of the sizes below bounds what the file holds (ADR 34): a run of any length is carried, as what decides a +/// header — its kind and its value. +/// +internal ref struct HeaderText +{ + /// The most runs of white space or digits before obj that decide a header: those it holds. + private const int MaxRuns = 4; + + /// The most digits a run of digits is carried as: those of . + private const int MaxDigits = 10; + + /// The most bytes carried into a read: the byte before the runs, two runs of white space and two of digits. + internal const int CarriedLength = 1 + (2 * (1 + MaxDigits)); + + /// The value a run of digits is taken for when it is larger: past any object number and generation. + private const long Beyond = (long)PdfObjectId.MaxNumber + 1; + + private readonly Span _carried; + private readonly Span _carriedAt; + private int _carriedLength; + private ReadOnlySpan _bytes; + private long _at; + + /// Starts a search at , before which nothing is read. + /// Room for bytes carried into a read. + /// Room for where the run each of them stands for starts in the file. + /// Where the data starts in the file. + public HeaderText(Span carried, Span carriedAt, long dataStart) + { + _carried = carried; + _carriedAt = carriedAt; + _carried[0] = (byte)' '; + _carriedAt[0] = dataStart; + _carriedLength = 1; + } + + /// Gets the bytes of the read. + public readonly ReadOnlySpan Bytes => _bytes; + + /// Gets the index of the first byte carried into the read: the byte before the runs, which no run reads. + private readonly int FirstCarried => -_carriedLength; + + private readonly byte this[int index] => index >= 0 ? _bytes[index] : _carried[_carriedLength + index]; + + /// Takes , read at in the file, as the read that follows what is carried. + public void Read(ReadOnlySpan bytes, long at) + { + _bytes = bytes; + _at = at; + } + + /// Gives where the byte at is in the file, or, for a byte carried, where its run starts. + public readonly long PositionOf(int index) => index >= 0 ? _at + index : _carriedAt[_carriedLength + index]; + + /// + /// Carries into the next read, which starts at in this one, the runs of white space and + /// of digits that end there, and the byte before them. + /// + public void CarryTo(long position) + { + Span runAt = stackalloc long[MaxRuns]; + Span runValue = stackalloc long[MaxRuns]; + var end = (int)(position - _at); + var runs = 0; + + while (runs < MaxRuns && end > FirstCarried + 1 && + (PdfCharacters.IsDigit(this[end - 1]) || PdfCharacters.IsWhitespace(this[end - 1]))) + { + var digits = PdfCharacters.IsDigit(this[end - 1]); + var start = RunStart(end, digits); + runAt[runs] = PositionOf(start); + runValue[runs] = digits ? Value(start, end) : -1; + end = start; + runs++; + } + + // What is carried is read whole before it is written over. + var before = this[end - 1]; + _carriedAt[0] = PositionOf(end - 1); + _carried[0] = before; + var length = 1; + + for (var run = runs - 1; run >= 0; run--) + { + var written = 1; + + if (runValue[run] < 0) + { + _carried[length] = (byte)' '; + } + else + { + // The room carried has ten digits for each run of digits, and no value carried has more. + _ = runValue[run].TryFormat(_carried[length..], out written, default, CultureInfo.InvariantCulture); + } + + _carriedAt.Slice(length, written).Fill(runAt[run]); + length += written; + } + + _carriedLength = length; + } + + /// + /// Finds the first object header whose obj ends before in the read, and gives where + /// it starts in the file; -1 when there is none. + /// + /// + /// A header is N G obj at a token boundary, as the file's bytes hold it: white space, a delimiter or the start + /// of the data before the number; a run of digits whose value the parser takes for an object number, 1 to + /// 2,147,483,647, and one whose value it takes for a generation, 0 to 65,535, however many zeros lead them; white + /// space before each of the generation and obj, as much as the file holds; and white space, a delimiter or the + /// end of the search after obj. endobj is no header, nor 10 0 objx, nor digits a regular + /// character precedes, nor tokens a comment separates. The scan that rebuilds an index refuses endobj and + /// tokens a comment separates too, but takes 10 0 objx and digits a regular character precedes (#171). + /// A header lies whole in what the search reads, from the start of the data to the next object the index places: the + /// stream's own header, and whatever precedes the data, is never one, and one that the next object's offset cuts is + /// not seen, that offset stopping the search first. The rule reads nothing but the bytes, so where it stops cannot + /// depend on what was read before; and what a read's start cuts of a header is carried into it, and obj with + /// the byte after it is whole in the read that decides it, so where the reads start cannot change it either: a + /// candidate whose obj ends the read, where the search goes on, is left to the next read, which repeats it. + /// + /// The rule is no guard (ADR 34): only a stream whose endstream does not follow its declared length is + /// searched with it, which no valid file holds (ISO 32000-1, 7.3.8.1); a cross-reference stream whose /Length + /// the chain could not read, which a valid file may hold, is searched without it (#220). A valid stream's data may + /// hold text that reads as a header — an embedded PDF written uncompressed —: its length, confirmed, is never + /// searched. An invalid stream whose data holds such text before its endstream stops there, as one whose next + /// object came first, and its data ends there or before (#174); text that only looks like a header is read past, and + /// an endstream after it ends the data. + /// + /// + /// Where the endstream found in the read starts, or its length. + /// Whether the read ends where the search does, where obj may end. + public readonly long FindHeader(int before, bool endsRange) + { + var from = 0; + + while (true) + { + var index = _bytes[from..before].IndexOf("obj"u8); + + if (index < 0) + { + return -1; + } + + var keyword = from + index; + var after = keyword + 3; + + // What follows obj past the end of the read is the next read's to see, which repeats it: the end of the read + // ends the token only where the search ends. + if ((after < _bytes.Length ? !PdfCharacters.IsRegular(_bytes[after]) : endsRange) && + TryFindStart(keyword, out var start)) + { + return PositionOf(start); + } + + from = after; + } + } + + /// + /// Reads back from the obj at for the rest of an object header, as + /// takes one, and gives where its number starts. + /// + private readonly bool TryFindStart(int keyword, out int start) + { + var generationEnd = RunStart(keyword, digits: false); + var generationStart = RunStart(generationEnd, digits: true); + var numberEnd = RunStart(generationStart, digits: false); + start = RunStart(numberEnd, digits: true); + + // Each run holds a byte at least, what precedes the number ends a token, and the values are an object number's + // and a generation's. + return generationEnd < keyword && generationStart < generationEnd && numberEnd < generationStart && + start < numberEnd && !PdfCharacters.IsRegular(this[start - 1]) && + Value(start, numberEnd) is >= 1 and <= PdfObjectId.MaxNumber && + Value(generationStart, generationEnd) <= PdfObjectId.MaxGeneration; + } + + /// + /// Gives where the run of digits, or of white space, that ends at starts, the first byte + /// carried left out. + /// + private readonly int RunStart(int end, bool digits) + { + var start = end; + + while (start > FirstCarried + 1 && + (digits ? PdfCharacters.IsDigit(this[start - 1]) : PdfCharacters.IsWhitespace(this[start - 1]))) + { + start--; + } + + return start; + } + + /// + /// Gives the value of the digits from to , or when + /// it is larger. + /// + private readonly long Value(int start, int end) + { + var value = 0L; + + for (var index = start; index < end; index++) + { + value = Math.Min((value * 10) + (this[index] - '0'), Beyond); + } + + return value; + } +} diff --git a/src/AdCodicem.Pdf/IO/IPdfFileEdge.cs b/src/AdCodicem.Pdf/IO/IPdfFileEdge.cs new file mode 100644 index 00000000..347627a2 --- /dev/null +++ b/src/AdCodicem.Pdf/IO/IPdfFileEdge.cs @@ -0,0 +1,22 @@ +namespace AdCodicem.Pdf.IO; + +/// +/// Says what the file — or an object stream's decoded data — holds right past the end of a parser's buffer, where a window's +/// edge leaves a token undecided. +/// +/// +/// A parser whose window does not end the data asks it of an endobj or an R its window's edge reaches, which +/// may be the start of a longer keyword — endobjX, RG —: an object a window ends exactly with — the last one +/// allows, say — reads whole, as under a larger window (#174, #224). +/// It is all a parser asks of the file when it reads an object for the chain without loading it, which reads no stream's +/// data; an object stream member's parser asks it of the decoded data, which is in memory. +/// +internal interface IPdfFileEdge +{ + /// + /// Determines whether a token that reaches the end of the parser's buffer at ends there: + /// the file ends there, or no regular character follows. The parser asks of an endobj or an R its window's + /// edge reaches, which may be the start of a longer keyword (#174, #224). + /// + bool TokenEndsAt(long absoluteOffset); +} diff --git a/src/AdCodicem.Pdf/IO/IPdfStreamDataProvider.cs b/src/AdCodicem.Pdf/IO/IPdfStreamDataProvider.cs index ccc4e942..25654573 100644 --- a/src/AdCodicem.Pdf/IO/IPdfStreamDataProvider.cs +++ b/src/AdCodicem.Pdf/IO/IPdfStreamDataProvider.cs @@ -11,51 +11,96 @@ namespace AdCodicem.Pdf.IO; /// provider is also the parser's way to the rest of the file, for what its buffer does not hold: the bytes /// after a stream's data, and the objects a stream's /Length names. /// -internal interface IPdfStreamDataProvider +internal interface IPdfStreamDataProvider : IPdfFileEdge { /// Gets the length of the file the stream data is read from. long SourceLength { get; } - /// Creates stream data for bytes at . - PdfStreamData Create(long absoluteOffset, int length); + /// + /// Creates stream data for bytes at , marked cut by a guard + /// when : a guard hid what would have said where the data ends (#224); or, with + /// , the data's, its end only the guard's to hide. + /// + PdfStreamData Create(long absoluteOffset, int length, bool cutByGuard = false, bool endHidden = false); /// - /// Determines whether an endstream keyword starts at , after at - /// most a few bytes of white space. The parser asks when that keyword may lie past the end of its buffer. + /// Gives where the data of a stream starts in the file: past the end-of-line after its stream keyword, which ends + /// at . The parser asks when its buffer ends on the keyword, or on the carriage + /// return after it, and so cannot say: a window a guard ended there is not grown for it, since stream data counts + /// toward no window (#174). /// - bool IsEndStreamAt(long absoluteOffset); + /// + /// Where the data starts, or -1 when a regular character follows the keyword, which makes it a longer one, as a larger + /// window says: the parser asks for one. + /// + long DataStart(long absoluteKeywordEnd); /// - /// Determines whether an endstream keyword starts at , as - /// does, and what follows it. The parser asks at the end of a stream whose data runs - /// past its buffer, where it can see neither. + /// Determines whether an endstream keyword follows , after white space and + /// comments, however many (#174), and what follows it. The parser asks at the end of a stream's declared length when its + /// buffer cannot say: the data runs past it, or the white space after it does. /// - /// Null when no endstream starts there; what follows it otherwise. + /// + /// The file is read on as far as the white space and comments run, the reads past the first counted with the searches + /// for endstream (); once those have read as much as they may, the + /// first read alone decides. + /// + /// Null when no endstream follows; what follows it otherwise. EndObjState? CheckEndStream(long absoluteOffset); /// - /// Looks for where the data of a stream that runs past the parser's buffer ends, when no endstream follows - /// its declared length: the first endstream after the start of the data, before the next object the file's - /// index places — as the file wrote it, or as the reader rebuilt it as the document opened when the file wrote none —, - /// before the first object header the file's bytes hold, and before the end of the file. + /// Looks for where the data of a stream ends when no endstream follows its declared length, or it has none: the + /// first endstream after the start of the data, in the bytes the parser holds and then in the file, before the + /// next object the file's index places — as the file wrote it, or as the reader rebuilt it as the document opened when + /// the file wrote none —, before the first object header the file's bytes hold, and before the end of the file. /// /// /// The file is searched once for each stream: asked again for the data at the same place — the stream parsed /// again, after the cache let it go or the index was rebuilt —, the provider answers what it found the first time. + /// When it finds no endstream, it says what the object's endobj does instead (#174): whether one follows + /// the declared length, where the one that ends the object before the stop lies, and whether the stretch holds one. /// /// /// The number of the object the stream is the value of, whose own entry in the index is no next object; 0 for a /// stream read alone. /// /// Where the data starts in the file. + /// Where the declared length ends in the file, or -1 when the stream has none it can take. /// The bytes of the data the parser holds, from its start, which are searched first. - StreamEndSearch FindEndStream(int number, long absoluteDataStart, ReadOnlySpan buffered); + /// + /// Whether the stream is a cross-reference stream whose /Length the chain could not read (#182): its rows may + /// spell an object header and end in a carriage return, so neither stops nor trims the search (#220). + /// + StreamEndSearch FindEndStream(int number, long absoluteDataStart, long declaredEnd, ReadOnlySpan buffered, bool deferred); + + /// + /// Determines whether the endstream that ends data at , which a search found, + /// is the stream's own beyond what its length could say: what follows its object's end — its endobj, where + /// says one follows it, or the endstream itself otherwise —, white space and comments + /// between, is the next object the index places, — there, or past the white space and + /// comments a row names before its header —, a cross-reference stream the chain read, the end of the file, or the + /// xref or startxref keyword. + /// Anything else may follow an endstream the data spells, which only the length tells from the stream's own + /// (#224). + /// + /// + /// Read once for each place, as far as the searches for endstream may read; past that, nothing is proven. + /// + bool IsOwnEndStream(long absoluteDataEnd, EndObjState endObj, long? nextObject); + + /// + /// Determines whether data a search ended at , at an endstream, may end + /// elsewhere among the white space before the keyword: white space lies between them, which the search left out, or the + /// data's last byte is white space. Only a length says which (#224). + /// + bool EndsBesideWhiteSpace(long absoluteDataEnd); /// /// Resolves the object a stream's /Length names, and says whether the file holds it and whether it could be - /// read — a reference to the stream itself, or to an object inside an object stream being decoded, cannot be. + /// read — a reference to the stream itself, or to an object inside an object stream being decoded, cannot be —, or + /// whether a guard cut it, or the object a chain of references from it leads to (#224). /// - PdfObject ResolveLength(PdfObjectId id, out ObjectPresence presence); + PdfObject ResolveLength(PdfReference reference, out ObjectPresence presence); /// /// Determines whether what is wrong with the length of the stream whose data starts at diff --git a/src/AdCodicem.Pdf/IO/ObjectPresence.cs b/src/AdCodicem.Pdf/IO/ObjectPresence.cs index 7f89e7ca..026f16ac 100644 --- a/src/AdCodicem.Pdf/IO/ObjectPresence.cs +++ b/src/AdCodicem.Pdf/IO/ObjectPresence.cs @@ -31,4 +31,12 @@ internal enum ObjectPresence : byte /// Whether the file holds the object is unknown, and a reference to it reads as null until the guard is lifted. /// Unindexed, + + /// + /// The object ran past one of the reader's guards, which cut it: what it holds may lie past the cut, and what was read + /// of it — null, or a token the cut parted — is the guard's until the guard is lifted (#224). Only a stream's + /// /Length asks: says whether the file holds the object, and + /// whether a guard cut it. + /// + Cut, } diff --git a/src/AdCodicem.Pdf/IO/PdfFileReader.cs b/src/AdCodicem.Pdf/IO/PdfFileReader.cs index 48aa5af9..b2358db8 100644 --- a/src/AdCodicem.Pdf/IO/PdfFileReader.cs +++ b/src/AdCodicem.Pdf/IO/PdfFileReader.cs @@ -37,7 +37,11 @@ internal sealed class PdfFileReader : IPdfObjectSource, IPdfStreamDataProvider /// What the codes of the faults of the object syntax start with. private const string SyntaxCodePrefix = "syntax."; - /// How much of a section is read to tell a classic table from a cross-reference stream. + /// + /// How much of a section is read first to tell a classic table from a cross-reference stream: a read that holds + /// white space and comments alone, or a first token its edge may cut, is read again eight times larger, as far as a + /// trailer's window reaches. + /// internal const int XRefProbeLength = 32; /// @@ -140,23 +144,50 @@ internal sealed class PdfFileReader : IPdfObjectSource, IPdfStreamDataProvider /// /// How many times the length of the file the searches for endstream of one document may read, together, - /// before a stream whose declared length no endstream follows keeps it without a search. + /// before a stream whose declared length no endstream follows keeps it without a search past the bytes the + /// parser holds, and white space after a declared length is read no further than a first read. /// /// - /// Not a guard a valid file reaches (ADR 34): a valid file is never searched. Each search reads from a stream's - /// data to the first object header the bytes hold after it, or to the next object the index as written places, - /// whichever is nearer, and its reads grow from twice what the parser holds of the data - /// (): it reads at most about twice what its object spans, and the searches of - /// objects whose headers follow one another read disjoint stretches — the file twice over at most. Only searches that - /// share a stretch read more: objects that overlap, the header of one inside the dictionary of another, before its - /// data; or a stream whose header the search does not take — a regular character glued before its number, where a - /// rebuild's scan takes the digits after it for one — and the index as written does not place, which only a rebuilt - /// one does. This keeps what they read in proportion to the file, as a window keeps the parsing of their - /// dictionaries. Which streams keep their declared length unsearched past it depends on the order they are read in: - /// crafted files reach it, and no document of the corpus comes near, whose searches read 0.35 of a file at most. + /// Not a guard a valid file reaches (ADR 34), argued here once for every read it counts. A valid file is read past + /// the parser's window in four ways only. Three read the data or what follows it: a cross-reference stream whose + /// /Length the chain cannot read (#220) is searched once to its endstream; a stream whose + /// /Length a guard cut or left out of the index (#224) is searched once to its endstream, or to a + /// header its data spells, which leaves the data up to it to the guard; and white space after a declared length + /// that runs past a first read is read on once for each place, and past an endstream once for each keyword + /// however many places lead to it (#174). The data of different streams lies apart, and so do the runs of white + /// space after them: together those reads cover the file once, and one read past each — about twice the file —, but + /// where one run is two ways' at once: the white space before the endstream of a cross-reference stream whose + /// /Length the chain cannot read is the data its search takes and the run its length, once read, is read on + /// through, about three times. The fourth adds a little to each stream of the second way: the proof that an endstream found is the + /// stream's own (#224) reads past it to its object's end and what follows, three first reads of + /// bytes, and a fourth past a row that names the white space before the next object's + /// header, each once however often the stream is parsed — 48 or 64 bytes, about what the smallest such stream spans + /// with its header. White + /// space it crosses adds to that: the run before the object's endobj, or before what follows the keyword where + /// no endobj does, it takes from the read-on that read it, the run after an endobj it doubles through, as + /// the read-on does, and the run a row names before the next header it + /// reads once from there, a first read reaching the header. No run is read more than three times, and the file + /// about three times at most in all, within the four the searches may read. Every other search is of an invalid + /// stream, whose endstream does not follow its declared length. Each reads from a stream's data to the first + /// object header the bytes hold after it, or to the next object the index as written places, whichever is nearer, + /// and its reads grow from twice what the parser holds of the data (): it reads + /// at most about twice what its object spans, and the searches of objects whose headers follow one another read + /// disjoint stretches — the file twice over at most. Only searches that share a stretch read more: objects that + /// overlap, the header of one inside the dictionary of another, before its data; or a stream whose header the + /// search does not take — a regular character glued before its number, where a rebuild's scan takes the digits + /// after it for one — and the index as written does not place, which only a rebuilt one does. This keeps what they + /// read in proportion to the file, as a window keeps the parsing of their dictionaries. Which streams keep their + /// declared length unsearched past it depends on the order they are read in: crafted files reach it, and no + /// document of the corpus comes near, whose searches read 0.35 of a file at most. /// internal const int EndStreamSearchPasses = 4; + /// + /// How many bytes the first read of each step of an endstream's proof reads (): + /// what follows a keyword is mostly an end-of-line away, and the reads double past it as white space runs on. + /// + private const int ProofRead = 16; + /// Asks for a direct object, with no object header: a trailer. private const int DirectObject = -1; @@ -175,11 +206,10 @@ internal sealed class PdfFileReader : IPdfObjectSource, IPdfStreamDataProvider /// parsing of the object it loads besides. No level holds decoded data while it loads the next: a stream resolves /// its filters and every parameter they read before its first filter runs (), so /// that a chain of loads holds at once one stream's decoding, and the object streams - /// keeps. One thing remains: each object stream loaded on the chain stays in the object - /// cache with the raw data it read, until the cache lets it go (#170). A stream an object stream holds, which ISO - /// 32000-1 (7.5.7) forbids, is no exception: it is no value, and its /Length is not read (#161). The count - /// alone does not bound the stack, since each load parses as deep as the parser allows: the stack is asked as well - /// before each load. + /// keeps; no stream keeps the raw data it read (#170). A stream an object stream + /// holds, which ISO 32000-1 (7.5.7) forbids, is no exception: it is no value, and its /Length is not read + /// (#161). The count alone does not bound the stack, since each load parses as deep as the parser allows: the stack + /// is asked as well before each load. /// private const int MaxNestedLoads = 64; @@ -190,6 +220,14 @@ internal sealed class PdfFileReader : IPdfObjectSource, IPdfStreamDataProvider /// private const int MaxReferenceLinks = 31; + /// + /// How many references are followed from a value read from the object a reference names and then resolved, as a stream's + /// /Length is parsed and a trailer's /Prev and /XRefStm are read: one more than + /// , the first object's value resolving through as many as a reference does. No valid file + /// reaches it either. + /// + private const int LengthLinks = MaxReferenceLinks + 1; + private static ReadOnlySpan ObjKeyword => "obj"u8; private static ReadOnlySpan TrailerKeyword => "trailer"u8; private static ReadOnlySpan StartXRefKeyword => "startxref"u8; @@ -262,10 +300,21 @@ internal sealed class PdfFileReader : IPdfObjectSource, IPdfStreamDataProvider /// /// The objects could not read while the chain is read because they run past - /// , or null while none did: what they hold is the guard's to hide. Let go - /// with . + /// , each with the integer the cut left of its value, or + /// when it left none; null while none did. What they hold is the guard's to hide, and the integer + /// only a place to look for endstream at (#224), read once however many sections ask. One whose value the cut left + /// a whole reference is read through, and is not among them. Let go with . /// - private HashSet? _cutWhileChainIsRead; + private Dictionary? _cutWhileChainIsRead; + + /// + /// The objects a guard cut that a cross-reference stream's values led to, which the chain could not read and ended at, and + /// those it read through, the cut having left their value a whole reference — but for an object it read at a place the + /// section may have been moved to, and refused; what the offset the chain names holds keeps all of its reading —: each + /// is loaded once the chain is read, so that its guard is reached where a load reaches it (#263). + /// Let go then, or when the chain is rebuilt. + /// + private List? _chainValuesCut; /// /// The object numbers whose rows a guard left unread in the cross-reference stream just read — past the data it cut, or @@ -351,10 +400,45 @@ internal sealed class PdfFileReader : IPdfObjectSource, IPdfStreamDataProvider private readonly Dictionary _streamLengthFaults = []; /// - /// What each search of the file for a stream's endstream found, by where the stream's data starts: the file - /// is searched once for each stream, however many times it is parsed. + /// What each search of the file for a stream's endstream found, by where the stream's data starts and whether + /// its /Length was one the chain could not read: the file is searched once for each stream, however many times + /// it is parsed. With it, the declared end it last said what follows of, read again only when that end changes. + /// + private readonly Dictionary<(long DataStart, bool Deferred), (StreamEndSearch Search, long DeclaredEnd)> _endStreamSearches = []; + + /// + /// What follows each place the reader read on past white space and comments to find an endstream (#174): read + /// once, however many times its stream is parsed. + /// + private readonly Dictionary _endStreamChecks = []; + + /// + /// Where the first significant byte lies past each endstream keyword the reader read on past (#174), an + /// endobj there or not: another place that leads to the keyword, and the proof that the keyword ends the stream's + /// own data (#224), take it rather than read the white space again. + /// + private readonly Dictionary _endObjStarts = []; + + /// + /// What follows the object's end after each place a search ended data at, with what was said follows the + /// endstream there, as reads it: where its first significant + /// byte lies, -1 where nothing could be read, and whether the end of the file or a cross-reference section's keyword + /// stands there. Read once, however many times its stream is parsed; each caller's next object is weighed against it. + /// + private readonly Dictionary<(long DataEnd, EndObjState EndObj), (long Position, bool Boundary)> _endStreamFollowers = []; + + /// + /// Where the first significant byte lies past each offset a row names before the place it is weighed against, as + /// reads it: -1 where nothing could be read. Read once, however many times its stream is + /// parsed. + /// + private readonly Dictionary _placedStarts = []; + + /// + /// Where the first significant byte lies past each place a search ended data at, the endstream keyword it found: + /// -1 where nothing could be read. Read once, however many times its stream is parsed. /// - private readonly Dictionary _endStreamSearches = []; + private readonly Dictionary _endStreamKeywords = []; /// /// Where the data starts of each stream whose length fault a reading the reader kept reported: the stream is not @@ -688,8 +772,11 @@ public PdfObject GetObject(PdfObjectId id) /// /// The object is read within , as a load reads it; one the guard cuts is /// left unread, and the guard is neither reported nor counted: the chain's sections are cut only by their own windows - /// and decoding, and a guard that holds is reached when the object is loaded once the chain is read. The objects read - /// through it nest as loads do, within . + /// and decoding, and a guard that holds is reached when the object is loaded once the chain is read. One whose value + /// the cut left a whole reference reads as that reference, as a load reads through it, and is loaded once the chain is + /// read for its guard to be reached — unless what read it was a place the section may have been moved to, which the + /// chain dropped (). The objects read through it nest as loads do, within + /// . /// /// private PdfObject? ReadWithoutLoading(PdfObjectId id) @@ -717,12 +804,19 @@ public PdfObject GetObject(PdfObjectId id) return null; } - var read = TryReadUnrecorded(id, PositionOf(entry), out var value, out var pastGuard) ? value : null; + // A reference the cut left whole leads on, as a load reads through it (#224): the guard hides nothing the chain + // needs of the object, and is reached once the chain is read, where loading the object reaches it. + var parsed = TryReadUnrecorded(id, PositionOf(entry), out var value, out var pastGuard); + var read = parsed || (pastGuard && value is PdfReference) ? value : null; (_readWhileChainIsRead ??= [])[id] = read; - if (pastGuard) + if (pastGuard && read is null) + { + (_cutWhileChainIsRead ??= [])[id] = value.AsInteger() is null ? PdfNull.Instance : value; + } + else if (pastGuard) { - (_cutWhileChainIsRead ??= []).Add(id); + (_chainValuesCut ??= []).Add(id); } return read; @@ -733,6 +827,25 @@ public PdfObject GetObject(PdfObjectId id) } } + /// + /// Reads, while the chain is read, what the /Length object leads to through as many + /// references as the parse resolves a /Length through (), each read with + /// ; null when one of them cannot be read — no section read so far places it, or a guard + /// cut it —: a value whose chain leads there waits for the chain, as one the first reference cannot read does. A value + /// still a reference past them is one the parse reads as null. + /// + private PdfObject? ReadThroughWithoutLoading(PdfObjectId id) + { + var read = ReadWithoutLoading(id); + + for (var link = 1; read is PdfReference next && link < LengthLinks; link++) + { + read = ReadWithoutLoading(next.Id); + } + + return read; + } + /// /// Says whether the file holds object and whether the reader could produce it, as /// of the last time it was asked for: call it after resolving the object. @@ -783,6 +896,12 @@ internal ObjectPresence GetPresence(int number) /// internal bool ChainStoppedByGuard => _chainFilledIndex || _cutByGuard; + /// + /// Gets the most filters a stream's data passes through as the reader decodes it + /// (): the chain ends at the element past them. + /// + internal int FilterChainBound => _guard.Bound(PdfLimit.FilterChain); + /// /// Gets where the index first had no room for an object number, reached — /// as the document opened, or in a rebuild run since —, or -1 while it had room for every number it met. @@ -1368,6 +1487,26 @@ private static bool Consume(HeaderReading reading, ref long consumed, long end) /// internal bool IsCutAtLimit(int number) => _cutAtLimit.Contains(number); + /// + /// Gets how many bytes the searches for endstream, and the proofs that what they found is a stream's own, have + /// read so far: what bounds. + /// + internal long EndStreamSearchBytes => _endStreamSearchBytes; + + /// + /// Determines whether the object leads to — through a chain of references too, as + /// follows it — is one a guard cut, or left out of the index: what it holds may lie + /// past the cut, or is unknown, and is the guard's until the guard is lifted, rather than a value the file gave + /// (#224, #263). Call it once the reference is resolved. + /// + /// The reference, as the file wrote it. + /// How many objects its chain is followed through (). + internal bool LeftByGuard(PdfReference reference, int objects = PdfReference.MaxChainLength - 1) + { + var target = reference.Target(objects).Number; + return IsCutAtLimit(target) || GetPresence(target) == ObjectPresence.Unindexed; + } + /// /// Determines whether object , as it was read, is a stream whose length the file did not /// confirm, and gives what the reader found of it. @@ -1484,42 +1623,237 @@ private long EntryPosition(XRefEntry entry) long IPdfStreamDataProvider.SourceLength => _source.Length; /// - PdfStreamData IPdfStreamDataProvider.Create(long absoluteOffset, int length) + PdfStreamData IPdfStreamDataProvider.Create(long absoluteOffset, int length, bool cutByGuard, bool endHidden) { // A declared length is a claim made by the file, so it is clamped to what the file can hold. // Without this, "/Length 2147483647" would ask for two gigabytes of memory. var available = (int)Math.Clamp(_source.Length - absoluteOffset, 0, int.MaxValue); - return new FileStreamData(this, absoluteOffset, Math.Clamp(length, 0, available), _guard); + var data = new FileStreamData(this, absoluteOffset, Math.Clamp(length, 0, available), _guard); + + if (cutByGuard) + { + data.MarkCutByGuard(); + } + + if (endHidden) + { + data.MarkEndHidden(); + } + + return data; } /// - bool IPdfStreamDataProvider.IsEndStreamAt(long absoluteOffset) + bool IPdfFileEdge.TokenEndsAt(long absoluteOffset) { - // A source is only asked for bytes it has, as GetWindow does: a subclass need not accept an offset - // or a length that runs past its end. - if (absoluteOffset < 0 || absoluteOffset >= _source.Length) + Span next = stackalloc byte[1]; + return absoluteOffset < 0 || absoluteOffset >= _source.Length || _source.Read(absoluteOffset, next) == 0 || + !PdfCharacters.IsRegular(next[0]); + } + + /// + long IPdfStreamDataProvider.DataStart(long absoluteKeywordEnd) + { + // The end-of-line after "stream" is a carriage return and a line feed, or either alone, as the lexer skips it. + Span endOfLine = stackalloc byte[2]; + var read = absoluteKeywordEnd < 0 || absoluteKeywordEnd >= _source.Length + ? 0 + : _source.Read(absoluteKeywordEnd, endOfLine[..(int)Math.Min(endOfLine.Length, _source.Length - absoluteKeywordEnd)]); + if (read > 0 && PdfCharacters.IsRegular(endOfLine[0])) { - return false; + return -1; + } + + var skipped = read > 0 && endOfLine[0] == (byte)'\r' ? 1 : 0; + + if (skipped < read && endOfLine[skipped] == (byte)'\n') + { + skipped++; } - Span tail = stackalloc byte[PdfObjectParser.EndStreamLookahead]; - var available = (int)Math.Min(tail.Length, _source.Length - absoluteOffset); - var read = _source.Read(absoluteOffset, tail[..available]); - return PdfObjectParser.IsEndStreamAt(tail[..read], 0); + return absoluteKeywordEnd + skipped; } /// EndObjState? IPdfStreamDataProvider.CheckEndStream(long absoluteOffset) { + // A source is only asked for bytes it has, as GetWindow does: a subclass need not accept an offset + // or a length that runs past its end. if (absoluteOffset < 0 || absoluteOffset >= _source.Length) { return null; } + // One read decides for all but a stream whose white space runs past it: a few bytes, on the stack. + Span tail = stackalloc byte[PdfObjectParser.EndObjLookahead]; + var read = _source.Read(absoluteOffset, tail[..(int)Math.Min(tail.Length, _source.Length - absoluteOffset)]); + var end = StreamEnds.ReadEnd(tail[..read], absoluteOffset + read >= _source.Length); + + return end.Kind switch + { + StreamEndKind.None => null, + StreamEndKind.EndStream when end.EndObj != EndObjState.Unseen => end.EndObj, + _ => CheckEndStreamOn(absoluteOffset, end), + }; + } + + /// + /// Reads on, past what one read of holds, through the white space + /// and comments that may lead to an endstream at , or to what follows it: once + /// for each place, however often its stream is parsed. + /// + /// Where the data ends. + /// What the first read said: undecided, or an endstream after which it saw nothing. + private EndObjState? CheckEndStreamOn(long absoluteOffset, StreamEnd first) + { + if (_endStreamChecks.TryGetValue(absoluteOffset, out var known)) + { + return known; + } + + EndObjState? state; + var keywordEnd = absoluteOffset + first.KeywordEnd; + + if (first.Kind == StreamEndKind.Undecided) + { + var at = NextSignificant(absoluteOffset); + Span keyword = stackalloc byte[StreamEnds.EndStreamKeyword.Length + 1]; + var read = at < 0 ? 0 : _source.Read(at, keyword[..(int)Math.Min(keyword.Length, _source.Length - at)]); + + if (!keyword[..read].StartsWith(StreamEnds.EndStreamKeyword)) + { + // Once the searches have read as much as they may, a run the first read could not see past confirms nothing. + _endStreamChecks[absoluteOffset] = null; + return null; + } + + // "endstreamx" is no keyword, and is what follows the data. + if (read == keyword.Length && PdfCharacters.IsRegular(keyword[^1])) + { + _endStreamChecks[absoluteOffset] = EndObjState.Absent; + return EndObjState.Absent; + } + + keywordEnd = at + StreamEnds.EndStreamKeyword.Length; + } + + // Two places may lead to one keyword — a cross-reference stream's data, whose carriage return is kept, and the + // length that ends before it —: what follows the keyword is read once. + if (!_endObjStarts.TryGetValue(keywordEnd, out var next)) + { + next = NextSignificant(keywordEnd); + + if (next >= 0) + { + _endObjStarts[keywordEnd] = next; + } + } + + state = next < 0 ? EndObjState.Unseen : IsEndObjKeywordAt(next) ? EndObjState.Present : EndObjState.Absent; + _endStreamChecks[absoluteOffset] = state; + return state; + } + + /// + /// Determines whether an endobj keyword, whole, follows the declared length of a stream whose data starts at + /// , white space and comments between, a comment only after white space: in the bytes the + /// parser holds when they say, in the file otherwise. + /// + private bool EndObjFollows(long dataStart, long declaredEnd, ReadOnlySpan buffered) => + (declaredEnd - dataStart <= buffered.Length + ? StreamEnds.IsEndObjAt(buffered, (int)(declaredEnd - dataStart), reachesEndOfFile: dataStart + buffered.Length >= _source.Length) + : null) ?? IsEndObjAt(declaredEnd); + + /// + /// Determines whether an endobj keyword, whole, follows , where a stream's declared + /// length ends, after white space and comments, a comment only after white space (): + /// the end of a stream's object that has no endstream. + /// + private bool IsEndObjAt(long absoluteOffset) + { + if (absoluteOffset < 0 || absoluteOffset >= _source.Length) + { + return false; + } + Span tail = stackalloc byte[PdfObjectParser.EndObjLookahead]; - var available = (int)Math.Min(tail.Length, _source.Length - absoluteOffset); - var read = _source.Read(absoluteOffset, tail[..available]); - return PdfObjectParser.ReadStreamEnd(tail[..read], absoluteOffset + read >= _source.Length); + var read = _source.Read(absoluteOffset, tail[..(int)Math.Min(tail.Length, _source.Length - absoluteOffset)]); + + if (StreamEnds.IsEndObjAt(tail[..read], 0, absoluteOffset + read >= _source.Length) is { } decided) + { + return decided; + } + + var next = NextSignificant(absoluteOffset); + return next >= 0 && IsEndObjKeywordAt(next); + } + + /// + /// Gives where the first byte at or after lies that is neither white space nor part of a + /// comment, reading the file on as far as they run, or the length of the file when they run to its end; -1 when the + /// searches for endstream have read as much as they may (), which these reads + /// count with, or the source gives less than it holds. The first read is bytes, and each + /// after it twice the one before, up to . + /// + private long NextSignificant(long position, int firstRead = PdfObjectParser.EndObjLookahead) + { + var size = firstRead; + var inComment = false; + + while (position < _source.Length) + { + if (_endStreamSearchBytes >= EndStreamSearchPasses * _source.Length) + { + return -1; + } + + using var window = _source.GetWindow(position, (int)Math.Min(size, _source.Length - position)); + var span = window.Memory.Span; + _endStreamSearchBytes += span.Length; + + if (span.IsEmpty) + { + return -1; + } + + for (var i = 0; i < span.Length; i++) + { + var current = span[i]; + + if (inComment) + { + // A comment runs to the end of the line; both line endings occur in the wild. + inComment = current is not ((byte)'\r' or (byte)'\n'); + } + else if (current == (byte)'%') + { + inComment = true; + } + else if (!PdfCharacters.IsWhitespace(current)) + { + return position + i; + } + } + + position += span.Length; + size = (int)Math.Min(2L * size, EndStreamSearchChunk); + } + + return _source.Length; + } + + /// Determines whether endobj, a whole keyword, starts at . + private bool IsEndObjKeywordAt(long position) + { + if (position >= _source.Length) + { + return false; + } + + Span bytes = stackalloc byte[StreamEnds.EndObjKeyword.Length + 1]; + var read = _source.Read(position, bytes[..(int)Math.Min(bytes.Length, _source.Length - position)]); + return bytes[..read].StartsWith(StreamEnds.EndObjKeyword) && + (read == StreamEnds.EndObjKeyword.Length ? position + read >= _source.Length : !PdfCharacters.IsRegular(bytes[^1])); } /// @@ -1546,58 +1880,236 @@ bool IPdfStreamDataProvider.IsEndStreamAt(long absoluteOffset) /// /// /// This bound is no guard (ADR 34): a valid file's endstream follows its declared length (ISO 32000-1, - /// 7.3.8.1) — so do those of the 35,873 streams of the corpus whose length is confirmed, a gap of 0 to 2 bytes - /// between them —, and only an invalid file is searched at all. What it reads is the stretch between the stream and - /// the next object, once for each stream: the result is kept, and a stream parsed again — after the cache let it go, - /// or a rebuild of the index — is not searched again. What the searches read together is bounded by + /// 7.3.8.1) — so do those of the 35,716 streams of the corpus whose length is confirmed, a gap of 0 to 2 bytes + /// between them, counted over the final index of each of its 401 documents —, and only an invalid stream is searched, + /// but for a cross-reference stream whose /Length the chain could not read, which is searched with neither + /// stop the rule for headers makes nor a carriage return trimmed, as : its rows may spell + /// a header, and end in a carriage return; and for a stream whose /Length a guard cut or left out of the index, + /// which a valid file may hold, searched as an invalid one is: a header its data spells — an embedded file's does — + /// leaves the data up to it to the guard (#224). What it reads is the stretch between the stream and the next object, + /// once for each stream: the result is kept, and a stream parsed again — after the cache let it go, or a rebuild of + /// the index — is not searched again. What the searches read together is bounded by /// . /// /// - StreamEndSearch IPdfStreamDataProvider.FindEndStream(int number, long absoluteDataStart, ReadOnlySpan buffered) + StreamEndSearch IPdfStreamDataProvider.FindEndStream( + int number, long absoluteDataStart, long declaredEnd, ReadOnlySpan buffered, bool deferred) + { + var found = _endStreamSearches.TryGetValue((absoluteDataStart, deferred), out var known); + + if (found && known.DeclaredEnd == declaredEnd) + { + return known.Search; + } + + var search = found ? known.Search : SearchEndStreamOnce(number, absoluteDataStart, buffered, deferred); + + // Whether the object's endobj follows the declared length depends on the length the stream is read with now, which a + // rebuild can change — the /Length an object another definition replaces —: it is read again when that length does, + // and the search, whose result the bytes and the index as written alone decide, is not. + if (search.Length is null && search.Searched) + { + search = search with + { + EndObjAfterDeclared = declaredEnd >= absoluteDataStart && declaredEnd <= (search.NextObject ?? _source.Length) && + EndObjFollows(absoluteDataStart, declaredEnd, buffered), + }; + } + + _endStreamSearches[(absoluteDataStart, deferred)] = (search, declaredEnd); + return search; + } + + /// + /// Searches, once for each stream, the data that starts at for its first + /// endstream, and reads what its object's endobj does when none is found; what follows the declared length + /// is left to . + /// + private StreamEndSearch SearchEndStreamOnce(int number, long absoluteDataStart, ReadOnlySpan buffered, bool deferred) + { + var nextObject = NextObjectStart(number, absoluteDataStart); + + // Once the searches have read as much of the file as they may, the bytes the parser holds are searched alone: they + // cost no read. + var readsFile = _endStreamSearchBytes < EndStreamSearchPasses * _source.Length; + var dataEnd = SearchEndStream( + absoluteDataStart, + buffered, + nextObject ?? _source.Length, + deferred, + readsFile, + out var header, + out var searched, + out var endObjSeen, + out var objectEnd); + + // A header the bytes hold is looked for only before the next object the index places: found, it is the nearer. + var stop = header >= 0 ? header : nextObject; + + // Past int.MaxValue bytes, no length the reader holds a stream by reaches the endstream found: only a file of more + // than 2 GB can put one there, and the stream is read as one whose endstream is not found. With none, what the + // object's endobj does is read once with the search, as long as the searches may read. + return dataEnd >= 0 && dataEnd - absoluteDataStart <= int.MaxValue + ? new StreamEndSearch( + (int)(dataEnd - absoluteDataStart), + stop, + ((IPdfStreamDataProvider)this).CheckEndStream(dataEnd) ?? EndObjState.Unseen) + : new StreamEndSearch(null, stop, EndObjState.Unknown, searched) + { + EndObjSeen = endObjSeen, + ObjectEnd = objectEnd, + HeaderStopped = header >= 0, + }; + } + + /// + bool IPdfStreamDataProvider.IsOwnEndStream(long absoluteDataEnd, EndObjState endObj, long? nextObject) + { + var (position, boundary) = EndStreamFollower(absoluteDataEnd, endObj); + return position >= 0 && (boundary || IsPlacedAt(position, nextObject) || IsStreamSectionAt(position)); + } + + /// + bool IPdfStreamDataProvider.EndsBesideWhiteSpace(long absoluteDataEnd) + { + // The keyword is found as the proof found it, once for each place: past the data's end, white space or comments. + if (absoluteDataEnd <= 0 || absoluteDataEnd > _source.Length) + { + return false; + } + + Span last = stackalloc byte[1]; + return SignificantAfterData(absoluteDataEnd) != absoluteDataEnd || + (_source.Read(absoluteDataEnd - 1, last) == 1 && PdfCharacters.IsWhitespace(last[0])); + } + + /// + /// Gives where the first significant byte lies past , where a search ended data: the + /// endstream keyword it found, past the white space and comments it left out; -1 where nothing could be read. + /// Read once for each place, counted with the searches. + /// + private long SignificantAfterData(long dataEnd) + { + if (!_endStreamKeywords.TryGetValue(dataEnd, out var keyword)) + { + keyword = NextSignificant(dataEnd, ProofRead); + _endStreamKeywords[dataEnd] = keyword; + } + + return keyword; + } + + /// + /// Determines whether the object of a cross-reference stream the chain read starts at : the + /// section places it, though no row of its own does — PDFBox writes it so (#224). + /// + /// + /// The sections are as many as lets the chain read, and each says where + /// its object's header lies, past the white space and comments before it: nothing is read. + /// + private bool IsStreamSectionAt(long position) + { + foreach (var section in _structure.Sections) + { + if (section.Kind == XRefSectionKind.Stream && section.StreamObjectNumber > 0 && section.Offset + section.Padding == position) + { + return true; + } + } + + return false; + } + + /// + /// Reads, once for each place, what follows the end of the object whose data a search ended at + /// : past the endstream keyword, whole, and the endobj after it where + /// says one follows, white space and comments between, through first reads of + /// bytes counted with the searches (). + /// + private (long Position, bool Boundary) EndStreamFollower(long dataEnd, EndObjState endObj) { - if (_endStreamSearches.TryGetValue(absoluteDataStart, out var known)) + if (_endStreamFollowers.TryGetValue((dataEnd, endObj), out var known)) { return known; } - StreamEndSearch search; + // "endstreamxref" is no keyword, and ends no data: what follows it proves nothing. + var keyword = SignificantAfterData(dataEnd); + var end = keyword >= 0 && keyword < _source.Length && IsKeywordAt(keyword, StreamEnds.EndStreamKeyword) + ? keyword + StreamEnds.EndStreamKeyword.Length + : -1; - if (_endStreamSearchBytes >= EndStreamSearchPasses * _source.Length) + if (end >= 0 && endObj == EndObjState.Present) { - search = new StreamEndSearch(null, null, EndObjState.Unknown, Searched: false); + end = _endObjStarts.TryGetValue(end, out var endObjAt) ? endObjAt : NextSignificant(end, ProofRead); + end = end < 0 || end >= _source.Length ? -1 : end + StreamEnds.EndObjKeyword.Length; } - else + + // What follows a keyword no endobj follows the reader read on to already, through the white space between. + var next = end < 0 ? -1 : endObj != EndObjState.Present && _endObjStarts.TryGetValue(end, out var after) + ? after + : NextSignificant(end, ProofRead); + var follower = (next, next >= 0 && + (next == _source.Length || IsKeywordAt(next, "xref"u8) || IsKeywordAt(next, StartXRefKeyword))); + _endStreamFollowers[(dataEnd, endObj)] = follower; + return follower; + } + + /// + /// Determines whether the object an index places at starts at : + /// there, or past the white space and comments its row names, as the reader reads an object at a row's offset. + /// + /// + /// What lies past an offset is read once, counted with the searches, however many times its stream is parsed. + /// + private bool IsPlacedAt(long position, long? placed) + { + if (placed is not { } at || at > position) { - var nextObject = NextObjectStart(number, absoluteDataStart); - var dataEnd = SearchEndStream(absoluteDataStart, buffered, nextObject ?? _source.Length, out var header); + return false; + } - // A header the bytes hold is looked for only before the next object the index places: found, it is the nearer. - var stop = header >= 0 ? header : nextObject; + if (at == position) + { + return true; + } - // Past int.MaxValue bytes, no length the reader holds a stream by reaches the endstream found: only a file of - // more than 2 GB can put one there, and the declared length is kept, as when none is found. - search = dataEnd >= 0 && dataEnd - absoluteDataStart <= int.MaxValue - ? new StreamEndSearch( - (int)(dataEnd - absoluteDataStart), - stop, - ((IPdfStreamDataProvider)this).CheckEndStream(dataEnd) ?? EndObjState.Unseen) - : new StreamEndSearch(null, stop, EndObjState.Unknown); + // The first read reaches the place weighed: white space before it is read once, not doubled through. + if (!_placedStarts.TryGetValue(at, out var start)) + { + start = NextSignificant(at, (int)Math.Clamp(position - at + 1, ProofRead, EndStreamSearchChunk)); + _placedStarts[at] = start; } - _endStreamSearches[absoluteDataStart] = search; - return search; + return start == position; + } + + /// + /// Determines whether , whole, starts at : the file ends after it, + /// or no regular character follows. + /// + private bool IsKeywordAt(long position, ReadOnlySpan keyword) + { + Span bytes = stackalloc byte[StartXRefKeyword.Length + 1]; + var length = Math.Min(keyword.Length + 1, _source.Length - position); + var read = _source.Read(position, bytes[..(int)length]); + return bytes[..read].StartsWith(keyword) && (read == keyword.Length ? position + read >= _source.Length : !PdfCharacters.IsRegular(bytes[keyword.Length])); } /// - PdfObject IPdfStreamDataProvider.ResolveLength(PdfObjectId id, out ObjectPresence presence) + PdfObject IPdfStreamDataProvider.ResolveLength(PdfReference reference, out ObjectPresence presence) { + var id = reference.Id; + if (_readingChain) { - // A /Length no section read so far places is read once the chain is read: the data is found by its endstream - // meanwhile, and the length checked then (#182). - var read = ReadWithoutLoading(id); + // A /Length no section read so far places — or one a chain of references from it leads to, or one a guard cut — is + // read once the chain is read: the data is found by its endstream meanwhile, and the length checked then (#182). + var read = ReadThroughWithoutLoading(id); presence = read is null ? ObjectPresence.Deferred : ObjectPresence.Defined; - return read ?? PdfNull.Instance; + + // A chain longer than the parse resolves a /Length through gives null, as it does once the chain is read. + return read is null or PdfReference ? PdfNull.Instance : read; } var reentries = _reentries; @@ -1605,6 +2117,24 @@ PdfObject IPdfStreamDataProvider.ResolveLength(PdfObjectId id, out ObjectPresenc // A null a guard against re-entry gave says nothing of the object: it could not be read from here. presence = value is PdfNull && reentries != _reentries ? ObjectPresence.Unproduced : GetPresence(id.Number); + + // What the object holds, or the one a chain of references from it leads to, is the guard's when a guard cut it or + // left it out of the index: a value the cut left, null or an integer it parted, is none the file gave (#224). + if (presence != ObjectPresence.Unindexed) + { + // The object is not loaded again unless its value leads further: the chain is resolved first, as the parse + // resolves it, so that the object it leads to is read when it is asked whether a guard cut it. + if (value is PdfReference next) + { + next.Resolve(); + } + + var target = value is PdfReference ? reference.Target(PdfReference.MaxChainLength).Number : id.Number; + presence = IsCutAtLimit(target) ? ObjectPresence.Cut + : target != id.Number && GetPresence(target) == ObjectPresence.Unindexed ? ObjectPresence.Unindexed + : presence; + } + return value; } @@ -1683,18 +2213,45 @@ internal static int EndStreamSearchFirstRead(int buffered) => /// for what each read repeats, and each read is searched for both: a header only before the endstream it holds, /// with what carries into it of the reads before. /// - private long SearchEndStream(long dataStart, ReadOnlySpan buffered, long limit, out long header) + /// Where the data starts in the file. + /// The bytes of the data the parser holds, from its start. + /// Where the next object the index places starts, or the end of the file. + /// Whether the stream's /Length is one the chain could not read, which no header stops. + /// Whether the file past the bytes the parser holds may be read. + /// Where the header that stopped the search starts, or -1. + /// Whether the search reached where it stops, rather than the end of what it might read. + /// Whether the bytes searched, before a header that stopped the search, hold an endobj. + /// + /// Where the data ends, when no endstream is found and the search reached where it stops, at the object's own + /// endobj that only white space and comments separate from there (); -1 otherwise. + /// + private long SearchEndStream( + long dataStart, + ReadOnlySpan buffered, + long limit, + bool deferred, + bool readsFile, + out long header, + out bool searched, + out bool endObjSeen, + out long objectEnd) { Span carried = stackalloc byte[HeaderText.CarriedLength]; Span carriedAt = stackalloc long[HeaderText.CarriedLength]; var text = new HeaderText(carried, carriedAt, dataStart); + var tail = new EndObjTail(dataStart); + searched = true; + objectEnd = -1; var inBuffer = (int)Math.Clamp(limit - dataStart, 0, buffered.Length); text.Read(buffered[..inBuffer], dataStart); - var found = SearchStretch(text, endsRange: dataStart + inBuffer >= limit, out header); + var found = StreamEnds.SearchStretch(text, endsRange: dataStart + inBuffer >= limit, deferred, out header); + endObjSeen = StreamEnds.HoldsEndObj(buffered[..inBuffer], header >= 0 ? (int)(header - dataStart) : inBuffer); + tail.Read(buffered[..inBuffer], dataStart, header >= 0 ? header : dataStart + inBuffer); if (found >= 0 || header >= 0) { + objectEnd = found < 0 ? tail.DataEnd(header) : -1; return found; } @@ -1702,6 +2259,13 @@ private long SearchEndStream(long dataStart, ReadOnlySpan buffered, long l if (dataStart + buffered.Length >= end) { + objectEnd = tail.DataEnd(end); + return -1; + } + + if (!readsFile) + { + searched = false; return -1; } @@ -1722,10 +2286,13 @@ private long SearchEndStream(long dataStart, ReadOnlySpan buffered, long l } text.Read(span, position); - found = SearchStretch(text, endsRange: position + span.Length >= end, out header); + found = StreamEnds.SearchStretch(text, endsRange: position + span.Length >= end, deferred, out header); + endObjSeen |= StreamEnds.HoldsEndObj(span, header >= 0 ? (int)(header - position) : span.Length); + tail.Read(span, position, header >= 0 ? header : position + span.Length); if (found >= 0 || header >= 0 || position + span.Length >= end) { + objectEnd = found < 0 ? tail.DataEnd(header >= 0 ? header : end) : -1; return found; } @@ -1736,273 +2303,10 @@ private long SearchEndStream(long dataStart, ReadOnlySpan buffered, long l } /// - /// Searches one read of a search for endstream — the parser's bytes, or a window of the file — for the first - /// endstream, and for the first object header before it: returns where the data ends in the file, the - /// end-of-line before the keyword left out, or -1 when the read holds no endstream or a header comes first, - /// whose start in the file is then , -1 otherwise. - /// - /// - /// The overlap between reads puts the end-of-line before a keyword first seen in a read inside it, unless the keyword - /// starts the data. - /// - /// The bytes read, after what the reads before carry into them. - /// Whether they end where the search does: at the next object the index places, or the end of the file. - /// Where the header that stopped the search starts in the file, or -1. - private static long SearchStretch(in HeaderText text, bool endsRange, out long header) - { - var span = text.Bytes; - var keyword = span.IndexOf("endstream"u8); - header = text.FindHeader(keyword >= 0 ? keyword : span.Length, endsRange); - - if (keyword < 0 || header >= 0) - { - return -1; - } - - if (keyword > 0 && span[keyword - 1] == (byte)'\n') - { - keyword--; - } - - if (keyword > 0 && span[keyword - 1] == (byte)'\r') - { - keyword--; - } - - return text.PositionOf(keyword); - } - - /// - /// One read of a search for endstream as the rule for object headers sees it (): its - /// bytes, after a few that stand for what the search read before them. - /// - /// - /// Whether obj ends an object header depends on the runs of white space and of digits before it, which may run - /// back past the start of a read, as far as the start of the data. What the reads before hold is carried into each one - /// as a few bytes that decide every header as they would: the runs of white space or of digits that end where the read - /// starts, four at most — each run of white space as one space, each run of digits as the digits of its value, or of - /// 2,147,483,648 for any larger one, which no number nor generation reaches however many digits follow —, and the byte - /// before them. A header holds four such runs, and asks of the byte before them only whether it ends a token: no run - /// reads that byte. The start of the data is carried as a space, which ends a token as white space does and holds no - /// digit. The bytes carried have negative indexes, and each records where the run it stands for starts in the file. - /// None of the sizes below bounds what the file holds (ADR 34): a run of any length is carried, as what decides a - /// header — its kind and its value. - /// - private ref struct HeaderText - { - /// The most runs of white space or digits before obj that decide a header: those it holds. - private const int MaxRuns = 4; - - /// The most digits a run of digits is carried as: those of . - private const int MaxDigits = 10; - - /// The most bytes carried into a read: the byte before the runs, two runs of white space and two of digits. - internal const int CarriedLength = 1 + (2 * (1 + MaxDigits)); - - /// The value a run of digits is taken for when it is larger: past any object number and generation. - private const long Beyond = (long)PdfObjectId.MaxNumber + 1; - - private readonly Span _carried; - private readonly Span _carriedAt; - private int _carriedLength; - private ReadOnlySpan _bytes; - private long _at; - - /// Starts a search at , before which nothing is read. - /// Room for bytes carried into a read. - /// Room for where the run each of them stands for starts in the file. - /// Where the data starts in the file. - public HeaderText(Span carried, Span carriedAt, long dataStart) - { - _carried = carried; - _carriedAt = carriedAt; - _carried[0] = (byte)' '; - _carriedAt[0] = dataStart; - _carriedLength = 1; - } - - /// Gets the bytes of the read. - public readonly ReadOnlySpan Bytes => _bytes; - - /// Gets the index of the first byte carried into the read: the byte before the runs, which no run reads. - private readonly int FirstCarried => -_carriedLength; - - private readonly byte this[int index] => index >= 0 ? _bytes[index] : _carried[_carriedLength + index]; - - /// Takes , read at in the file, as the read that follows what is carried. - public void Read(ReadOnlySpan bytes, long at) - { - _bytes = bytes; - _at = at; - } - - /// Gives where the byte at is in the file, or, for a byte carried, where its run starts. - public readonly long PositionOf(int index) => index >= 0 ? _at + index : _carriedAt[_carriedLength + index]; - - /// - /// Carries into the next read, which starts at in this one, the runs of white space and - /// of digits that end there, and the byte before them. - /// - public void CarryTo(long position) - { - Span runAt = stackalloc long[MaxRuns]; - Span runValue = stackalloc long[MaxRuns]; - var end = (int)(position - _at); - var runs = 0; - - while (runs < MaxRuns && end > FirstCarried + 1 && - (PdfCharacters.IsDigit(this[end - 1]) || PdfCharacters.IsWhitespace(this[end - 1]))) - { - var digits = PdfCharacters.IsDigit(this[end - 1]); - var start = RunStart(end, digits); - runAt[runs] = PositionOf(start); - runValue[runs] = digits ? Value(start, end) : -1; - end = start; - runs++; - } - - // What is carried is read whole before it is written over. - var before = this[end - 1]; - _carriedAt[0] = PositionOf(end - 1); - _carried[0] = before; - var length = 1; - - for (var run = runs - 1; run >= 0; run--) - { - var written = 1; - - if (runValue[run] < 0) - { - _carried[length] = (byte)' '; - } - else - { - // The room carried has ten digits for each run of digits, and no value carried has more. - _ = runValue[run].TryFormat(_carried[length..], out written, default, CultureInfo.InvariantCulture); - } - - _carriedAt.Slice(length, written).Fill(runAt[run]); - length += written; - } - - _carriedLength = length; - } - - /// - /// Finds the first object header whose obj ends before in the read, and gives where - /// it starts in the file; -1 when there is none. - /// - /// - /// A header is N G obj at a token boundary, as the file's bytes hold it: white space, a delimiter or the start - /// of the data before the number; a run of digits whose value the parser takes for an object number, 1 to - /// 2,147,483,647, and one whose value it takes for a generation, 0 to 65,535, however many zeros lead them; white - /// space before each of the generation and obj, as much as the file holds; and white space, a delimiter or the - /// end of the search after obj. endobj is no header, nor 10 0 objx, nor digits a regular - /// character precedes, nor tokens a comment separates. The scan that rebuilds an index refuses endobj and - /// tokens a comment separates too, but takes 10 0 objx and digits a regular character precedes (#171). - /// A header lies whole in what the search reads, from the start of the data to the next object the index places: the - /// stream's own header, and whatever precedes the data, is never one, and one that the next object's offset cuts is - /// not seen, that offset stopping the search first. The rule reads nothing but the bytes, so where it stops cannot - /// depend on what was read before; and what a read's start cuts of a header is carried into it, and obj with - /// the byte after it is whole in the read that decides it, so where the reads start cannot change it either: a - /// candidate whose obj ends the read, where the search goes on, is left to the next read, which repeats it. - /// - /// The rule is no guard (ADR 34): only a stream whose endstream does not follow its declared length is - /// searched, which no valid file holds (ISO 32000-1, 7.3.8.1). A valid stream's data may hold text that reads as a - /// header — an embedded PDF written uncompressed —: its length, confirmed, is never searched. An invalid stream whose - /// data holds such text before its endstream stops there, keeps its declared length and is reported, as one - /// whose next object came first; text that only looks like a header is read past, and an endstream after it - /// ends the data. - /// - /// - /// Where the endstream found in the read starts, or its length. - /// Whether the read ends where the search does, where obj may end. - public readonly long FindHeader(int before, bool endsRange) - { - var from = 0; - - while (true) - { - var index = _bytes[from..before].IndexOf(ObjKeyword); - - if (index < 0) - { - return -1; - } - - var keyword = from + index; - var after = keyword + ObjKeyword.Length; - - // What follows obj past the end of the read is the next read's to see, which repeats it: the end of the read - // ends the token only where the search ends. - if ((after < _bytes.Length ? !PdfCharacters.IsRegular(_bytes[after]) : endsRange) && - TryFindStart(keyword, out var start)) - { - return PositionOf(start); - } - - from = after; - } - } - - /// - /// Reads back from the obj at for the rest of an object header, as - /// takes one, and gives where its number starts. - /// - private readonly bool TryFindStart(int keyword, out int start) - { - var generationEnd = RunStart(keyword, digits: false); - var generationStart = RunStart(generationEnd, digits: true); - var numberEnd = RunStart(generationStart, digits: false); - start = RunStart(numberEnd, digits: true); - - // Each run holds a byte at least, what precedes the number ends a token, and the values are an object number's - // and a generation's. - return generationEnd < keyword && generationStart < generationEnd && numberEnd < generationStart && - start < numberEnd && !PdfCharacters.IsRegular(this[start - 1]) && - Value(start, numberEnd) is >= 1 and <= PdfObjectId.MaxNumber && - Value(generationStart, generationEnd) <= PdfObjectId.MaxGeneration; - } - - /// - /// Gives where the run of digits, or of white space, that ends at starts, the first byte - /// carried left out. - /// - private readonly int RunStart(int end, bool digits) - { - var start = end; - - while (start > FirstCarried + 1 && - (digits ? PdfCharacters.IsDigit(this[start - 1]) : PdfCharacters.IsWhitespace(this[start - 1]))) - { - start--; - } - - return start; - } - - /// - /// Gives the value of the digits from to , or when - /// it is larger. - /// - private readonly long Value(int start, int end) - { - var value = 0L; - - for (var index = start; index < end; index++) - { - value = Math.Min((value * 10) + (this[index] - '0'), Beyond); - } - - return value; - } - } - - /// - /// Closes the reader as its document is disposed: whatever asks it for an object or for a stream's data from now on - /// is refused with an that names the document, and what it kept — the objects - /// it cached, the object streams it decoded, its index, what it noted of each object it read, the source — is let go, - /// though a reference the caller holds keeps the reader itself. Closing it again does nothing. + /// Closes the reader as its document is disposed: whatever asks it for an object or for a stream's data from now on + /// is refused with an that names the document, and what it kept — the objects + /// it cached, the object streams it decoded, its index, what it noted of each object it read, the source — is let go, + /// though a reference the caller holds keeps the reader itself. Closing it again does nothing. /// internal void Close() { @@ -2044,6 +2348,16 @@ internal void Close() _streamLengthFaults.TrimExcess(); _endStreamSearches.Clear(); _endStreamSearches.TrimExcess(); + _endStreamChecks.Clear(); + _endStreamChecks.TrimExcess(); + _endObjStarts.Clear(); + _endObjStarts.TrimExcess(); + _endStreamFollowers.Clear(); + _endStreamFollowers.TrimExcess(); + _placedStarts.Clear(); + _placedStarts.TrimExcess(); + _endStreamKeywords.Clear(); + _endStreamKeywords.TrimExcess(); Release(_lengthFaultsReported); Release(_syntaxReported); Release(_limitsReached); @@ -2079,6 +2393,8 @@ private void Initialize() if (!chainRead) { + // A rebuild reads every object again, and loads none the chain read through. + _chainValuesCut = null; Repair(); return; } @@ -2088,6 +2404,7 @@ private void Initialize() _structure.SizeAsWritten = Trailer.GetRaw(PdfName.Size); _structure.RootAsWritten = Trailer.GetRaw(PdfName.Root); CheckDeferredLengths(); + ReachChainValuesCut(); if (!HasUsableRoot()) { @@ -2116,8 +2433,10 @@ private void Initialize() /// The /Length is loaded as any object is: what that load finds or changes comes after the chain's index is /// copied (). A length the endstream the chain stopped at confirms — the white /// space before it counted, or not all of it — is the stream's own, as the parser takes it for any stream, and says - /// nothing. One confirmed by an endstream past it gives data the chain did not read, rows included: they are not - /// read again, and the report says so. The stream is not parsed again, and each is reported once. + /// nothing; so is one the data was taken again to, the section's own row placing it (). + /// One confirmed by an endstream past it gives data the chain did not read, rows included — the section's row + /// for the /Length among them, or one that does not stand —: they are not read again, and the report says so. + /// The stream is not parsed again, and each is reported once. /// private void CheckDeferredLengths() { @@ -2133,21 +2452,26 @@ private void CheckDeferredLengths() var (number, generation) = (deferred.Length.Number, deferred.Length.Generation); string? message = null; - if (value.AsInteger() is not { } length || length is < 0 or > int.MaxValue) - { - var presence = value is PdfNull && reentries != _reentries ? ObjectPresence.Unproduced : GetPresence(number); + // The chain is resolved as the parse resolves a /Length, one object further than a reference: the object it + // leads to is read before it is asked whether a guard cut it, and what it holds is what is judged. + var resolved = value.Resolve(); - // A /Length a guard left out of the index is unknown, not wrong: the guard's report says why. - if (presence == ObjectPresence.Unindexed) - { - continue; - } + // A /Length a guard cut or left out of the index — or the object a chain of references from it leads to — is + // unknown, not wrong: the guard's report says why, and what the cut left of it, null or a parted integer, is + // none the file gave (#224). + if (LeftByGuard(new PdfReference(deferred.Length, this), PdfReference.MaxChainLength)) + { + continue; + } + if (resolved.AsInteger() is not { } length || length is < 0 or > int.MaxValue) + { + var presence = value is PdfNull && reentries != _reentries ? ObjectPresence.Unproduced : GetPresence(number); var held = presence switch { ObjectPresence.Missing => "which the file lacks", ObjectPresence.Unproduced => "which could not be read", - _ => "which holds " + DescribeValue(value) + ", not a length", + _ => "which holds " + DescribeValue(resolved) + ", not a length", }; message = string.Create( CultureInfo.InvariantCulture, @@ -2157,7 +2481,7 @@ private void CheckDeferredLengths() { var stopped = EndStreamAfter(dataStart + deferred.Taken); - if (!((IPdfStreamDataProvider)this).IsEndStreamAt(dataStart + length)) + if (((IPdfStreamDataProvider)this).CheckEndStream(dataStart + length) is null) { message = string.Create(CultureInfo.InvariantCulture, $"The stream declared {length} bytes but ended after {deferred.Taken}."); } @@ -2177,17 +2501,55 @@ private void CheckDeferredLengths() } /// - /// Gives where the endstream keyword starts that the end-of-line or white space at - /// leads to, or itself when none follows within the gap the parser allows. + /// Forgets the objects read through since held of them, which a + /// place the section may have been moved to, and that is dropped, led to: nothing of its reading is kept, and a value + /// the chain keeps that leads to one of them reads it anew, for it to be loaded once the chain is read. + /// + private void ForgetReadThrough(int mark) + { + if (_chainValuesCut is not { } cut) + { + return; + } + + for (var i = mark; i < cut.Count; i++) + { + _readWhileChainIsRead?.Remove(cut[i]); + } + + cut.RemoveRange(mark, cut.Count - mark); + } + + /// + /// Loads, once the chain is read, each object a guard cut that a cross-reference stream's value led to, which the chain + /// ended at or read through: the guard is reached, and reported, where a load reaches it — and thrown, when the caller + /// asked for it —, as for a /Length read once the chain is (#263). + /// + private void ReachChainValuesCut() + { + if (_chainValuesCut is not { } cut) + { + return; + } + + _chainValuesCut = null; + + foreach (var id in cut) + { + GetObject(id); + } + } + + /// + /// Gives where the endstream keyword starts that the white space and comments at + /// lead to, or itself when none follows them. /// private long EndStreamAfter(long position) { - Span tail = stackalloc byte[PdfObjectParser.EndStreamLookahead]; - var read = position < 0 || position >= _source.Length - ? 0 - : _source.Read(position, tail[..(int)Math.Min(tail.Length, _source.Length - position)]); - var keyword = tail[..read].IndexOf("endstream"u8); - return keyword < 0 ? position : position + keyword; + var at = NextSignificant(position); + Span keyword = stackalloc byte[StreamEnds.EndStreamKeyword.Length]; + var read = at < 0 || at >= _source.Length ? 0 : _source.Read(at, keyword[..(int)Math.Min(keyword.Length, _source.Length - at)]); + return keyword[..read].SequenceEqual(StreamEnds.EndStreamKeyword) ? at : position; } /// @@ -2313,7 +2675,7 @@ private bool LeadsPastIndex(PdfObject? value, out PdfObject? read) var number = reference.Id.Number; read = null; return number > 0 && _refusedNumbers?.Contains(number) != true && - (_cutWhileChainIsRead?.Contains(reference.Id) == true || (!_xref.TryGet(number, out _) && MayBeLeftUnread(number))); + (_cutWhileChainIsRead?.ContainsKey(reference.Id) == true || (!_xref.TryGet(number, out _) && MayBeLeftUnread(number))); } } @@ -2767,7 +3129,11 @@ private void ReportMissingSection(string naming, long offset, long namedFrom) /// places in all, over the whole document: each is read through a window /// of up to 64 KB, the file chooses how many headers lie near a section, and a chain can name a section for /// each of its links. Found, the section's record takes what - /// was read there; not found, it keeps what the named offset held. + /// was read there — a section a guard kept from being read too, which is not found to be read, the chain ending there as + /// the guard's —; not found, it keeps what the named offset held. A place is a guess: a table's xref keyword, or + /// an object that declares a cross-reference stream and no other type. A guard that kept it from being read keeps it the + /// section where a value it is read with led to an object the guard cut, or where it declared itself one before the cut + /// — the keyword, or /Type /XRef —; a dictionary cut before it said what it is may be anything, and is dropped. /// private bool TryRelocateXRefSection(long offset, XRefSectionRecord section, out long previous, out long hybrid) { @@ -2805,8 +3171,17 @@ private bool TryRelocateXRefSection(long offset, XRefSectionRecord section, out } var attempt = new XRefSectionRecord(section.NamedBy, candidate, section.NamedFrom); - - if (TryReadXRefSection(candidate, attempt, out previous, out hybrid, candidate: true) == XRefSectionState.Read) + var readThrough = _chainValuesCut?.Count ?? 0; + var state = TryReadXRefSection(candidate, attempt, out previous, out hybrid, candidate: true); + + // A section a guard kept from being read is the section all the same, where a value it is read with led to an + // object a guard cut, or where what is there declared itself one before the cut — the xref keyword, /Type /XRef —: + // found where it was moved, it is where a guard cut the chain, as where it is named (#263). A place a guard cut the + // dictionary of before it said what it is may be anything, and is dropped as any other is, with what it read + // through. + if (state == XRefSectionState.Read || + (state == XRefSectionState.Malformed && attempt.CutByLimit && + (attempt.CutByValue || attempt.Kind == XRefSectionKind.Table || attempt.TypedXRef))) { section.RelocateTo(attempt); _diagnostics.Repair( @@ -2815,8 +3190,17 @@ private bool TryRelocateXRefSection(long offset, XRefSectionRecord section, out CultureInfo.InvariantCulture, $"The cross-reference section {section.NamedBy} names at offset {absolute} was found {candidate - absolute} bytes from there."), candidate); - return true; + + if (state == XRefSectionState.Read) + { + return true; + } + + section.State = XRefSectionState.Malformed; + break; } + + ForgetReadThrough(readThrough); } previous = -1; @@ -2904,10 +3288,24 @@ private XRefSectionState TryReadXRefSection( return section.State; } - using (var probe = _source.GetWindow(absolute, XRefProbeLength)) + // A probe that holds white space and comments alone, or a first token its edge may cut, is read on as far as a + // trailer's window reaches, past which a stream's header is not looked for either: the keyword is the table's + // wherever within that window the offset falls short of it. + var probeLength = XRefProbeLength; + + while (true) { + using var probe = _source.GetWindow(absolute, probeLength); var lexer = new PdfLexer(probe.Memory.Span); var first = lexer.Read(); + + if ((first.Kind == PdfTokenKind.EndOfInput || first.End >= probe.Length) && probe.Length == probeLength && + probeLength < _guard.Bound(PdfLimit.Trailer)) + { + probeLength = (int)Math.Min(_guard.Bound(PdfLimit.Trailer), (long)probeLength * 8); + continue; + } + section.Padding = first.Kind == PdfTokenKind.EndOfInput ? probe.Length : first.Start; if (first.IsKeyword("xref"u8)) @@ -2918,6 +3316,8 @@ private XRefSectionState TryReadXRefSection( : XRefSectionState.Malformed; return section.State; } + + break; } section.State = TryReadXRefStream(absolute, section, out previous, candidate); @@ -2987,11 +3387,26 @@ private bool TryReadClassicTable(long absolute, XRefSectionRecord section, out l { section.TrailerPosition = absolute + token.Start; - if (ReadTrailer(window.Memory, absolute, lexer.Position, windowFull, section) is { } trailer) + var trailer = ReadTrailer(window.Memory, absolute, lexer.Position, windowFull, section); + + if (trailer is not null) { MergeSectionTrailer(trailer, section); - previous = SectionOffset(trailer, PdfName.Prev, absolute); - hybrid = SectionOffset(trailer, PdfName.XRefStm, absolute); + } + + // A table an /XRefStm names, where ISO 32000-1 names a cross-reference stream (Table 19), stands in its + // place: as a hybrid file's stream's own /Prev, what its trailer names is not read, to name a section + // missing or end the chain for a guard, the chain going on through the table that names it. + if (trailer is not null && section.NamedBy != "/XRefStm") + { + previous = SectionOffset(trailer, PdfName.Prev, absolute, out var previousHidden); + hybrid = SectionOffset(trailer, PdfName.XRefStm, absolute, out var streamHidden); + section.ChainHiddenByGuard = streamHidden ?? previousHidden; + section.ChainHiddenAtStream = streamHidden is not null; + + // A stream a guard hid holds rows newer than the older sections': the chain ends with it, as with a + // stream a guard cut, since their rows would stand for the stream's. + previous = streamHidden is null ? previous : -1; } return true; @@ -3218,7 +3633,7 @@ private void ReadTrailerPastRows(long from, XRefSectionRecord section) section.TrailerPosition = found; var guards = _guardsReached; - _ = TryParseNumberedAt(found + TrailerKeyword.Length, DirectObject, PdfLimit.Trailer, out var value, out _, out var syntaxFault); + _ = TryParseNumberedAt(found + TrailerKeyword.Length, DirectObject, PdfLimit.Trailer, out var value, out _, out _, out var syntaxFault); if (_guardsReached != guards) { @@ -3283,7 +3698,7 @@ private bool StartsToken(long position) // The trailer starts inside the window, so inside the file as long as the source's length holds; // where nothing can be parsed the value is null, and anything that is not a dictionary is no trailer. var guards = _guardsReached; - _ = TryParseNumberedAt(absolute + position, DirectObject, PdfLimit.Trailer, out var value, out _, out var syntaxFault); + _ = TryParseNumberedAt(absolute + position, DirectObject, PdfLimit.Trailer, out var value, out _, out _, out var syntaxFault); if (_guardsReached != guards) { @@ -3355,12 +3770,6 @@ private static bool SyntaxFaultSince(PdfDiagnostics diagnostics, PdfDiagnosticsM return false; } - /// - /// Determines whether a guard was reached since , its report kept or dropped: a cut taken for - /// the file's damage once the capacity is reached would let an older section's rows stand for what the cut hid. - /// - private static bool LimitSince(PdfDiagnostics diagnostics, PdfDiagnosticsMark mark) => diagnostics.LimitsReached > mark.Limits; - /// Says what a token is, for a message about what stood where something else should have. private static string DescribeToken(PdfToken token) => token.Kind switch { @@ -3432,11 +3841,16 @@ private bool ReadSubsection(ref PdfLexer lexer, int first, int count, XRefSectio /// /// /// - /// An object that declares itself a cross-reference stream — /Type /XRef, or a /W — and cannot be - /// read is a malformed section; any other object there, or none, is no section at all. At a place the section may have - /// been moved to (), nothing such an object's reading met is reported or recorded — its - /// search for an endstream aside, which is kept as every search is —: it was read as a section, through a - /// section's window, and is read as itself when something asks for it. + /// An object that declares itself a cross-reference stream — /Type /XRef, or a /W, beside no other + /// /Type at a place the section may have been moved to — and cannot be read is a malformed section; any + /// other object there, or none, is no section at all, and a guard that cut the dictionary of what declares another + /// type cut no section, though one that cut a value of a stream with a /W where the chain names the section + /// leaves it the section, as read whole — a place it may have been moved to takes no stream of another type. What + /// makes it unreadable whatever a value a guard cut holds is the file's fault, as read whole + /// (). At a place the section may have been moved to (), + /// nothing such an object's reading met is reported or recorded — its search for an endstream aside, which is + /// kept as every search is —: it was read as a section, through a section's window, and is read as itself when + /// something asks for it. /// /// /// ISO 32000-1 (7.5.8.2) makes the values of this dictionary direct, the elements of /W and /Index @@ -3446,14 +3860,24 @@ private bool ReadSubsection(ref PdfLexer lexer, int first, int count, XRefSectio /// malformed when the rows cannot be read without it (); a /Length no section read /// so far places is read once the chain is read (). /// + /// + /// The data of such a /Length runs to its first endstream, which the file is searched for past the bytes + /// the dictionary was read with — up to the next object the sections read so far place, or the end of the file —, as + /// a stream's whose length is not confirmed, but neither stopped by an object header its rows spell nor trimmed of a + /// carriage return its last row ends in (#220). With none there, the data ends nowhere the section says, and the + /// section is malformed. Rows that spell an endstream are told from the data's end by the section's own row for + /// the /Length, when it lies before them: the data is taken again to the length the object it places gives, + /// where an endstream confirms it (). + /// bounds the dictionary alone. + /// /// private XRefSectionState TryReadXRefStream(long absolute, XRefSectionRecord section, out long previous, bool candidate) { previous = -1; - // The first window, of 64 KB, holds the data of all but the largest cross-reference streams, so the - // parser checks their /Length against the data rather than only at the end the /Length gives. The - // dictionary is the trailer, and grows its window no further than a trailer may. + // The first window, of 64 KB, holds the data of all but the largest cross-reference streams. The dictionary is the + // trailer, and grows its window no further than a trailer may: the data never counts toward it, and a /Length the + // chain cannot read has the data searched for its endstream past the window (#174, #220). var guards = _guardsReached; // A place the section may have been moved to is a guess: what is there and is no section leaves nothing of its @@ -3464,10 +3888,11 @@ private XRefSectionState TryReadXRefStream(long absolute, XRefSectionRecord sect PdfLimit.Trailer, out var value, out var number, + out var header, out var syntaxFault, XRefWindow, - candidate ? DeclaresCrossReferenceStream : null) || - !DeclaresCrossReferenceStream(value)) + candidate ? DeclaresOnlyCrossReferenceStream : null) || + !(candidate ? DeclaresOnlyCrossReferenceStream(value) : DeclaresCrossReferenceStream(value))) { if (!candidate && _guardsReached != guards && value is PdfDictionary { } cut && !cut.ContainsKey(PdfName.Type)) { @@ -3490,197 +3915,860 @@ private XRefSectionState TryReadXRefStream(long absolute, XRefSectionRecord sect var dictionary = value is PdfStream parsed ? parsed.Dictionary : (PdfDictionary)value; section.Kind = XRefSectionKind.Stream; section.StreamObjectNumber = number; + + // The header lies where the object read starts, past however much white space the offset falls short by. + section.Padding = (int)(header - absolute); section.TrailerPosition = absolute; - section.CutByLimit |= _guardsReached != guards; + section.TypedXRef = dictionary.IsOfType(PdfName.XRef); + + // A guard that cut the dictionary of what declares another type cut no section: such a dictionary — a CIDFont's, + // whose /W gives widths — has no stream data read whole, and is no section. A stream of that type, which the cut + // hides, would be read as the section whole: the cut takes it for none (#263). + section.CutByLimit |= _guardsReached != guards && DeclaresOnlyCrossReferenceStream(value); + + if (value is not PdfStream xrefStream) + { + section.Fault = "its dictionary is not followed by stream data"; + return XRefSectionState.Malformed; + } + + section.Trailer = dictionary; + + if (syntaxFault) + { + section.TrailerFault = XRefTrailerFault.Malformed; + } + + // A /Length no section read so far places: the data was taken up to the first endstream it holds, and the /Length is + // read, and checked, once the chain is, if the section is read — or before its rows are, where the section places + // it among them. So is one a chain of references from it leads to, or a guard cut, which the check then passes over. + var deferred = dictionary.GetRaw(PdfName.Length) is PdfReference length && ReadThroughWithoutLoading(length.Id) is null && + !section.CutByLimit + ? new DeferredLength(length.Id, xrefStream.Data.Length, number) + : (DeferredLength?)null; + + // Data a search found no endstream for, before the next object the sections read so far place or the end of the + // file, ends nowhere the section says: its rows are not read (#220). Where the /Length leads to an object a guard cut, + // the length that would have said is the guard's, as a value its rows are read with is (#263): the section counts as + // cut, the chain ends here, and the guard is reached once the chain is read. + if (deferred is not null && _endStreamSearches.TryGetValue((xrefStream.Data.Position, true), out var known) && + known.Search is { Length: null } search) + { + // The fault names the link of the chain that could not be read — the one no section read so far places, or the + // one a guard cut —, which the chain read without loading stopped at, as Follow does. + var lengthFault = Follow(PdfName.Length, dictionary.GetRaw(PdfName.Length), entry: true, out _, out var cutObject, LengthLinks); + + if (cutObject is { } past) + { + // What makes the section unreadable whatever the length the guard hid holds is the file's, as read whole: a + // value no section read so far places, or what the values read give. + var file = UnreadValue(dictionary, out var unreadCut) is { } unplaced && unreadCut is null + ? unplaced + : FaultBesideCut(dictionary, null, out _); + section.Fault = file ?? lengthFault; + return file is null ? CutByValue(section, past) : XRefSectionState.Malformed; + } + + var where = search.NextObject is { } next + ? string.Create(CultureInfo.InvariantCulture, $"before the next object, at {next}") + : "before the end of the file"; + section.Fault = string.Create( + CultureInfo.InvariantCulture, + $"{lengthFault}, and {(search.Searched ? "no endstream follows its data " + where : "its data was not searched for its endstream, the document's searches for endstream having read as much of the file as they may")}"); + return XRefSectionState.Malformed; + } + + if (UnreadValue(dictionary, out var valueCut) is { } unread) + { + section.Fault = unread; + + // A value no section read so far places is the file's fault. One a guard cut is the guard's, as rows a guard cut + // are (CutByValue), but for what makes the section unreadable whatever the value holds: the file's, as read whole. + if (valueCut is not { } cut) + { + return XRefSectionState.Malformed; + } + + if (FaultBesideCut(dictionary, deferred is null && !xrefStream.Data.CutByGuard && !xrefStream.Data.EndHidden ? xrefStream : null, out var holds) is { } file) + { + section.Fault = file; + + if (holds is not { } rows) + { + return XRefSectionState.Malformed; + } + + // Data whose extent the file gives that cannot hold the rows the values read declare lacks rows whatever the + // cut value holds: the section is read as read whole, the file's to lack them — its trailer merged, the index + // rebuilt —, and the guard is reached once the chain is read. The rows the data may hold, at the fewest bytes + // a row may take, are read with the cut value: the guard's, which a whole read reads, the chain ending here, as + // where rows a guard cut, since an older section's rows would stand for them. Data that holds none leaves the + // chain going on. + _indexIncomplete = true; + section.Incomplete = true; + _xref.MergeTrailer(dictionary); + (_chainValuesCut ??= []).Add(cut); + + if (rows > 0) + { + _cutByGuard = true; + section.CutByLimit = true; + return XRefSectionState.Read; + } + + string? hiddenPrevious = null; + previous = section.NamedBy == "/XRefStm" ? -1 : SectionOffset(dictionary, PdfName.Prev, absolute, out hiddenPrevious); + section.ChainHiddenByGuard = hiddenPrevious; + return XRefSectionState.Read; + } + + return CutByValue(section, cut); + } + + var widths = dictionary.GetArray(PdfName.W); + + if (widths is null || widths.Count < 3) + { + section.Fault = WidthsMissing; + return XRefSectionState.Malformed; + } + + // A width counts bytes, and ISO 32000-1 bounds none (Table 17): a field wider than the value the reader keeps holds + // in its leading bytes what no entry can (ReadField). Each is narrowed once the data is decoded, against its length. + Span declaredWidths = stackalloc long[3]; + var anyWidth = false; + + for (var i = 0; i < 3; i++) + { + var width = widths.Resolved(i).AsInteger(); + + if (WidthFault(width) is { } widthFault) + { + section.Fault = widthFault; + return XRefSectionState.Malformed; + } + + declaredWidths[i] = width.GetValueOrDefault(); + anyWidth |= width > 0; + } + + if (!anyWidth) + { + section.Fault = NoRowBytes; + return XRefSectionState.Malformed; + } + + // An /Index that is null, or names an object that holds null, is one the dictionary does not have (7.3.7, 7.3.10): + // one that names an object no section read so far places is UnreadValue's fault. + var written = dictionary.GetRaw(PdfName.Index); + var index = written?.Resolve() is { } given and not PdfNull ? given : null; + + // Beside an /Index, the rows are read without the /Size: it is left as written (UnreadValue). + var size = index is null ? dictionary.GetInteger(PdfName.Size) : null; + + // The numbering is checked before any row is read: a section that numbers its rows wrongly gives none to believe. + // Each value is read once, so that what is read is what was checked. + if (NumberingFault(written, index, size, out var subsections) is { } numbering) + { + section.Fault = numbering; + return XRefSectionState.Malformed; + } + + long declared = size.GetValueOrDefault(); + + if (index is not null) + { + // Each count is within the object numbers, and an /Index holds no more pairs than a trailer: their sum is + // within a long. + declared = 0; + + for (var i = 1; i < subsections.Length; i += 2) + { + declared += subsections[i]; + } + } + + // What a decoding reports of data taken again is dropped with it, and may be met again. + var decoding = deferred is null ? _diagnostics.GetMark() : _diagnostics.GetMarkWithDecodingReports(); + Span fieldWidths = stackalloc int[3]; + ReadOnlySpan data; + bool decodedWhole; + int row; + long held; + var retaken = false; + + // Whether a guard cut the /Length the section's own row places, which leaves how far the data runs unknown. + var lengthCut = false; + + // Whether where the data ends is the guard's to hide, which the data is decoded knowing. + var endHidden = false; + + // Whether the decoding before stopped at MaxDecodedStreamLength, which the data decoded again knowing its end is + // hidden keeps reached: the whole data holds at least what that decoding read. + var reachedBefore = false; + + // How many rows the decoding before held where it stopped at MaxDecodedStreamLength, which the data decoded again + // knowing its end hidden reads no more of. + long heldBefore = 0; + + while (true) + { + var decoded = Filters.PdfFilterPipeline.Decode(xrefStream, _diagnostics, endHidden, reachedBefore, out var leftEncoded, out var cut); + + // Data the guard on filters left encoded holds none of the section's rows: the section is cut before its first. + data = leftEncoded ? default : decoded.Span; + decodedWhole = !cut && !leftEncoded; + + // A row longer than the decoded data is one the data does not hold. Not a guard (ADR 34): the data's length is + // bounded by MaxDecodedStreamLength, a guard, and a valid section's rows are in its data. Each width is narrowed + // once it is known to be within the data, whose length an int holds, and their sum is then within a long. + long rowLength = 0; + + for (var i = 0; i < 3; i++) + { + fieldWidths[i] = (int)Math.Min(declaredWidths[i], data.Length + 1L); + rowLength += fieldWidths[i]; + } + + row = rowLength <= data.Length ? (int)rowLength : 0; + held = row == 0 ? 0 : data.Length / row; + + // The data decoded again knowing its end hidden lacks the group a filter dropped there, and a filter after it, + // given less, may be given more room than the decoding before gave it, and read rows past where that one stopped + // at a guard on decoding: the guard stops it where it stops the data read whole, the decoding before (#220). A + // decoding no guard stopped holds fewer rows only where its end, decoded as the file's, lost a predicted row the + // data decoded again undoes as far as it goes: those it completes are the file's. + if (endHidden && reachedBefore && held > heldBefore) + { + held = heldBefore; + data = data[..(int)(held * row)]; + } + + // Fewer rows than the section declares, in data a /Length the chain could not read left to the first endstream: + // where the section's own row for that /Length, among those held, gives more data, which an endstream confirms, + // the rows spelled an endstream, and the data is taken again to that length (#220). Data that holds no row whole + // is asked too: a newer section may place the object, which a guard cut. + if (!retaken && !endHidden && deferred is { } unsettled && declared > held && decodedWhole) + { + long[] ranges = index is null ? [0, declared] : subsections; + var whole = RetakeDeferred(xrefStream, unsettled, data, fieldWidths, row, held, ranges, out lengthCut, out var numbered); + + // The row for the /Length may lie in a predicted row the spelled endstream parts, which the data decoded as + // data whose end the file gives dropped, left as decoded — the first predicted row parted — or left still + // differenced: the data decoded knowing its end hidden undoes it as far as it goes, and holds the rows before + // the endstream whole. What that decoding reports is dropped with it, but for a guard on decoding it reached, + // which the whole data, holding at least as much, reaches too: where it found no row for the /Length, where + // the rows end is the guard's, as where the decoding before reached it. + if (whole is null && numbered && row > 0) + { + var again = _diagnostics.GetMarkWithDecodingReports(); + var undone = Filters.PdfFilterPipeline.Decode(xrefStream, _diagnostics, endHidden: true, reachedBefore: false, out var leftAgain, out var reached); + + if (!leftAgain) + { + whole = RetakeDeferred(xrefStream, unsettled, undone.Span, fieldWidths, row, undone.Length / row, ranges, out lengthCut, out _); + } + + if (whole is null && reached) + { + _diagnostics.RollBackBesideLimits(decoding); + reachedBefore = true; + heldBefore = undone.Length / row; + endHidden = true; + continue; + } + + _diagnostics.RollBack(again); + } + + if (whole is not null) + { + _diagnostics.RollBack(decoding); + xrefStream = whole; + deferred = unsettled with { Taken = whole.Data.Length }; + retaken = true; + continue; + } + } + + // Where a guard cut the length, the endstream the data ends at — as the chain took it, or took it again — is the + // stream's own, and the rows it lacks the file's, only where what follows proves it, as a load of the stream + // reads it: the rows the data holds place objects too (#224). + if (lengthCut && !endHidden && EndsAtOwnEndStream(xrefStream, data, fieldWidths, row, held)) + { + lengthCut = false; + } + + // Data a /Length the chain could not read left to an endstream nothing confirmed, where the length that would say + // whether it is the data's own is one a guard cut, or where a guard on decoding kept the data, which lacks rows, + // from being taken again to the length: where it ends is the guard's, and a filter run out there says nothing of + // the file. The data is decoded once more knowing it, as data a guard cut, a group its end parts dropped, the + // faults in what it read standing (#220, #224). A guard on decoding the first decoding reached stands too, with + // its report: a filter after the one that drops the group, given less, may stay within the bound. + if (!endHidden && deferred is not null && !xrefStream.Data.CutByGuard && + (lengthCut || (!retaken && !decodedWhole && declared > held))) + { + reachedBefore = cut; + heldBefore = held; + + if (reachedBefore) + { + _diagnostics.RollBackBesideLimits(decoding); + } + else + { + _diagnostics.RollBack(decoding); + } + + endHidden = true; + continue; + } + + break; + } + + section.CutByLimit |= !decodedWhole; + var position = 0; + + // A declared count is the file's claim: the index makes room for no more rows than the data holds. + ReserveIndex(Math.Min(declared, held)); + + if (index is null) + { + // Rows past the data are not read, so a /Size past an int numbers no row past one: the data holds fewer. + if (row > 0) + { + ReadXRefStreamRows(data, ref position, fieldWidths, row, 0, (int)Math.Min(declared, int.MaxValue), section); + } + } + else + { + // A count is bounded by what it numbers, and its rows by the decoded data: the most rows an int counts lie past + // any data a stream decodes to, and a count past them describes rows that are not read. + for (var i = 0; i < subsections.Length && row > 0; i += 2) + { + ReadXRefStreamRows(data, ref position, fieldWidths, row, (int)subsections[i], (int)Math.Min(subsections[i + 1], int.MaxValue), section); + } + } + + // Compared by division: a /Size can be as large as a long, and its product with the row length overflow one. + var rowsCut = false; + + if (declared > held) + { + if (decodedWhole && !lengthCut) + { + // Rows the data does not hold, when the reader decoded all of it, are the file's to answer for, and the + // objects they stood for the index's to find. Every fault met before returned, and a refused row is kept + // apart: this is the section's first. + _indexIncomplete = true; + section.Incomplete = true; + section.Fault = string.Create( + CultureInfo.InvariantCulture, + $"it holds {held:N0} {(held == 1 ? "row" : "rows")} where its /Index and /Size declare {declared:N0}"); + } + else + { + // Rows past what MaxDecodedStreamLength let the data decode to are the guard's — or past the endstream the + // rows spelled, when a guard cut the /Length the section's own row places, which would say how far the data + // runs —: what they index reads as null until it is lifted, and the chain ends here, since an older + // section's rows would stand for them. + _cutByGuard = true; + section.CutByLimit = true; + rowsCut = true; + } + } + + // Data a guard cut that lacks rows — MaxStreamLength as the data was read, or a guard on decoding it — is the guard's, + // which says so (ADR 34), and is not checked against a length: the cut kept the data from being taken again to it. + // Data that holds every row the section declares, which nothing would take again, is checked whatever a guard did — + // the check reads the length and where the search found the endstream, which no cut moves —, but not against a + // length a guard cut, which is no length the file gave. + if (deferred is { } check && (decodedWhole || (!lengthCut && declared <= held))) + { + (_deferredLengths ??= [])[xrefStream.Data.Position] = check; + } + + // The rows a guard left unread are the only ones that may yet place an object the dictionary names, before any older + // section is read: from the row the index had no room for, which the reading passed — every row, for an index full + // before the first —, up to the last the data holds; or every row past the data a guard cut. Rows that data decoded + // whole does not hold are none: the file's to lack. + var stopped = _indexFullAt >= 0 && row > 0 ? Math.Max(0, (position / row) - 1) : long.MaxValue; + stopped = rowsCut ? Math.Min(stopped, held) : stopped; + _rowsLeftUnread = RowsFrom(index is null ? [0, declared] : subsections, stopped, rowsCut ? long.MaxValue : held); + + _xref.MergeTrailer(dictionary); + + // A hybrid file's stream gives a /Prev of its own, which the reader does not follow: the chain goes on through the + // table's, as ISO 32000-1 (7.5.8.4) describes, and the stream's is not read, to name a section missing or end the + // chain for a guard. + string? previousHidden = null; + previous = rowsCut || section.NamedBy == "/XRefStm" ? -1 : SectionOffset(dictionary, PdfName.Prev, absolute, out previousHidden); + section.ChainHiddenByGuard = previousHidden; + return XRefSectionState.Read; + } + + /// + /// Determines whether the endstream cross-reference stream 's data ends at, as the chain + /// took it or took it again, is the stream's own beyond what a /Length a guard cut could say + /// (): where the data ends is then the file's (#224). The next + /// object an index places is the nearer of the one the sections read so far place and the first the + /// rows of place past the start of the data, as the index a load of the + /// stream reads with holds them. + /// + private bool EndsAtOwnEndStream(PdfStream stream, ReadOnlySpan data, ReadOnlySpan widths, int row, long held) + { + var dataStart = stream.Data.Position; + var dataEnd = dataStart + stream.Data.Length; + var provider = (IPdfStreamDataProvider)this; + + if (!_endStreamSearches.TryGetValue((dataStart, true), out var known) || + provider.CheckEndStream(dataEnd) is not EndObjState endObj || endObj is not (EndObjState.Absent or EndObjState.Present)) + { + return false; + } + + var next = known.Search.NextObject; + + // Each row lies within the data, whose length an int holds. + for (var at = 0L; at < held; at++) + { + var fields = data.Slice((int)at * row, row); + var cursor = 0; + var type = widths[0] == 0 ? 1 : ReadField(fields, ref cursor, widths[0]); + var offset = ReadField(fields, ref cursor, widths[1]); + + if (type == 1 && offset <= long.MaxValue && PositionOf((long)offset) is var placed && placed > dataStart && + (next is null || placed < next)) + { + next = placed; + } + } + + return provider.IsOwnEndStream(dataEnd, endObj, next); + } + + /// + /// Gives cross-reference stream with its data taken again to the length its /Length + /// gives, which the chain could not read and took the data up to the first endstream for, when the rows + /// holds show that endstream was in the rows (#220): the section's own row for the object + /// the /Length names, among those held, places it, and the length it gives runs past the data taken, to an + /// endstream that confirms it. Null otherwise: the length is checked once the chain is read — and a subsection + /// numbers the object, its row past those held or not standing, where says so. + /// + /// + /// The section's row for the object is the first that numbers it with a type the reading of its rows takes: one of a + /// reserved type is ignored there, as the specification says (). The object is read + /// where that row places it as reads one: nothing is corrected, cached, recorded or + /// reported. One a guard cuts gives what the cut left of it, an integer read before the cut only a place to look for + /// endstream at (#224), and says so, whether an endstream confirms it or not: + /// rows the data taken again still lacks are the guard's. A reference it holds, which a cut after it leaves whole, leads + /// on, as does the /Length when a newer section gave the number: to a length an object read whole holds, the + /// file's, or to the integer the cut left of an object a guard cut, read where the sections read so far place it, only a + /// place to look at then (). A row that does not stand otherwise — a newer section gave + /// the number, or refused it; the row lies past those held, places the object in an object stream, or under another + /// generation — leaves the data as it was taken. + /// + private PdfStream? RetakeDeferred( + PdfStream stream, + DeferredLength deferred, + ReadOnlySpan data, + ReadOnlySpan widths, + int row, + long held, + long[] subsections, + out bool lengthCut, + out bool numbered) + { + lengthCut = false; + numbered = false; + var number = deferred.Length.Number; + + if (_xref.TryGet(number, out _) || _refusedNumbers?.Contains(number) == true) + { + // The object a newer section places, which the chain could not read: a guard that cut it, or the object a chain + // of references from it leads to, leaves how far the data runs to the integer the cut left, where an endstream + // confirms it, and unknown otherwise. + return TakeAgainThrough(stream, deferred, stream.Dictionary.GetRaw(PdfName.Length), LengthLinks, out lengthCut); + } + + // The rows that number the object, in the order the subsections give them, up to the first of a type the reading + // takes: their counts sum within a long, as the section's do. + long ordinal = 0; + ulong type = 0; + ulong offset = 0; + ulong generation = 0; + var taken = false; + + for (var i = 0; i < subsections.Length && !taken; i += 2) + { + if (number >= subsections[i] && number - subsections[i] < subsections[i + 1]) + { + var at = ordinal + (number - subsections[i]); + numbered = true; + + if (at >= held) + { + return null; + } + + // The row lies within the data, whose length an int holds. + var fields = data.Slice((int)at * row, row); + var cursor = 0; + type = widths[0] == 0 ? 1 : ReadField(fields, ref cursor, widths[0]); + offset = ReadField(fields, ref cursor, widths[1]); + generation = ReadField(fields, ref cursor, widths[2]); + taken = type <= 2; + } + + ordinal += subsections[i + 1]; + } + + if (!taken || type != 1 || offset > long.MaxValue || generation != (ulong)deferred.Length.Generation) + { + return null; + } + + if (!TryReadUnrecorded(deferred.Length, PositionOf((long)offset), out var value, out var pastGuard) && !pastGuard) + { + return null; + } + + // The object read holds the chain's second link, which a cut after it leaves whole: the chain goes on as far as the + // parse resolves a /Length through. + if (value is PdfReference) + { + return TakeAgainThrough(stream, deferred, value, LengthLinks - 1, out lengthCut); + } + + // Data taken again to an integer a guard's cut left is no length the file gave: rows it still lacks are the guard's. + lengthCut = pastGuard; + return TakeAgain(stream, deferred, value); + } + + /// + /// Gives cross-reference stream with its data taken again to the length + /// , an integer, gives, when it gives more than the data taken and an endstream confirms + /// it; null otherwise. + /// + private PdfStream? TakeAgain(PdfStream stream, DeferredLength deferred, PdfObject value) + { + var dataStart = stream.Data.Position; + + return value.AsInteger() is { } length && length > deferred.Taken && length <= int.MaxValue && + length <= _source.Length - dataStart && ((IPdfStreamDataProvider)this).CheckEndStream(dataStart + length) is not null + ? new PdfStream(stream.Dictionary, ((IPdfStreamDataProvider)this).Create(dataStart, (int)length)) + : null; + } + + /// + /// Gives cross-reference stream with its data taken again to the length + /// leads to, read through the sections read so far and references, + /// as takes it; null when the chain reaches no length. + /// + /// + /// An object a guard cut ends the chain at the integer the cut left of it, read once, where a section read so far places + /// it (#224): only a place to look for endstream at, which says, the data taken again + /// or not — rows it still lacks are the guard's. One whose value the cut left a whole reference is read through, as a load + /// reads through it (). A chain that reaches no value — an object no section read so far + /// places, or a reference past the links — leaves the data as it was taken, as a whole read leaves it. + /// + private PdfStream? TakeAgainThrough(PdfStream stream, DeferredLength deferred, PdfObject? written, int links, out bool lengthCut) + { + lengthCut = false; + var value = written; + + for (var link = 0; value is PdfReference reference; link++) + { + if (link == links) + { + return null; + } + + if (ReadWithoutLoading(reference.Id) is { } read) + { + value = read; + } + else if (_cutWhileChainIsRead?.TryGetValue(reference.Id, out var left) == true) + { + value = left; + lengthCut = true; + } + else + { + return null; + } + } + + return value is null ? null : TakeAgain(stream, deferred, value); + } + + /// + /// Determines whether what an offset of the chain holds declares itself a cross-reference stream: /Type /XRef, + /// or a /W. + /// + private static bool DeclaresCrossReferenceStream(PdfObject value) => + (value is PdfStream stream ? stream.Dictionary : value as PdfDictionary) is { } dictionary && + (dictionary.IsOfType(PdfName.XRef) || dictionary.ContainsKey(PdfName.W)); + + /// + /// Takes a section a value's cut keeps from being read for the guard's: the section counts as cut, the chain ends there + /// and nothing is rebuilt for it (ADR 34), and the object the guard cut is loaded once the chain is read, which reaches + /// the guard (#263). Where the chain names the section, whatever /Type the stream gives, as a stream with a + /// /W is read there as the section whole — a place it may have been moved to takes no stream of another type —: + /// the value hides whether its rows are, but for what finds wrong whatever it holds. + /// + /// The section, whose fault names the value. + /// The object the guard cut that the value leads to. + private XRefSectionState CutByValue(XRefSectionRecord section, PdfObjectId cut) + { + section.CutByLimit = true; + section.CutByValue = true; + (_chainValuesCut ??= []).Add(cut); + return XRefSectionState.Malformed; + } + + /// + /// Says what makes a cross-reference stream unreadable whatever the values a guard cut hold, as a whole read says it: + /// the widths its /W gives, then how it numbers its rows, as far as the values read say, then, for + /// , whether its data can hold the rows they declare. A value a guard cut, or an element of + /// one, says nothing, nor does what depends on it — the /Size an /Index so cut may stand beside —; an + /// /Index that holds no pairs is wrong whatever its elements hold. Null when nothing read is wrong: the section is + /// then the guard's. + /// + /// + /// followed every value read here, and found none that no section read so far places: a value + /// that cannot be read is one a guard cut. + /// + /// The stream's dictionary. + /// The stream, whose data's extent is the file's, or null when it is not. + /// + /// The most rows the data holds, where the fault is that it holds fewer than the section declares; null otherwise. + /// + private string? FaultBesideCut(PdfDictionary dictionary, PdfStream? stream, out long? held) + { + held = null; + + // The fewest bytes a row may hold, which the widths read give, and the fewest rows the counts read declare. + long row = 0; + long declared = -1; + var exact = true; - if (value is not PdfStream xrefStream) + if (Unread(dictionary, PdfName.W, out var w, out _) is null) { - section.Fault = "its dictionary is not followed by stream data"; - return XRefSectionState.Malformed; - } + if (w is not PdfArray widths || widths.Count < 3) + { + return WidthsMissing; + } - section.Trailer = dictionary; + var known = true; + var anyWidth = false; - if (syntaxFault) - { - section.TrailerFault = XRefTrailerFault.Malformed; - } + for (var i = 0; i < 3; i++) + { + if (Follow(PdfName.W, widths[i], entry: false, out var element, out _) is not null) + { + known = false; + continue; + } - // A /Length no section read so far places: the data was taken up to the first endstream it holds, and the /Length is - // read, and checked, once the chain is, if the section is read. Data a guard cut is the guard's, which says so - // (ADR 34), and is not checked against a length. - var deferred = dictionary.GetRaw(PdfName.Length) is PdfReference length && ReadWithoutLoading(length.Id) is null && - !section.CutByLimit && !xrefStream.Data.CutByGuard - ? new DeferredLength(length.Id, xrefStream.Data.Length, number) - : (DeferredLength?)null; + var width = element?.AsInteger(); - if (UnreadValue(dictionary) is { } unread) - { - section.Fault = unread; - return XRefSectionState.Malformed; - } + if (WidthFault(width) is { } fault) + { + return fault; + } - var widths = dictionary.GetArray(PdfName.W); + anyWidth |= width > 0; - if (widths is null || widths.Count < 3) - { - section.Fault = "its /W does not give the widths of three fields"; - return XRefSectionState.Malformed; - } + // A width past what an int counts makes a row longer than any data decodes to: three so narrowed are within + // a long. + row += Math.Min(width.GetValueOrDefault(), int.MaxValue + 1L); + } - // A width counts bytes, and ISO 32000-1 bounds none (Table 17): a field wider than the value the reader keeps holds - // in its leading bytes what no entry can (ReadField). Each is narrowed once the data is decoded, against its length. - Span declaredWidths = stackalloc long[3]; - var anyWidth = false; + if (known && !anyWidth) + { + return NoRowBytes; + } + } - for (var i = 0; i < 3; i++) + if (Unread(dictionary, PdfName.Index, out var index, out _) is not null) { - var width = widths.Resolved(i).AsInteger(); + return null; + } - if (width is null) + if (index is null or PdfNull) + { + if (Unread(dictionary, PdfName.Size, out var size, out _) is not null) { - section.Fault = "its /W gives a field a width that is not an integer"; - return XRefSectionState.Malformed; + return null; } - if (width < 0) + if (NumberingFault(null, null, size?.AsInteger(), out _) is { } sizeFault) { - section.Fault = "its /W gives a field a negative width"; - return XRefSectionState.Malformed; + return sizeFault; } - declaredWidths[i] = width.Value; - anyWidth |= width > 0; + declared = size?.AsInteger() ?? 0; } - - if (!anyWidth) + else { - section.Fault = "its /W gives rows of no bytes"; - return XRefSectionState.Malformed; - } - - // An /Index that is null, or names an object that holds null, is one the dictionary does not have (7.3.7, 7.3.10): - // one that names an object no section read so far places is UnreadValue's fault. - var written = dictionary.GetRaw(PdfName.Index); - var index = written?.Resolve() is { } given and not PdfNull ? given : null; - var size = dictionary.GetInteger(PdfName.Size); + if (index is not PdfArray ranges || PairsFault(ranges) is not null) + { + return NumberingFault(dictionary.GetRaw(PdfName.Index), index, null, out _); + } - // The numbering is checked before any row is read: a section that numbers its rows wrongly gives none to believe. - // Each value is read once, so that what is read is what was checked. - if (NumberingFault(written, index, size, out var subsections) is { } numbering) - { - section.Fault = numbering; - return XRefSectionState.Malformed; - } + declared = 0; - var decoding = _diagnostics.GetMark(); - var decoded = Filters.PdfFilterPipeline.Decode(xrefStream, _diagnostics, out var leftEncoded); + for (var i = 0; i < ranges.Count; i += 2) + { + var startRead = Follow(PdfName.Index, ranges[i], entry: false, out var start, out _) is null; + var countRead = Follow(PdfName.Index, ranges[i + 1], entry: false, out var count, out _) is null; - // Data the guard on filters left encoded holds none of the section's rows: the section is cut before its first. - var data = leftEncoded ? default : decoded.Span; - var decodedWhole = !LimitSince(_diagnostics, decoding) && !xrefStream.Data.CutByGuard; - section.CutByLimit |= !decodedWhole; + if ((startRead && countRead + ? SubsectionFault(ranges, i, start?.AsInteger(), count?.AsInteger()) + : HalfSubsectionFault(ranges, i, startRead, start?.AsInteger(), countRead, count?.AsInteger())) is { } fault) + { + return fault; + } - // A row longer than the decoded data is one the data does not hold. Not a guard (ADR 34): the data's length is - // bounded by MaxDecodedStreamLength, a guard, and a valid section's rows are in its data. Each width is narrowed - // once it is known to be within the data, whose length an int holds, and their sum is then within a long. - Span fieldWidths = stackalloc int[3]; - long rowLength = 0; + // Each count read is within the object numbers, and an /Index holds no more pairs than a trailer: their sum + // is within a long. A count a guard cut is one more, of 0 rows or more. + if (countRead) + { + declared += count!.AsInteger().GetValueOrDefault(); + } + else + { + exact = false; + } + } + } - for (var i = 0; i < 3; i++) + if (stream is null || declared <= 0 || RowsHeldAtMost(stream, Math.Max(row, 1)) is not { } most || declared <= most.Rows) { - fieldWidths[i] = (int)Math.Min(declaredWidths[i], data.Length + 1L); - rowLength += fieldWidths[i]; + return null; } - var row = rowLength <= data.Length ? (int)rowLength : 0; - var held = row == 0 ? 0 : data.Length / row; - var position = 0; - long declared = size.GetValueOrDefault(); + held = most.Rows; + return string.Create( + CultureInfo.InvariantCulture, + $"its data, {most.Length:N0} {(most.Length == 1 ? "byte" : "bytes")} decoded, holds at most {most.Rows:N0} of the {declared:N0}{(exact ? string.Empty : " or more")} {(declared == 1 && exact ? "row" : "rows")} its /Index and /Size declare"); + } - if (index is not null) + /// + /// Gives how many rows of bytes or more the data of cross-reference stream + /// holds at most, and how long it decodes to, when its filters are written direct and decode + /// it whole; null otherwise: a value of them a reference holds is read as the chain reads one, and its decoding is what + /// the section's reading does. + /// + private static (long Rows, int Length)? RowsHeldAtMost(PdfStream stream, long row) + { + if (!IsWrittenDirect(stream.Dictionary.GetRaw(PdfName.Filter)) || !IsWrittenDirect(stream.Dictionary.GetRaw(PdfName.DecodeParms))) { - // Each count is within the object numbers, and an /Index holds no more pairs than a trailer: their sum is - // within a long. - declared = 0; - - for (var i = 1; i < subsections.Length; i += 2) - { - declared += subsections[i]; - } + return null; } - // A declared count is the file's claim: the index makes room for no more rows than the data holds. - ReserveIndex(Math.Min(declared, held)); + var decoded = Filters.PdfFilterPipeline.Decode(stream, diagnostics: null, endHidden: false, reachedBefore: false, out var leftEncoded, out var cut); + return leftEncoded || cut ? null : (decoded.Length / row, decoded.Length); + } - if (index is null) + /// + /// Determines whether a stream's /Filter or /DecodeParms is written with no reference: itself, the + /// elements of an array, and the values of a dictionary, its own or an element's. + /// + private static bool IsWrittenDirect(PdfObject? value) + { + if (value is PdfArray elements) { - // Rows past the data are not read, so a /Size past an int numbers no row past one: the data holds fewer. - if (row > 0) + for (var i = 0; i < elements.Count; i++) { - ReadXRefStreamRows(data, ref position, fieldWidths, row, 0, (int)Math.Min(declared, int.MaxValue), section); + if (elements[i] is PdfReference || (elements[i] is PdfDictionary parameters && !HoldsNoReference(parameters))) + { + return false; + } } + + return true; } - else + + return value is not PdfReference && (value is not PdfDictionary dictionary || HoldsNoReference(dictionary)); + } + + /// Determines whether none of the values of is a reference. + private static bool HoldsNoReference(PdfDictionary dictionary) + { + foreach (var entry in dictionary) { - // A count is bounded by what it numbers, and its rows by the decoded data: the most rows an int counts lie past - // any data a stream decodes to, and a count past them describes rows that are not read. - for (var i = 0; i < subsections.Length && row > 0; i += 2) + if (entry.Value is PdfReference) { - ReadXRefStreamRows(data, ref position, fieldWidths, row, (int)subsections[i], (int)Math.Min(subsections[i + 1], int.MaxValue), section); + return false; } } - // Compared by division: a /Size can be as large as a long, and its product with the row length overflow one. - var rowsCut = false; + return true; + } - if (declared > held) + /// + /// Says what is wrong with the subsection an /Index gives at , one of whose two elements a + /// guard cut, whatever that element holds: the other one is no integer, or a count below 0 or past what object numbers + /// hold, or a first number outside them. Null when the element read is right for some value of the other. + /// + /// The /Index, whose written elements a fault describes. + /// Where the subsection's first number lies. + /// Whether the first number was read. + /// The first number, when it was read and is an integer. + /// Whether the count was read. + /// The count, when it was read and is an integer. + private static string? HalfSubsectionFault(PdfArray ranges, int at, bool startRead, long? start, bool countRead, long? count) + { + if ((startRead && start is null) || (countRead && count is null)) { - if (decodedWhole) - { - // Rows the data does not hold, when the reader decoded all of it, are the file's to answer for, and the - // objects they stood for the index's to find. Every fault met before returned, and a refused row is kept - // apart: this is the section's first. - _indexIncomplete = true; - section.Incomplete = true; - section.Fault = string.Create( - CultureInfo.InvariantCulture, - $"it holds {held:N0} {(held == 1 ? "row" : "rows")} where its /Index and /Size declare {declared:N0}"); - } - else - { - // Rows past what MaxDecodedStreamLength let the data decode to are the guard's: what they index reads as - // null until it is lifted, and the chain ends here, since an older section's rows would stand for them. - _cutByGuard = true; - rowsCut = true; - } + return $"its /Index holds {DescribeValue(ranges[startRead && start is null ? at : at + 1])} where an integer belongs"; } - if (deferred is { } check) + if (countRead && count < 0) { - (_deferredLengths ??= [])[xrefStream.Data.Position] = check; + return "its /Index gives a subsection a count of rows out of range"; } - // The rows a guard left unread are the only ones that may yet place an object the dictionary names, before any older - // section is read: from the row the index had no room for, which the reading passed — every row, for an index full - // before the first —, up to the last the data holds; or every row past the data a guard cut. Rows that data decoded - // whole does not hold are none: the file's to lack. - var stopped = _indexFullAt >= 0 && row > 0 ? Math.Max(0, (position / row) - 1) : long.MaxValue; - stopped = rowsCut ? Math.Min(stopped, held) : stopped; - _rowsLeftUnread = RowsFrom(index is null ? [0, declared] : subsections, stopped, rowsCut ? long.MaxValue : held); + if (countRead && count > PdfObjectId.MaxNumber + 1L) + { + return string.Create( + CultureInfo.InvariantCulture, + $"its /Index gives a subsection of {count:N0} rows, more than object numbers 0 to {PdfObjectId.MaxNumber} hold"); + } - _xref.MergeTrailer(dictionary); - previous = rowsCut ? -1 : SectionOffset(dictionary, PdfName.Prev, absolute); - return XRefSectionState.Read; + return startRead && start is < 0 or > PdfObjectId.MaxNumber + ? string.Create( + CultureInfo.InvariantCulture, + $"its /Index gives a subsection from object {start}, outside object numbers 0 to {PdfObjectId.MaxNumber}") + : null; } + /// The fault of a /W that is no array of three widths. + private const string WidthsMissing = "its /W does not give the widths of three fields"; + + /// The fault of a /W whose widths are all 0. + private const string NoRowBytes = "its /W gives rows of no bytes"; + + /// Says what is wrong with a width a /W gives, or null when nothing is. + /// The width, when it is an integer. + private static string? WidthFault(long? width) => width switch + { + null => "its /W gives a field a width that is not an integer", + < 0 => "its /W gives a field a negative width", + _ => null, + }; + /// - /// Determines whether what an offset of the chain holds declares itself a cross-reference stream: /Type /XRef, - /// or a /W. + /// Determines whether what a place a section may have been moved to holds declares itself a cross-reference stream, and + /// no other type: /Type /XRef, or a /W beside no /Type that names another. A guess is no section + /// where it says it is something else — a CIDFont's /W gives its glyphs' widths. /// - private static bool DeclaresCrossReferenceStream(PdfObject value) => + private static bool DeclaresOnlyCrossReferenceStream(PdfObject value) => (value is PdfStream stream ? stream.Dictionary : value as PdfDictionary) is { } dictionary && - (dictionary.IsOfType(PdfName.XRef) || dictionary.ContainsKey(PdfName.W)); + (dictionary.IsOfType(PdfName.XRef) || (dictionary.ContainsKey(PdfName.W) && dictionary.GetRaw(PdfName.Type) is not PdfName)); /// /// Says which value a cross-reference stream's dictionary refers to, among those its rows are read with, that no section @@ -3696,43 +4784,93 @@ private static bool DeclaresCrossReferenceStream(PdfObject value) => /// rows are read without — /Type, /Size beside an /Index, an entry no filter reads — leaves the /// section readable; the validator reports how it is written. /// - private string? UnreadValue(PdfDictionary dictionary) + /// The cross-reference stream's dictionary. + /// The object a guard cut that the value which cannot be read leads to, or null when no guard did. + /// + /// The first value that no section read so far places, which the sections a whole read reads before make the file's + /// fault; otherwise the first a guard cut, the values past it read on — but for what depends on it: the /Size + /// an /Index so cut may stand beside, and the filters' values past one (#263). + /// + private string? UnreadValue(PdfDictionary dictionary, out PdfObjectId? cut) { - if (Unread(dictionary, PdfName.W, out var widths) is { } fault) + var first = default(FirstUnread); + + if (first.Ends(Unread(dictionary, PdfName.W, out var widths, out cut), cut)) { - return fault; + return first.Fault; } for (var i = 0; widths is PdfArray w && i < Math.Min(3, w.Count); i++) { - if (Follow(PdfName.W, w[i], entry: false, out _) is { } elementFault) + if (first.Ends(Follow(PdfName.W, w[i], entry: false, out _, out cut), cut)) { - return elementFault; + return first.Fault; } } - if (Unread(dictionary, PdfName.Index, out var index) is { } indexFault) + var indexFault = Unread(dictionary, PdfName.Index, out var index, out cut); + + if (first.Ends(indexFault, cut)) { - return indexFault; + return first.Fault; } if (index is PdfArray ranges) { foreach (var element in ranges) { - if (Follow(PdfName.Index, element, entry: false, out _) is { } elementFault) + if (first.Ends(Follow(PdfName.Index, element, entry: false, out _, out cut), cut)) { - return elementFault; + return first.Fault; } } } - else if ((index is null or PdfNull) && Unread(dictionary, PdfName.Size, out _) is { } sizeFault) + else if (indexFault is null && (index is null or PdfNull) && first.Ends(Unread(dictionary, PdfName.Size, out _, out cut), cut)) { - return sizeFault; + return first.Fault; } var following = new Following(this); - return Filters.FilterChain.TryFollow(dictionary, _guard.Bound(PdfLimit.FilterChain), ref following) ? null : following.Fault; + + if (!Filters.FilterChain.TryFollow(dictionary, _guard.Bound(PdfLimit.FilterChain), ref following)) + { + first.Ends(following.Fault, following.Cut); + } + + cut = first.Cut; + return first.Fault; + } + + /// + /// Keeps the first value of a cross-reference stream's dictionary a guard cut, until one that no section read so far + /// places, which replaces it. + /// + private struct FirstUnread + { + /// Gets what could not be read. + public string? Fault { get; private set; } + + /// Gets the object a guard cut that the value which could not be read leads to, if a guard cut one. + public PdfObjectId? Cut { get; private set; } + + /// Takes what a value could not be read for, and says whether it ends the reading: no guard cut it. + /// What the value could not be read for, or null when it could be. + /// The object a guard cut that left it unread, or null when none did. + public bool Ends(string? fault, PdfObjectId? cut) + { + if (fault is null) + { + return false; + } + + if (cut is null || Fault is null) + { + Fault = fault; + Cut = cut; + } + + return cut is null; + } } /// @@ -3745,10 +4883,14 @@ private struct Following(PdfFileReader reader) : Filters.IFilterValueReader /// Gets what could not be read, once a value could not be. public string? Fault { get; private set; } + /// Gets the object a guard cut that the value which could not be read leads to, if a guard cut one. + public PdfObjectId? Cut { get; private set; } + /// public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? value) { - Fault = reader.Follow(key, written, entry, out value); + Fault = reader.Follow(key, written, entry, out value, out var cut); + Cut = cut; return Fault is null; } } @@ -3757,8 +4899,8 @@ public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? /// Follows the value gives in , and says so when a reference it is, /// or leads to, cannot be read. /// - private string? Unread(PdfDictionary dictionary, PdfName key, out PdfObject? value) => - Follow(key, dictionary.GetRaw(key), entry: true, out value); + private string? Unread(PdfDictionary dictionary, PdfName key, out PdfObject? value, out PdfObjectId? cut) => + Follow(key, dictionary.GetRaw(key), entry: true, out value, out cut); /// /// Follows through as many references as does, each read @@ -3769,13 +4911,20 @@ public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? /// The value as the file wrote it. /// Whether is the entry's own value, rather than an element of it. /// The value it leads to, or null when it leads to none. - private string? Follow(PdfName key, PdfObject? written, bool entry, out PdfObject? value) + /// The object a guard cut that left the value unread, or null when none did. + /// + /// How many references are followed: one more for a value read from the object a reference names and then resolved — + /// a stream's /Length, a trailer's /Prev or /XRefStm —, as says. + /// + private string? Follow( + PdfName key, PdfObject? written, bool entry, out PdfObject? value, out PdfObjectId? cut, int links = MaxReferenceLinks) { value = written; + cut = null; for (var link = 0; value is PdfReference reference; link++) { - if (link == MaxReferenceLinks) + if (link == links) { value = null; return $"its {FileQuote.Name(key)} leads from reference to reference without reaching a value"; @@ -3783,10 +4932,19 @@ public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? if (ReadWithoutLoading(reference.Id) is not { } read) { + // An object a section read so far places, which runs past MaxObjectLength: the guard's, not the file's. + var place = "which no section read before it places where it can be read"; + + if (_cutWhileChainIsRead?.ContainsKey(reference.Id) == true) + { + cut = reference.Id; + place = "which one of the reader's limits cut"; + } + value = null; return string.Create( CultureInfo.InvariantCulture, - $"its {FileQuote.Name(key)} {(entry && link == 0 ? "is" : "holds")} the reference {reference.Id.Number} {reference.Id.Generation} R, which no section read before it places where it can be read"); + $"its {FileQuote.Name(key)} {(entry && link == 0 ? "is" : "holds")} the reference {reference.Id.Number} {reference.Id.Generation} R, {place}"); } value = read; @@ -3825,11 +4983,9 @@ public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? return $"its /Index is {DescribeValue(written)}, not an array"; } - if (ranges.Count == 0 || ranges.Count % 2 != 0) + if (PairsFault(ranges) is { } pairs) { - return string.Create( - CultureInfo.InvariantCulture, - $"its /Index holds {ranges.Count} {(ranges.Count == 1 ? "value" : "values")}, not pairs of a first object number and a count of rows"); + return pairs; } var checkedSubsections = new long[ranges.Count]; @@ -3839,33 +4995,55 @@ public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? var start = ranges.Resolved(i).AsInteger(); var count = ranges.Resolved(i + 1).AsInteger(); - if (start is null || count is null) - { - return $"its /Index holds {DescribeValue(ranges[start is null ? i : i + 1])} where an integer belongs"; - } - - if (count < 0) - { - return "its /Index gives a subsection a count of rows out of range"; - } - - // The last number is compared by subtraction: a count as large as a long would carry start + count - 1 past what - // one holds, and the rows past int.MaxValue would be read under negative numbers. - if (start is < 0 or > PdfObjectId.MaxNumber || count > PdfObjectId.MaxNumber - start + 1) + if (SubsectionFault(ranges, i, start, count) is { } fault) { - return string.Create( - CultureInfo.InvariantCulture, - $"its /Index gives a subsection of {count:N0} {(count == 1 ? "row" : "rows")} from object {start}, outside object numbers 0 to {PdfObjectId.MaxNumber}"); + return fault; } - checkedSubsections[i] = start.Value; - checkedSubsections[i + 1] = count.Value; + checkedSubsections[i] = start.GetValueOrDefault(); + checkedSubsections[i + 1] = count.GetValueOrDefault(); } subsections = checkedSubsections; return null; } + /// Says what is wrong with how many values an /Index holds, or null when it holds pairs. + private static string? PairsFault(PdfArray ranges) => ranges.Count == 0 || ranges.Count % 2 != 0 + ? string.Create( + CultureInfo.InvariantCulture, + $"its /Index holds {ranges.Count} {(ranges.Count == 1 ? "value" : "values")}, not pairs of a first object number and a count of rows") + : null; + + /// + /// Says what is wrong with the subsection an /Index gives at , or null when nothing is: its + /// first number is an object number, and its count numbers object numbers alone. + /// + /// The /Index, whose written elements a fault describes. + /// Where the subsection's first number lies. + /// The first number, when it is an integer. + /// The count, when it is an integer. + private static string? SubsectionFault(PdfArray ranges, int at, long? start, long? count) + { + if (start is null || count is null) + { + return $"its /Index holds {DescribeValue(ranges[start is null ? at : at + 1])} where an integer belongs"; + } + + if (count < 0) + { + return "its /Index gives a subsection a count of rows out of range"; + } + + // The last number is compared by subtraction: a count as large as a long would carry start + count - 1 past what one + // holds, and the rows past int.MaxValue would be read under negative numbers. + return start is < 0 or > PdfObjectId.MaxNumber || count > PdfObjectId.MaxNumber - start + 1 + ? string.Create( + CultureInfo.InvariantCulture, + $"its /Index gives a subsection of {count:N0} {(count == 1 ? "row" : "rows")} from object {start}, outside object numbers 0 to {PdfObjectId.MaxNumber}") + : null; + } + /// /// Reads the offset a section's /Prev or /XRefStm gives, or -1 when it gives none. /// @@ -3877,10 +5055,21 @@ public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? /// /Prev 576066 0 R, naming an object the file lacks — names no section, and the sections it should have named /// are reported missing, to be looked for by rebuilding the index. A negative integer is no offset either. Once a guard /// filled the index, though, a reference that leads to what the guard hides () names what is - /// unknown, and the chain ends there for the guard, which is reported, not for the file (#175). + /// unknown, and the chain ends there for the guard, which is reported, not for the file (#175). So does one that leads + /// to an object cut, as a cross-reference stream's values do: the guard is + /// reached once the chain is read (#263). /// - private long SectionOffset(PdfDictionary trailer, PdfName key, long section) + /// The section's trailer, or its stream's dictionary. + /// The entry to read, /Prev or /XRefStm. + /// The offset of the section, which a report names. + /// + /// What the entry is, when it leads to an object a guard cut, which ends the chain — "its /Prev is the reference 9 0 R, + /// which one of the reader's limits cut" —; null otherwise. + /// + private long SectionOffset(PdfDictionary trailer, PdfName key, long section, out string? hidden) { + hidden = null; + var value = trailer.GetRaw(key); if (value is null) @@ -3902,6 +5091,17 @@ private long SectionOffset(PdfDictionary trailer, PdfName key, long section) return -1; } + // So does one leading to an object a guard cut, whose loading, once the chain is read, reaches the guard (#263). The + // older sections, and their trailers, are the guard's to hide. + if (value is PdfReference && Follow(key, value, entry: true, out _, out var cut, LengthLinks) is { } fault && cut is { } past) + { + _cutByGuard = true; + _structure.TrailersHiddenByGuard = true; + (_chainValuesCut ??= []).Add(past); + hidden = fault; + return -1; + } + _indexIncomplete = true; _structure.Add(new XRefSectionRecord(FileQuote.Name(key), -1, section) { @@ -4080,7 +5280,7 @@ private static int RowsHeld(ReadOnlySpan rows) => /// /// Makes room in the index for the a section is about to give, so that the map is sized once /// rather than grown by doubling, and for no more numbers than lets it - /// take from here on, nor than the newest section's /Size numbers once it is read: one more than the document's + /// take from here on, nor than the newest section's /Size numbers, written direct, once it is read: one more than the document's /// highest object number (ISO 32000-1, 7.5.5), which an older section's rows, superseded, add nothing to. The caller /// bounds the rows by those the section holds: a count the file declares is a claim. /// @@ -4095,7 +5295,10 @@ private void ReserveIndex(long rows) var most = _guard.Bound(PdfLimit.IndexEntries); var room = (int)Math.Min(rows, int.MaxValue); - if (_xref.Trailer.GetInteger(PdfName.Size) is { } size) + // The /Size as written, which ISO 32000-1 makes direct (Table 15, 7.5.8.2), an integral real read as the integer it + // is: a reference is not read through to make room by, as the newest cross-reference stream's beside an /Index, which + // numbers no row, would be while the chain is. + if (_xref.Trailer.GetRaw(PdfName.Size) is { } written and not PdfReference && written.AsInteger() is { } size) { // A size the index holds as many entries as already asks for no room, nor does one below zero, which is not // narrowed to an int that would. One the sections read so far number past is stale, and they stand for it. @@ -4350,7 +5553,7 @@ private PdfObject LoadFromEntry(PdfObjectId id, XRefEntry entry) /// past its end. Returns false when nothing with that number is there. /// private bool TryParseObjectAt(int number, long offset, out PdfObject value, long recordedAt = RecordedAnywhere) => - TryParseNumberedAt(offset, number, PdfLimit.Object, out value, out _, out _, recordedAt: recordedAt); + TryParseNumberedAt(offset, number, PdfLimit.Object, out value, out _, out _, out _, recordedAt: recordedAt); /// /// Parses what starts at an exact offset — object , any object for @@ -4372,6 +5575,7 @@ private bool TryParseObjectAt(int number, long offset, out PdfObject value, long /// The guard that bounds the window. /// The object read. /// The number of the object found, even when refuses it. + /// Where in the file the object read starts, past the white space and comments before it. /// /// Whether the reading kept met a fault of the syntax, or may have, the pending diagnostics full: reported now or /// already, by an earlier reading of the same bytes. @@ -4393,6 +5597,7 @@ private bool TryParseNumberedAt( PdfLimit limit, out PdfObject value, out int foundNumber, + out long headerAt, out bool syntaxFault, int initialWindow = InitialObjectWindow, Func? keep = null, @@ -4400,6 +5605,7 @@ private bool TryParseNumberedAt( { value = PdfNull.Instance; foundNumber = 0; + headerAt = offset; syntaxFault = false; if (offset < 0 || offset >= _source.Length) @@ -4505,6 +5711,7 @@ private bool TryParseNumberedAt( RecordEndObj(noted ? number : DirectObject, foundNumber, parser.EndObj, cut, offset); RecordStreamLength(noted ? number : DirectObject, foundNumber, parsed, lengthFault); value = parsed; + headerAt = offset + parser.HeaderStart; return true; } } @@ -4519,8 +5726,8 @@ private bool TryParseNumberedAt( /// Parses object at an exact offset, as does, but keeps nothing of the /// reading: what the parser notices is dropped, nothing is recorded, and an object that still runs past the window /// once allows no larger one is not read rather than reported, which - /// says. A stream is not read: its data is looked for in the window alone, and the method - /// returns false. + /// says, then giving what the cut left of it. A stream is not + /// read: whatever its data, the method returns false. /// private bool TryReadUnrecorded(PdfObjectId id, long offset, out PdfObject value, out bool pastGuard) { @@ -4542,13 +5749,14 @@ private bool TryReadUnrecorded(PdfObjectId id, long offset, out PdfObject value, { using var window = _source.GetWindow(offset, windowSize); var cut = window.Length == windowSize && offset + window.Length < _source.Length; - var parser = new PdfObjectParser(window.Memory, offset, this, _pending, endsData: !cut); + // The file says where a token the window's edge reaches ends, as it does to a load; no stream's data is read. + var parser = new PdfObjectParser(window.Memory, offset, this, _pending, endsData: !cut, fileEdge: this); var read = parser.TryReadIndirectObject(out var found, out var parsed); // A header read whole that names another object, or another generation, is not this object, however far // its body runs, the chain reading each object under its reference's own number and generation: no guard - // hides it. - if (read && found != id) + // hides it. A stream is not read either, wherever its data ends. + if (read && (found != id || parsed is PdfStream)) { return false; } @@ -4557,7 +5765,10 @@ private bool TryReadUnrecorded(PdfObjectId id, long offset, out PdfObject value, { if (windowSize >= maxWindow) { + // What the cut left of the object, which a caller may take as a place to look at, never as its value: a + // header read whole that names another object, or a stream, returned above. pastGuard = true; + value = read ? parsed : PdfNull.Instance; return false; } @@ -4608,7 +5819,10 @@ private long RecordedAtFor(int number, XRefEntry entry) /// Records whether endobj follows an object read at : it does not when another /// token follows it, or when the file ends first — not when the window the reader offered ended first, where /// what follows was never seen. A stream whose data runs past the window is seen to its end all the same: the - /// file is asked what follows its endstream (#55). + /// file is asked what follows its endstream (#55). A reading that finds it, or cannot tell where the object + /// ends — a stream whose end a guard hides —, takes back what an earlier one recorded: the chain reads a + /// cross-reference stream whose /Length it cannot read up to the first endstream it holds, which its rows + /// may spell (#224). /// private void RecordEndObj(int number, int found, EndObjState state, bool cut, long offset) { @@ -4621,7 +5835,7 @@ private void RecordEndObj(int number, int found, EndObjState state, bool cut, lo { _endObjMissing[found] = offset; } - else if (state == EndObjState.Present) + else if (state is EndObjState.Present or EndObjState.Unknown) { _endObjMissing.Remove(found); } @@ -4655,8 +5869,9 @@ private void RecordStreamLength(int number, int found, PdfObject parsed, StreamL } /// - /// Marks a stream whose data reaches the edge of the window a guard stopped at: the reader cut it there, - /// where its endstream was still to be found, and its lost tail is the guard's, not the file's. + /// Marks a stream whose data reaches the edge of the window a guard stopped at — its dictionary ran to that edge, the + /// data no search can be anchored at past it —: the reader cut it there, and its lost tail is the guard's, not the + /// file's. A stream whose data starts within the window is found by its endstream, which no window bounds. /// private static void MarkCutByGuard(PdfObject parsed, long windowEnd) { @@ -4669,7 +5884,7 @@ private static void MarkCutByGuard(PdfObject parsed, long windowEnd) private static string LimitSubject(int number, int bound) => number switch { DirectObject => $"A trailer runs past {PdfLimitGuard.FormatLength(bound)}; only what lies within it was read.", - AnyObject => $"A cross-reference stream's dictionary runs past {PdfLimitGuard.FormatLength(bound)}; only what lies within it was read.", + AnyObject => $"The dictionary of the object read as a cross-reference stream runs past {PdfLimitGuard.FormatLength(bound)}; only what lies within it was read.", _ => string.Create( CultureInfo.InvariantCulture, $"Object {number} runs past {PdfLimitGuard.FormatLength(bound)}, its stream data aside; only what lies within it was read."), @@ -4690,6 +5905,22 @@ private void ReachLimit(PdfLimit limit, string what, long position) } } + /// + /// Reaches for the stream whose data starts at , + /// as does: its message is made once, as the stream is reported once, however often its data is + /// read. + /// + private void ReachStreamLength(long position) + { + _guardsReached++; + + if (!_limitsReached.Contains((PdfLimit.StreamLength, position))) + { + _guard.Reach(PdfLimit.StreamLength, _diagnostics, _guard.DescribeStreamLengthCut(), position); + _limitsReached.Add((PdfLimit.StreamLength, position)); + } + } + /// /// Reports, once for each member, that cut member /// of the object stream whose data starts at : its members share that place, and each is @@ -4852,9 +6083,7 @@ internal ObjectStreamHeaderResult ReadObjectStreamHeader( return ObjectStreamHeaderResult.Unreadable; } - var mark = _diagnostics.GetMark(); - var data = Filters.PdfFilterPipeline.Decode(stream, _diagnostics, out var leftEncoded); - var cut = LimitSince(_diagnostics, mark) || stream.Data.CutByGuard; + var data = Filters.PdfFilterPipeline.Decode(stream, _diagnostics, out var leftEncoded, out var cut); // Data the guard on filters left encoded holds none of the stream's objects: what its header says is unknown. if (leftEncoded) @@ -5872,14 +7101,22 @@ private void PreserveChainIndex() } /// - /// A stream's data in the file, read the first time something asks for it, and refused, read or not, once the - /// document is disposed. + /// A stream's data in the file: where it lies and how long it is, read each time it is asked for, kept by nothing + /// (#170), and refused once the document is disposed. /// + /// + /// Streams whose data share a stretch of the file — overlapping objects —, or the many streams of a document read once + /// each, hold none of it: what a caller keeps is its own, and memory follows what is being decoded. A source in + /// memory lends the bytes where they lie, without a copy, valid for as long as its memory is; any other reads them + /// into an array of their length. No more than bytes are read: past it, + /// the data is cut there, marked cut by a guard, and the guard reported, once for each stream. + /// private sealed class FileStreamData(PdfFileReader reader, long offset, int length, PdfLimitGuard guard) : PdfStreamData { - private byte[]? _bytes; private bool _cutByGuard; + private bool _endHidden; + public override int Length => length; public override long Position => offset; @@ -5888,22 +7125,40 @@ private sealed class FileStreamData(PdfFileReader reader, long offset, int lengt internal override bool CutByGuard => _cutByGuard; + internal override bool EndHidden => _endHidden; + public void MarkCutByGuard() => _cutByGuard = true; + public void MarkEndHidden() => _endHidden = true; + /// The document was disposed. + /// + /// The data runs past , and the document was opened to throw when it does. + /// public override ReadOnlyMemory GetBytes() { ObjectDisposedException.ThrowIf(reader._closed, typeof(PdfDocument)); - if (_bytes is not null) + var count = length; + var bound = guard.Bound(PdfLimit.StreamLength); + + if (count > bound) + { + _cutByGuard = true; + count = bound; + reader.ReachStreamLength(offset); + } + + if (reader._source.TryLend(offset, count, out var lent)) { - return _bytes; + return lent; } - var buffer = new byte[length]; + // A source that gives less than it holds leaves the array longer than what it gave: the bytes it gave are + // handed back, not copied again. + var buffer = new byte[count]; var read = reader._source.Read(offset, buffer); - _bytes = read == length ? buffer : buffer[..read]; - return _bytes; + return buffer.AsMemory(0, read); } } @@ -6072,7 +7327,7 @@ public void Clear() } /// The objects packed inside one object stream, and where each of them starts. - private sealed class ObjectStreamContents + private sealed class ObjectStreamContents : IPdfFileEdge { private readonly ReadOnlyMemory _data; private readonly int[] _numbers; @@ -6120,6 +7375,14 @@ private ObjectStreamContents( public int Count => _numbers.Length; + /// + /// Determines whether a token that reaches a member window's edge at byte of the + /// decoded data ends there: no regular character follows. A window MaxObjectLength bounds ends inside the data; past + /// the end of data a guard cut there is nothing to ask, and the token stays undecided. + /// + bool IPdfFileEdge.TokenEndsAt(long absoluteOffset) => + absoluteOffset < _data.Length && !PdfCharacters.IsRegular(_data.Span[(int)absoluteOffset]); + /// /// Gets what keeping the contents costs, in bytes: the decoded data, the header's two tables and the lookup of its /// numbers, whether built yet or not — four bytes for each table and eight for the lookup, for each member (#160). @@ -6181,9 +7444,7 @@ private ObjectStreamContents( return null; } - var mark = diagnostics.GetMark(); - var data = Filters.PdfFilterPipeline.Decode(stream, diagnostics, out var leftEncoded); - var cutByGuard = LimitSince(diagnostics, mark) || stream.Data.CutByGuard; + var data = Filters.PdfFilterPipeline.Decode(stream, diagnostics, out var leftEncoded, out var cutByGuard); // Data the guard on filters left encoded holds none of the stream's objects, as data a guard cut before /First. if (leftEncoded) @@ -6310,7 +7571,7 @@ private ObjectStreamContents( var window = ObjectStreamMemberWindow.Of(_offsets, index, _first, _data.Length, _cutByGuard, maxLength); var parser = PdfObjectParser.ForObjectStreamMember( - _data[..window.End], _number, _position, expectedNumber, source, parsing, window.EndsData, window.EndsAtNextMember); + _data[..window.End], _number, _position, expectedNumber, source, parsing, window.EndsData, window.EndsAtNextMember, this); parser.Position = window.Start; var value = parser.ParseObject(); diff --git a/src/AdCodicem.Pdf/IO/PdfFileSource.cs b/src/AdCodicem.Pdf/IO/PdfFileSource.cs index 04aa2331..35c18077 100644 --- a/src/AdCodicem.Pdf/IO/PdfFileSource.cs +++ b/src/AdCodicem.Pdf/IO/PdfFileSource.cs @@ -94,6 +94,21 @@ private protected virtual bool TryGetMemory(long offset, int length, out ReadOnl return false; } + /// + /// Lends the bytes at where the source holds them, without a copy; + /// false for a source that reads them from a file, or bytes it does not hold. + /// + internal bool TryLend(long offset, int length, out ReadOnlyMemory memory) + { + if (offset < 0 || length < 0 || offset > Length - length) + { + memory = default; + return false; + } + + return TryGetMemory(offset, length, out memory); + } + /// /// Returns a view of bytes at , clamped to the file. /// The window must be disposed; for a file-backed source it holds a pooled buffer. diff --git a/src/AdCodicem.Pdf/IO/PdfLimit.cs b/src/AdCodicem.Pdf/IO/PdfLimit.cs index 7968a0ee..2aed807d 100644 --- a/src/AdCodicem.Pdf/IO/PdfLimit.cs +++ b/src/AdCodicem.Pdf/IO/PdfLimit.cs @@ -9,6 +9,9 @@ internal enum PdfLimit /// What one stream may decode to. DecodedStream, + /// How much of one stream's encoded data may be read at once. + StreamLength, + /// How long an object may be, the data of a stream aside. Object, diff --git a/src/AdCodicem.Pdf/IO/PdfLimitGuard.cs b/src/AdCodicem.Pdf/IO/PdfLimitGuard.cs index 32e003d9..aabebd62 100644 --- a/src/AdCodicem.Pdf/IO/PdfLimitGuard.cs +++ b/src/AdCodicem.Pdf/IO/PdfLimitGuard.cs @@ -36,6 +36,7 @@ internal sealed class PdfLimitGuard(PdfReaderLimits limits, bool throwOnLimit, P public int Bound(PdfLimit limit) => limit switch { PdfLimit.DecodedStream => limits.MaxDecodedStreamLength, + PdfLimit.StreamLength => limits.MaxStreamLength, PdfLimit.Object => limits.MaxObjectLength, PdfLimit.XRefSectionLength => limits.MaxXRefSectionLength, PdfLimit.XRefSectionCount => limits.MaxXRefSectionCount, @@ -67,6 +68,13 @@ public void Reach(PdfLimit limit, PdfDiagnostics? diagnostics, string what, long (diagnostics ?? documentDiagnostics)?.ReachLimit(Code(limit), message, position); } + /// + /// Says what did to a stream's data it cut, as its report starts: the + /// reader's report, once for the stream, and decoding's, in diagnostics a caller passes of its own. + /// + public string DescribeStreamLengthCut() => + $"A stream's data runs past {FormatLength(Bound(PdfLimit.StreamLength))}; only what lies within it was read."; + /// Writes a length the way a message states a bound: 256 MB, 64 KB, 1,000 bytes. public static string FormatLength(int length) => length switch { @@ -78,6 +86,7 @@ public void Reach(PdfLimit limit, PdfDiagnostics? diagnostics, string what, long private static string Code(PdfLimit limit) => limit switch { PdfLimit.DecodedStream => PdfDiagnosticCodes.LimitDecodedStream, + PdfLimit.StreamLength => PdfDiagnosticCodes.LimitStreamLength, PdfLimit.Object => PdfDiagnosticCodes.LimitObject, PdfLimit.XRefSectionLength => PdfDiagnosticCodes.LimitXRefSectionLength, PdfLimit.XRefSectionCount => PdfDiagnosticCodes.LimitXRefSectionCount, @@ -89,6 +98,7 @@ public void Reach(PdfLimit limit, PdfDiagnostics? diagnostics, string what, long private static string Name(PdfLimit limit) => limit switch { PdfLimit.DecodedStream => nameof(PdfReaderLimits.MaxDecodedStreamLength), + PdfLimit.StreamLength => nameof(PdfReaderLimits.MaxStreamLength), PdfLimit.Object => nameof(PdfReaderLimits.MaxObjectLength), PdfLimit.XRefSectionLength => nameof(PdfReaderLimits.MaxXRefSectionLength), PdfLimit.XRefSectionCount => nameof(PdfReaderLimits.MaxXRefSectionCount), diff --git a/src/AdCodicem.Pdf/IO/PdfObjectParser.cs b/src/AdCodicem.Pdf/IO/PdfObjectParser.cs index efe31f00..a92ad7a9 100644 --- a/src/AdCodicem.Pdf/IO/PdfObjectParser.cs +++ b/src/AdCodicem.Pdf/IO/PdfObjectParser.cs @@ -26,30 +26,24 @@ internal ref struct PdfObjectParser /// private const int MaxDepth = 128; - /// The most white space accepted between a stream's data and its endstream keyword. - private const int MaxEndStreamGap = 4; - - /// How many bytes after a stream's data can decide whether endstream follows it. - internal const int EndStreamLookahead = MaxEndStreamGap + 9; - /// - /// How many bytes after a stream's data can decide, besides whether endstream follows it, whether - /// endobj follows that: the white space and keyword allows, then the few - /// bytes of white space or comment files put between the two keywords, and endobj with the byte after it. + /// How many bytes after a stream's data the reader reads first to decide whether endstream follows it, and + /// whether endobj follows that: the few bytes of white space or comment files put before and between the two + /// keywords, and the keywords with the byte after each. /// /// - /// No guard (ADR 34): it bounds what the reader sees of an object's end, not what it reads. A file that puts more - /// between the keywords reads the same, and whether its object ends with endobj goes unseen and unjudged, as - /// past a window's edge. + /// No bound (ADR 34): a first read, not what the reader reads. White space and comments that run past it are read on as + /// far as they go (#174), and what follows them decides. /// internal const int EndObjLookahead = 64; - private static ReadOnlySpan EndStreamKeyword => "endstream"u8; + private static ReadOnlySpan EndStreamKeyword => StreamEnds.EndStreamKeyword; private readonly ReadOnlyMemory _memory; private readonly IPdfObjectSource? _source; private readonly PdfDiagnostics? _diagnostics; private readonly IPdfStreamDataProvider? _streamData; + private readonly IPdfFileEdge? _fileEdge; private readonly long _baseOffset; /// @@ -63,27 +57,46 @@ internal ref struct PdfObjectParser /// Whether the parse met the end of its buffer, where it may have cut an object short. private bool _truncated; - /// - /// Whether a stream's /Length runs past what the file holds, which only a window reaching the end of the file - /// can settle: a larger buffer is asked for, though the parse met no edge, and what it reads after the stream is whole. - /// - private bool _lengthUnsettled; - /// /// Whether the end of the data was reported already: a cut leaves every construct around it open, and is reported /// once, for the innermost. /// private bool _cutReported; - private bool _endStreamMissing; + /// + /// Where the header of the object after this one starts, in the buffer, when one ended the containers this object left + /// open, where a value or a key was expected (#230); -1 when none did. + /// + private int _headerAt = -1; + + /// + /// Whether the header says is one a window's edge left undecided, which a larger window may show + /// is none: what the edge made of it, which ends the containers as a header does, but reports nothing and gives the key + /// before it no value (#230). + /// + private bool _headerUndecided; + + /// + /// Whether a window's edge left the look-ahead of an integer undecided — "12 0" that may be what the edge left of + /// "12 0 R", or "12 0 R" that may be what it left of "12 0 RG" —: what the parse reads after the integer is what the + /// edge made of the reference, not what follows the object (#224). + /// + private bool _lookAheadCut; + + /// + /// Whether the last stream read kept a declared length that neither endstream nor endobj follows: where + /// its object ends is unknown, and its endobj is not judged. + /// + private bool _endObjUnknown; + private EndObjState _endObj; /// What was found of the last stream read, when the file did not confirm its length. private StreamLengthFault? _lengthFault; /// - /// What follows the endstream of the last stream read, when its data ran past the buffer and the provider - /// was asked; null otherwise. + /// What follows the data of the last stream read, when the provider was asked: what follows its endstream, or, + /// when it has none, whether its object's endobj ends it; null otherwise. /// private EndObjState? _endObjAfterStream; @@ -93,6 +106,12 @@ internal ref struct PdfObjectParser /// private int _objectNumber; + /// + /// Whether the value is an indirect object's, read after its header: a report of the header that ended a container left + /// open names the object it ended, which a trailer, or a value read alone, is not. + /// + private bool _inObject; + /// /// The object stream member the buffer holds, when it is an object stream's decoded data rather than bytes of the /// file; null otherwise. @@ -115,13 +134,18 @@ internal ref struct PdfObjectParser /// Whether the bytes end where the file does; when they end at a window's edge instead, what that edge cuts short is /// not reported, and asks for a larger window. /// + /// + /// Says what the file holds right past the bytes given, for a parser no backs; one that + /// serves stream data says it too. + /// public PdfObjectParser( ReadOnlyMemory memory, long baseOffset = 0, IPdfObjectSource? source = null, PdfDiagnostics? diagnostics = null, IPdfStreamDataProvider? streamData = null, - bool endsData = true) + bool endsData = true, + IPdfFileEdge? fileEdge = null) { _memory = memory; _lexer = new PdfLexer(memory.Span); @@ -129,6 +153,7 @@ public PdfObjectParser( _source = source; _diagnostics = diagnostics; _streamData = streamData; + _fileEdge = fileEdge ?? streamData; _endsData = endsData; } @@ -148,6 +173,10 @@ public PdfObjectParser( /// data —, rather than where a guard cut it: only then is what its end cuts short the data's fault. /// /// Whether ends at the next member's start, which its reports then name. + /// + /// Says what the decoded data holds past 's end, where a window's edge leaves a keyword undecided: + /// the data is in memory, so a member a window ends exactly with reads whole, as under a larger window. + /// public static PdfObjectParser ForObjectStreamMember( ReadOnlyMemory data, int streamNumber, @@ -156,9 +185,10 @@ public static PdfObjectParser ForObjectStreamMember( IPdfObjectSource source, PdfDiagnostics diagnostics, bool endsData = true, - bool endsAtNextMember = false) + bool endsAtNextMember = false, + IPdfFileEdge? dataEdge = null) { - var parser = new PdfObjectParser(data, 0, source, diagnostics, endsData: endsData); + var parser = new PdfObjectParser(data, 0, source, diagnostics, endsData: endsData, fileEdge: dataEdge); parser._member = new ObjectStreamMember(streamNumber, dataStart, objectNumber, diagnostics, endsAtNextMember); return parser; } @@ -168,7 +198,7 @@ public static PdfObjectParser ForObjectStreamMember( /// buffer could read differently. A caller that can offer one should; a caller whose buffer is the /// whole input keeps what was read. /// - public readonly bool IsTruncated => _truncated || _lengthUnsettled; + public readonly bool IsTruncated => _truncated; /// /// Gets a value indicating whether the object stream member being read is written as a stream, or holds one: a @@ -197,7 +227,13 @@ public int Position } /// Parses the object at the current position. - public PdfObject ParseObject() => ParseValue(_lexer.Read(), 0); + public PdfObject ParseObject() + { + _headerAt = -1; + _headerUndecided = false; + _lookAheadCut = false; + return ParseValue(_lexer.Read(), 0); + } /// /// Determines whether an object header may still begin with and , @@ -260,12 +296,16 @@ public bool TryReadIndirectObject(out PdfObjectId id, out PdfObject value) id = new PdfObjectId((int)number.Integer, (int)generation.Integer); _objectNumber = id.Number; _headerStart = number.Start; + _headerAt = -1; + _headerUndecided = false; + _lookAheadCut = false; // An empty object, "2 0 obj endobj", reads its endobj as its value, and says so; that endobj is its own. var beforeValue = _lexer.Position; var empty = _lexer.Read().IsKeyword("endobj"u8); _lexer.Position = beforeValue; + _inObject = true; value = ParseObject(); if (empty) @@ -276,18 +316,22 @@ public bool TryReadIndirectObject(out PdfObjectId id, out PdfObject value) var afterValue = _lexer.Position; var next = _lexer.Read(); - var unseen = next.Kind == PdfTokenKind.EndOfInput || next.End >= _memory.Length; + + // A header a window's edge left undecided ended the value, or a reference the edge cut after its generation: what + // follows it is the edge's, not seen (#224, #230). + var unseen = next.Kind == PdfTokenKind.EndOfInput || next.End >= _memory.Length || _headerUndecided || _lookAheadCut; // What follows a stream whose data runs past the buffer lies past it too: the file was asked at the stream's // end, and its answer stands where the buffer has none (#55). - _endObj = next.IsKeyword("endobj"u8) + var endObj = IsEndObj(next); + _endObj = endObj ? EndObjState.Present - : _endStreamMissing ? EndObjState.Unknown + : _endObjUnknown ? EndObjState.Unknown : !unseen ? EndObjState.Absent : value is PdfStream && _endObjAfterStream is { } afterStream ? afterStream : EndObjState.Unseen; - if (!next.IsKeyword("endobj"u8)) + if (!endObj) { // A value followed by the end of the buffer, or by a token the end of the buffer may have cut, // may itself have been cut there: a string that lost its closing parenthesis, a dictionary whose @@ -308,8 +352,8 @@ public bool TryReadIndirectObject(out PdfObjectId id, out PdfObject value) private PdfObject ParseValue(PdfToken token, int depth) { // A value that reaches the end of the buffer may go on past it: "12" of "1234", a string without - // its closing delimiter, the "<" of a "<<". - if (token.Kind != PdfTokenKind.EndOfInput && token.End >= _memory.Length) + // its closing delimiter, the "<" of a "<<". An endobj the file says ends there does not: an empty object's. + if (token.Kind != PdfTokenKind.EndOfInput && token.End >= _memory.Length && !IsEndObj(token)) { _truncated = true; } @@ -333,7 +377,7 @@ private PdfObject ParseValue(PdfToken token, int depth) return PdfNull.Instance; case PdfTokenKind.Integer: - return ParseIntegerOrReference(token); + return ParseIntegerOrReference(token, depth); // A token a window's edge cut is the guard's, or the next window's, to judge: nothing is reported of it. case PdfTokenKind.Real when !double.IsFinite(token.Real) && AtWindowEdge(token): @@ -382,7 +426,7 @@ private PdfObject ParseValue(PdfToken token, int depth) return PdfNull.Instance; default: - if (!AtWindowEdge(token)) + if (!AtWindowEdge(token) || IsEndObj(token)) { Report(PdfDiagnosticCodes.SyntaxUnexpectedToken, "A token was found where a value was expected.", token.Start); } @@ -391,7 +435,11 @@ private PdfObject ParseValue(PdfToken token, int depth) } } - private PdfObject ParseIntegerOrReference(PdfToken token) + /// + /// Parses an integer, or the reference it starts, inside containers: inside one, the header of + /// the next object it starts ends them (#230), and reads as null where says. + /// + private PdfObject ParseIntegerOrReference(PdfToken token, int depth) { // "12 0 R" is only distinguishable from "12" by looking two tokens ahead. "0 0 R" is a reference too: object 0 // heads the free list and is never in use (ISO 32000-1, 7.5.4), so it reads as null, as a reference to any @@ -400,18 +448,42 @@ private PdfObject ParseIntegerOrReference(PdfToken token) var second = _lexer.Read(); var third = second.Kind == PdfTokenKind.Integer ? _lexer.Read() : second; - if (second.Kind == PdfTokenKind.Integer && third.IsKeyword("R"u8) && - token.Integer is >= 0 and <= PdfObjectId.MaxNumber && - second.Integer is >= 0 and <= PdfObjectId.MaxGeneration) + // An R a window's edge reaches is one only where the file says it ends there: "12 0 Rx" is no reference. + var referenced = second.Kind == PdfTokenKind.Integer && IsWholeKeyword(third, "R"u8); + + if (referenced && token.Integer is >= 0 and <= PdfObjectId.MaxNumber && second.Integer is >= 0 and <= PdfObjectId.MaxGeneration) { return new PdfReference(new PdfObjectId((int)token.Integer, (int)second.Integer), _source); } - // The buffer ended before the look-ahead could tell: "12 0" may be all there is, or what a - // window's edge left of "12 0 R". - if (third.Kind == PdfTokenKind.EndOfInput) + // The next object's header where an element or a value should be: this object lost both its closing delimiter and + // its endobj, and its containers end where the next object starts (#230). A value alone is no container. + if (depth > 0 && IsHeader(token, second, third)) + { + _headerAt = token.Start; + _lexer.Position = saved; + return PdfNull.Instance; + } + + // Inside a container, a window's edge may have cut a keyword that obj starts with after two integers, "12 0 ob": no + // reference, and maybe the next object's header, which a larger window shows. The containers end there, as at a + // header, and the number is no value of the file's. "12 0" alone may be what the edge left of a reference, and is + // kept as far as it was read, as any value the edge cuts. + if (depth > 0 && third.Kind == PdfTokenKind.Keyword && HeaderUndecided(token, second, third)) { _truncated = true; + _headerAt = token.Start; + _headerUndecided = true; + _lexer.Position = saved; + return PdfNull.Instance; + } + + // The buffer ended before the look-ahead could tell: "12 0" may be all there is, or what a window's edge left of + // "12 0 R" — or "12 0 R" what it left of "12 0 RG", which a larger window shows (#174). + if (third.Kind == PdfTokenKind.EndOfInput || (second.Kind == PdfTokenKind.Integer && third.IsKeyword("R"u8) && !referenced)) + { + _truncated = true; + _lookAheadCut = !_endsData; } _lexer.Position = saved; @@ -455,14 +527,37 @@ private PdfObject ParseArray(int depth, int openedAt) return array; // An endobj where an element should be ends the object, the array with it, as PDFBox and pdfium read it: - // the array takes nothing past it, though it is the next object's when this one lost its own (#230). + // the array takes nothing past it. An object that lost its own endobj meets the next object's header + // first, which ends it there (#230). case PdfTokenKind.Keyword when IsEndObj(token): _lexer.Position = token.Start; ReportCut("an array", openedAt, depth, atEndObj: true); return array; default: - array.Add(ParseValue(token, depth + 1)); + var element = ParseValue(token, depth + 1); + + if (_headerAt >= 0) + { + // The next object's header ends the array: where an element should be, reported here; inside an + // array or a dictionary the element opened, reported there, and that element kept as it was read. + if (_headerAt == token.Start) + { + if (!_headerUndecided) + { + ReportCut("an array", openedAt, depth, atHeader: true); + } + } + else + { + array.Add(element); + } + + _lexer.Position = _headerAt; + return array; + } + + array.Add(element); if (IsMemberStream) { @@ -506,8 +601,8 @@ private PdfObject ParseDictionaryOrStream(int depth, int openedAt) return dictionary; } - // An endobj where a key should be ends the object, the dictionary with it: the dictionary takes nothing past it, - // though it is the next object's when this one lost its own (#230). + // An endobj where a key should be ends the object, the dictionary with it: the dictionary takes nothing past it. + // An object that lost its own endobj meets the next object's header first, which ends it there (#230). if (IsEndObj(keyToken)) { _lexer.Position = keyToken.Start; @@ -517,6 +612,14 @@ private PdfObject ParseDictionaryOrStream(int depth, int openedAt) if (keyToken.Kind != PdfTokenKind.Name) { + if (StartsHeader(keyToken)) + { + _headerAt = keyToken.Start; + _lexer.Position = keyToken.Start; + ReportCut("a dictionary", openedAt, depth, atHeader: true); + return dictionary; + } + // A string where a key should be is a string all the same, which the end of the data can leave open inside // the dictionary: it is the innermost construct then. if (keyToken.Kind is PdfTokenKind.LiteralString) @@ -574,6 +677,29 @@ private PdfObject ParseDictionaryOrStream(int depth, int openedAt) return dictionary; } + if (_headerAt >= 0) + { + // The next object's header ends the dictionary: where the value should be, the key given none and reported + // here; inside an array or a dictionary the value opened, reported there, and that value kept. + if (_headerAt == valueToken.Start && _headerUndecided) + { + // One a window's edge left undecided is the edge's: the key keeps what it held, and nothing is said. + Give(dictionary, ref nullKeys, keyToken, key, PdfNull.Instance, keyAtEdge: true, valueCut: true); + } + else if (_headerAt == valueToken.Start) + { + ReportCut("a dictionary", openedAt, depth, valueMissing: true, atHeader: true); + Give(dictionary, ref nullKeys, keyToken, key, PdfNull.Instance, keyAtEdge, valueCut: false); + } + else + { + Give(dictionary, ref nullKeys, keyToken, key, value, keyAtEdge, valueCut: false); + } + + _lexer.Position = _headerAt; + return dictionary; + } + // A value the parse met a window's edge in may be what the edge made of it — a keyword cut short reads as null —: // it is kept as far as it was read, but removes nothing. Give(dictionary, ref nullKeys, keyToken, key, value, keyAtEdge, valueCut: PastWindowEdge); @@ -611,6 +737,20 @@ private PdfObject ParseDictionaryOrStream(int depth, int openedAt) return dictionary; } + /// + /// Reads a stream's data, whose stream keyword the lexer just read, and finds where it ends (#55, #174). + /// + /// + /// The declared length is confirmed by an endstream after it, white space and comments between, however many. + /// Unconfirmed, or with no length to confirm, the data is searched for its first endstream, which ends it: in the + /// bytes the parser holds, then in the file, up to the nearer of the first object header the bytes hold and the next + /// object the index places, or to the end of the data. Where a guard hid the length, a header the data spells that + /// stops the search leaves the data up to it to the guard. Stream data never counts toward the bytes the parser is given + /// to read an object through: a window is never grown to find an endstream. With none before the stop, the + /// object's endobj plays its part: it ends the data where it follows the declared length, or where it ends the + /// object before the stop; without it, a declared length that ends before the stop is kept, and the data otherwise + /// runs to the stop. + /// private PdfStream ReadStream(PdfDictionary dictionary) { var span = _memory.Span; @@ -621,140 +761,319 @@ private PdfStream ReadStream(PdfDictionary dictionary) var dataStart = _lexer.Position; _lengthFault = null; _endObjAfterStream = null; + _endObjUnknown = false; // The end-of-line after "stream" is not data. A buffer that ends on the keyword, or on the carriage - // return of a CR LF, cannot say where the data starts; a larger one can. - if (dataStart >= span.Length && (dataStart == keywordEnd || span[dataStart - 1] != (byte)'\n')) + // return of a CR LF, cannot say where the data starts: the file says, as it says what follows the data, since the + // data counts toward no window — not even one a guard ended there (#174) —; a parser the reader does not back + // needs a larger buffer, as does a keyword a regular character goes on, which is no stream keyword. One that ends + // the data says: the stream's data, which starts there, holds nothing. + if (!_endsData && dataStart >= span.Length && (dataStart == keywordEnd || span[dataStart - 1] != (byte)'\n')) { - _truncated = true; + var start = _streamData?.DataStart(_baseOffset + keywordEnd) ?? -1; + + if (start < 0) + { + _truncated = true; + } + else + { + dataStart = (int)(start - _baseOffset); + } } var declared = ReadLength(dictionary); var length = declared.Form == StreamLengthForm.Integer ? (int)declared.Value.GetValueOrDefault() : -1; - var beyondBuffer = length >= 0 && dataStart + (long)length > span.Length; + if (_truncated) + { + return Finish(dictionary, dataStart, Math.Max(length, 0), span.Length); + } + + // An integer read from a /Length a guard cut is a place to check, never a length kept when nothing confirms it. + var checkedAt = declared.Hint ?? length; + var confirmed = ConfirmsLength(span, dataStart, checkedAt); + + if (confirmed is null) + { + // Undecided, the buffer a window the parser alone was given: a larger one says. + _truncated = true; + return Finish(dictionary, dataStart, length, span.Length); + } + + if (confirmed == true) + { + return Finish(dictionary, dataStart, checkedAt, span.Length); + } + + // A cross-reference stream whose /Length the chain could not read keeps a carriage return before the line feed as + // data, which it may be — rows can end in 0x0D, and the length, once read, is checked against what was taken —, + // and is not stopped by rows that spell an object header (#182, #220). + var deferred = declared.Form == StreamLengthForm.Deferred; + var declaredEnd = length < 0 ? -1 : _baseOffset + dataStart + length; + var search = _streamData is not null + ? _streamData.FindEndStream(_objectNumber, _baseOffset + dataStart, declaredEnd, span[Math.Min(dataStart, span.Length)..], deferred) + : SearchBuffer(span, dataStart, declaredEnd, deferred); - if (beyondBuffer && _streamData is not null) + if (search is not { } settled) { - // The data lives past the window the reader gave us, into what the file holds: the file is asked - // whether endstream follows the declared length (#55). An attempt the reader makes again, its buffer - // too short to say where the data starts, asks nothing. - if (_baseOffset + dataStart + (long)length <= _streamData.SourceLength) + _truncated = true; + return Finish(dictionary, dataStart, Math.Max(length, 0), span.Length); + } + + if (settled.Length is { } found) + { + // A length read once the chain is, or one a guard left out of the index or cut, is not judged: nothing is said + // of it, nor a message made for it. + var hidden = declared.Form is StreamLengthForm.ObjectUnindexed or StreamLengthForm.ObjectCut; + + if (length != found && declared.Form != StreamLengthForm.Deferred && !hidden && + KeepLengthFault(Fault(declared, dataStart, found, found, EndStreamState.Found, null))) { - return IsTruncated - ? Finish(dictionary, dataStart, length, span.Length) - : ReadPastBuffer(_streamData, dictionary, declared, dataStart, length); + WarnLengthFault(PdfDiagnosticCodes.StreamLengthInvalid, DescribeRecovered(declared, found), dataStart); } - // Unless the file cannot hold it. Then only a window that reaches the end of the file can say - // whether the data stops at an "endstream" — the length is wrong — or at the end of the file — - // the stream is cut —, and the search below says which once the reader has offered one. The parse - // met no edge for that: what it reads after an endstream the search finds is whole. - _lengthUnsettled = true; + // Where a guard cut the /Length or left it out of the index, an endstream the data spells, as a valid stream's may — + // in a string, followed by endobj or not —, ends the search as the stream's own would: the length that tells them + // apart is the guard's. One what follows proves its own — the next object the index places, a cross-reference + // stream the chain read, the end of the file, a cross-reference section's keyword, after the object's endobj or + // in its place — ends the data, the file's. + // Any other's data is the guard's, cut by it, the object's endobj unjudged, and nothing is said of the file, as + // at a header the data spells (#224): one past which the searches could read nothing more, too. + if (hidden && _streamData is { } file && + (settled.EndObj is not (EndObjState.Absent or EndObjState.Present) || + !file.IsOwnEndStream(_baseOffset + dataStart + found, settled.EndObj, settled.NextObject))) + { + _endObjUnknown = true; + return Finish(dictionary, dataStart, found, span.Length, cutByGuard: true); + } + + // Proven its own, the endstream ends data that is the file's; but white space before it, which the search left out + // or took in, only the hidden length says is the data's or not: its end is the guard's to hide when decoded (#224). + _endObjAfterStream = _streamData is null ? null : settled.EndObj; + var endHidden = hidden && _streamData is { } proven && proven.EndsBesideWhiteSpace(_baseOffset + dataStart + found); + return Finish(dictionary, dataStart, found, span.Length, endHidden: endHidden); } - if (length < 0 || beyondBuffer || !ConfirmsLength(span, dataStart + length)) - { - // A length read once the cross-reference chain is keeps a carriage return before the line feed as data, which it - // may be: rows can end in 0x0D, and the length, once read, is checked against what was taken (#182). - var recovered = FindEndStream(span, dataStart, keepCarriageReturn: declared.Form == StreamLengthForm.Deferred); + return FinishWithoutEndStream(dictionary, declared, dataStart, length, settled); + } - if (recovered < 0) + /// + /// Ends the data of a stream whose search found no endstream before it stopped (#174): at the declared length, + /// when its object's endobj follows it; at that endobj, when it ends the object before the stop; at the + /// declared length, when it ends before the stop; at the stop otherwise — the next object, or the end of the data. + /// + private PdfStream FinishWithoutEndStream( + PdfDictionary dictionary, DeclaredLength declared, int dataStart, int length, StreamEndSearch search) + { + var span = _memory.Span; + var endOfData = _streamData is null ? span.Length : _streamData.SourceLength - _baseOffset; + var stop = search.NextObject is { } next ? next - _baseOffset : endOfData; + var fits = length >= 0 && dataStart + (long)length <= stop; + var upToStop = (int)Math.Min(stop - dataStart, int.MaxValue); + var missing = search.NextObject is null ? EndStreamState.MissingBeforeEndOfFile : EndStreamState.MissingBeforeNextObject; + + if (!search.Searched) + { + // The document's searches read as much of the file as they may: what lies past the bytes held is not looked at. + // Where a /Length a guard cut or left out of the index would say how far the data runs, where it ends is the + // guard's, as where an endstream found is not proven the stream's own: the data up to the stop is cut by it, the + // object's endobj unjudged, and nothing is said of the file (#224). + if (declared.Form is StreamLengthForm.ObjectCut or StreamLengthForm.ObjectUnindexed) { - _truncated = true; + _endObjUnknown = true; + return Finish(dictionary, dataStart, upToStop, span.Length, cutByGuard: true); + } - // An endobj after the data says the object ends there, without its endstream, and where it ends is - // then the reader's guess; with none, the file ends inside the object, and so without its endobj. - _endStreamMissing = span[dataStart..].IndexOf("endobj"u8) >= 0; - var rest = span.Length - dataStart; + var kept = fits ? length : upToStop; - ReportLengthFault( - PdfDiagnosticCodes.StreamTruncated, - _endStreamMissing + if (KeepLengthFault(Fault(declared, dataStart, kept, null, EndStreamState.NotSearched, null))) + { + WarnLengthFault( + PdfDiagnosticCodes.StreamLengthInvalid, + fits ? string.Create( CultureInfo.InvariantCulture, - $"The stream has no endstream before the endobj that follows its data; the {rest} bytes to the end of the file are taken as its data.") - : "A stream ran past the end of the file.", - Fault(declared, dataStart, rest, null, _endStreamMissing ? EndStreamState.MissingBeforeEndObj : EndStreamState.MissingBeforeEndOfFile, null)); + $"The stream declared {length} bytes, and no endstream follows them; the declared length is kept without a search, the document's searches for endstream having read as much of the file as they may.") + : string.Create( + CultureInfo.InvariantCulture, + $"{Describe(declared)}, and no endstream follows its data in the bytes read with its object; the {kept} bytes {(search.NextObject is null ? "to the end of the file" : "up to the next object")} are taken as its data without a search, the document's searches for endstream having read as much of the file as they may."), + dataStart); + } - return Finish(dictionary, dataStart, rest, span.Length); + _endObjUnknown = true; + return Finish(dictionary, dataStart, kept, span.Length); + } + + if (fits && search.EndObjAfterDeclared) + { + if (KeepLengthFault(Fault(declared, dataStart, length, null, EndStreamState.MissingBeforeEndObj, null))) + { + WarnLengthFault( + PdfDiagnosticCodes.StreamTruncated, + string.Create( + CultureInfo.InvariantCulture, + $"The stream has no endstream: its object's endobj follows the {length} bytes it declared, which are taken as its data."), + dataStart); + } + + _endObjAfterStream = EndObjState.Present; + return Finish(dictionary, dataStart, length, span.Length); + } + + if (search.ObjectEnd >= 0) + { + var taken = (int)Math.Min(search.ObjectEnd - _baseOffset - dataStart, int.MaxValue); + + if (KeepLengthFault(Fault(declared, dataStart, taken, null, EndStreamState.MissingBeforeEndObj, search.NextObject))) + { + WarnLengthFault( + PdfDiagnosticCodes.StreamTruncated, + string.Create( + CultureInfo.InvariantCulture, + $"{Describe(declared)}, and no endstream follows its data {Where(search.NextObject)}: its object's endobj ends it, after {taken} bytes."), + dataStart); } - if (length != recovered) + _endObjAfterStream = EndObjState.Present; + return Finish(dictionary, dataStart, taken, span.Length); + } + + if (fits) + { + if (KeepLengthFault(Fault(declared, dataStart, length, null, missing, search.NextObject))) { - ReportLengthFault( + WarnLengthFault( PdfDiagnosticCodes.StreamLengthInvalid, - DescribeRecovered(declared, recovered), - Fault(declared, dataStart, recovered, recovered, EndStreamState.Found, null)); + string.Create( + CultureInfo.InvariantCulture, + $"The stream declared {length} bytes, and no endstream follows them {Where(search.NextObject)}; the declared length is kept."), + dataStart); } - length = recovered; + // Where the object ends is unknown, and its endobj is not judged. + _endObjUnknown = true; + return Finish(dictionary, dataStart, length, span.Length); + } + + // A /Length a guard cut or left out of the index hides where the data ends, which a header the data spells stops the + // search at, as a valid stream's may — an embedded file's does —, no endobj of the object's own before it: the data + // up to it is the guard's, cut by it, the object's endobj unjudged, and nothing is said of the file (#224). + if (declared.Form is StreamLengthForm.ObjectCut or StreamLengthForm.ObjectUnindexed && search.HeaderStopped) + { + _endObjUnknown = true; + return Finish(dictionary, dataStart, upToStop, span.Length, cutByGuard: true); } - return Finish(dictionary, dataStart, length, span.Length); + // The data runs to the next object, which no declared length passes, or to the end of the file. The object lacks its + // endobj, unless what is taken as its data holds one, which may be the object's own: where it ends is then unknown. + if (KeepLengthFault(Fault(declared, dataStart, upToStop, null, missing, search.NextObject))) + { + WarnLengthFault( + search.NextObject is null ? PdfDiagnosticCodes.StreamTruncated : PdfDiagnosticCodes.StreamLengthInvalid, + string.Create( + CultureInfo.InvariantCulture, + $"{Describe(declared)}, and no endstream follows its data {Where(search.NextObject)}; the {upToStop} bytes {(search.NextObject is null ? "to the end of the file" : "up to that object")} are taken as its data."), + dataStart); + } + + _endObjUnknown = search.EndObjSeen; + _endObjAfterStream = search.EndObjSeen ? EndObjState.Unknown : EndObjState.Absent; + return Finish(dictionary, dataStart, upToStop, span.Length); + + // Where the search stopped, as a report says it. + static string Where(long? nextObject) => nextObject is { } at + ? string.Create(CultureInfo.InvariantCulture, $"before the next object, at {at}") + : "before the end of the file"; } /// - /// Reads a stream whose declared length runs past the buffer, into what the file holds (#55). The file is asked - /// whether endstream follows the declared length, and when it does not, searched for the first one after the - /// start of the data, before the next object or the end of the file: found, it ends the data; not found, or not - /// searched for once the document's searches have read as much as they may, the declared length is kept. Either is - /// reported, and the file answers what follows the endstream too. + /// Determines whether the declared length is confirmed by an endstream after it, white space and comments + /// between, however many: in the buffer, or, where the buffer cannot say, in the file, which the provider reads on as + /// far as they run (#174). Null when neither can say: the buffer is a window given to the parser alone. /// - private PdfStream ReadPastBuffer( - IPdfStreamDataProvider provider, PdfDictionary dictionary, DeclaredLength declared, int dataStart, int length) + private bool? ConfirmsLength(ReadOnlySpan span, int dataStart, int length) { - var span = _memory.Span; - - if (provider.CheckEndStream(_baseOffset + dataStart + (long)length) is { } afterEndStream) + if (length < 0) { - _endObjAfterStream = afterEndStream; - return Finish(dictionary, dataStart, length, span.Length); + return false; } - var search = provider.FindEndStream(_objectNumber, _baseOffset + dataStart, span[dataStart..]); - _endObjAfterStream = search.EndObj; + var dataEnd = dataStart + (long)length; - if (search.Length is { } found) + if (dataEnd <= span.Length) { - ReportLengthFault( - PdfDiagnosticCodes.StreamLengthInvalid, - string.Create(CultureInfo.InvariantCulture, $"The stream declared {length} bytes but ended after {found}."), - Fault(declared, dataStart, found, found, EndStreamState.Found, null)); + var end = StreamEnds.ReadEnd(span[(int)dataEnd..], reachesEndOfFile: _endsData); - return Finish(dictionary, dataStart, found, span.Length); + if (end.Kind != StreamEndKind.Undecided) + { + // What follows an endstream the buffer holds, when the buffer ends before it shows, is read on in the file, + // as it is past the buffer: where the window ends decides nothing of the object's endobj (#174). + if (end is { Kind: StreamEndKind.EndStream, EndObj: EndObjState.Unseen } && _streamData is not null) + { + _endObjAfterStream = _streamData.CheckEndStream(_baseOffset + dataEnd); + } + + return end.Kind == StreamEndKind.EndStream; + } + } + else if (_endsData) + { + return false; } - if (!search.Searched) + if (_streamData is null) { - ReportLengthFault( - PdfDiagnosticCodes.StreamLengthInvalid, - string.Create( - CultureInfo.InvariantCulture, - $"The stream declared {length} bytes, and no endstream follows them; the declared length is kept without a search, the document's searches for endstream having read as much of the file as they may."), - Fault(declared, dataStart, length, null, EndStreamState.NotSearched, null)); + return null; + } - return Finish(dictionary, dataStart, length, span.Length); + if (_baseOffset + dataEnd > _streamData.SourceLength) + { + return false; } - var before = search.NextObject is { } next - ? string.Create(CultureInfo.InvariantCulture, $"before the next object, at {next}") - : "before the end of the file"; + _endObjAfterStream = _streamData.CheckEndStream(_baseOffset + dataEnd); + return _endObjAfterStream is not null; + } - ReportLengthFault( - PdfDiagnosticCodes.StreamLengthInvalid, - string.Create( - CultureInfo.InvariantCulture, - $"The stream declared {length} bytes, and no endstream follows them {before}; the declared length is kept."), - Fault( - declared, - dataStart, - length, - null, - search.NextObject is null ? EndStreamState.MissingBeforeEndOfFile : EndStreamState.MissingBeforeNextObject, - search.NextObject)); - - return Finish(dictionary, dataStart, length, span.Length); + /// + /// Searches the buffer of a parser the reader does not back for the first endstream after the start of the data, + /// up to the first object header it holds, as the reader searches the file, and says what the object's endobj + /// does when it finds none; null when the buffer holds neither, and is a window a larger one may show more of. + /// + private readonly StreamEndSearch? SearchBuffer(ReadOnlySpan span, int dataStart, long declaredEnd, bool deferred) + { + Span carried = stackalloc byte[HeaderText.CarriedLength]; + Span carriedAt = stackalloc long[HeaderText.CarriedLength]; + var start = _baseOffset + dataStart; + var text = new HeaderText(carried, carriedAt, start); + text.Read(span[dataStart..], start); + + var dataEnd = StreamEnds.SearchStretch(text, endsRange: _endsData, deferred, out var header); + + if (dataEnd >= 0) + { + return new StreamEndSearch((int)(dataEnd - start), null, EndObjState.Unseen); + } + + if (header < 0 && !_endsData) + { + return null; + } + + var stop = header < 0 ? span.Length : (int)(header - _baseOffset); + var stretch = span[dataStart..stop]; + var tail = new EndObjTail(start); + tail.Read(stretch, start, _baseOffset + stop); + + return new StreamEndSearch(null, header < 0 ? null : header, EndObjState.Unknown) + { + HeaderStopped = header >= 0, + EndObjSeen = StreamEnds.HoldsEndObj(stretch, stretch.Length), + EndObjAfterDeclared = declaredEnd >= start && declaredEnd - _baseOffset <= stop && + StreamEnds.IsEndObjAt(span[..stop], (int)(declaredEnd - _baseOffset), reachesEndOfFile: true) == true, + ObjectEnd = tail.DataEnd(_baseOffset + stop), + }; } /// @@ -780,7 +1099,7 @@ private readonly DeclaredLength ReadLength(PdfDictionary dictionary) if (_streamData is not null) { - value = _streamData.ResolveLength(named.Id, out presence).Resolve(); + value = _streamData.ResolveLength(named, out presence).Resolve(); } else { @@ -790,6 +1109,22 @@ private readonly DeclaredLength ReadLength(PdfDictionary dictionary) presence = value is PdfNull ? ObjectPresence.Unproduced : ObjectPresence.Defined; } + // An integer the cut left, whole or parted, is only a place to look for endstream at, and no declared length; the + // object the guard cut is the one a chain of references from the /Length leads to (#224). + if (presence == ObjectPresence.Cut) + { + return new DeclaredLength( + StreamLengthForm.ObjectCut, + Reference: named.Target(PdfReference.MaxChainLength), + Hint: value.AsInteger() is { } hint and >= 0 and <= int.MaxValue ? (int)hint : null); + } + + // The object a guard left out of the index, as the one it cut, is the one the chain of references leads to. + if (presence == ObjectPresence.Unindexed) + { + return new DeclaredLength(StreamLengthForm.ObjectUnindexed, Reference: named.Target(PdfReference.MaxChainLength)); + } + if (presence != ObjectPresence.Defined) { return new DeclaredLength( @@ -797,7 +1132,6 @@ private readonly DeclaredLength ReadLength(PdfDictionary dictionary) { ObjectPresence.Missing => StreamLengthForm.ObjectMissing, ObjectPresence.Deferred => StreamLengthForm.Deferred, - ObjectPresence.Unindexed => StreamLengthForm.ObjectUnindexed, _ => StreamLengthForm.ObjectUnreadable, }, Reference: reference); @@ -835,16 +1169,23 @@ private readonly DeclaredLength ReadLength(PdfDictionary dictionary) }; /// Says what was wrong with a stream's /Length, its endstream found after bytes. - private static string DescribeRecovered(DeclaredLength declared, int recovered) + private static string DescribeRecovered(DeclaredLength declared, int recovered) => declared.Form == StreamLengthForm.Integer + ? string.Create(CultureInfo.InvariantCulture, $"The stream declared {declared.Value} bytes but ended after {recovered}.") + : string.Create(CultureInfo.InvariantCulture, $"{Describe(declared)}; its data ends after {recovered} bytes."); + + /// + /// Says how a stream's /Length was written, as a report starts: the length it declared, or why it gives none. + /// + private static string Describe(DeclaredLength declared) { if (declared.Form == StreamLengthForm.Integer) { - return string.Create(CultureInfo.InvariantCulture, $"The stream declared {declared.Value} bytes but ended after {recovered}."); + return string.Create(CultureInfo.InvariantCulture, $"The stream declared {declared.Value} bytes"); } if (declared.Form == StreamLengthForm.Absent) { - return string.Create(CultureInfo.InvariantCulture, $"The stream has no /Length; its data ends after {recovered} bytes."); + return "The stream has no /Length"; } var written = declared.Reference is not { } reference @@ -853,10 +1194,12 @@ private static string DescribeRecovered(DeclaredLength declared, int recovered) { StreamLengthForm.ObjectMissing => Names(reference, "which the file lacks"), StreamLengthForm.ObjectUnreadable => Names(reference, "which could not be read"), + StreamLengthForm.ObjectCut => Names(reference, "which one of the reader's limits cut"), + StreamLengthForm.ObjectUnindexed => Names(reference, "which one of the reader's limits left out of the index"), _ => Names(reference, "which holds " + Value(declared)), }; - return string.Create(CultureInfo.InvariantCulture, $"The stream's /Length {written}; its data ends after {recovered} bytes."); + return "The stream's /Length " + written; static string Names(PdfObjectId reference, string what) => string.Create(CultureInfo.InvariantCulture, $"names object {reference.Number} {reference.Generation}, {what}"); @@ -894,32 +1237,36 @@ private readonly StreamLengthFault Fault( }; /// - /// Keeps what was found of the stream being read, and reports it unless a reading of the same stream the reader - /// kept already has: a stream parsed again — after the cache let it go, or the index was rebuilt — is reported once. + /// Keeps what was found of the stream being read, and says whether to report it: unless a reading of the same stream the + /// reader kept already has — a stream parsed again, after the cache let it go or the index was rebuilt, is reported + /// once —, or no diagnostics are kept. The message is made only then. /// - private void ReportLengthFault(string code, string message, StreamLengthFault fault) + private bool KeepLengthFault(StreamLengthFault fault) { // A /Length read once the cross-reference chain is: whether the data ends where it should is known then (#182). One a - // guard left out of the index is unknown, and the guard's report says why. - if (fault.Form is StreamLengthForm.Deferred or StreamLengthForm.ObjectUnindexed) + // guard left out of the index, or cut, is unknown, and the guard's report says why; a missing endstream is still the + // file's (#224), and the only fault reported of it, where no header the data spells, nor an endstream its endobj + // does not follow, stops the search. + if (fault.Form == StreamLengthForm.Deferred) { - return; + return false; } _lengthFault = fault; - - if (_streamData?.IsLengthFaultReported(fault.DataStart) != true) - { - _diagnostics?.Warn(code, message, fault.DataStart); - } + return _diagnostics is not null && _streamData?.IsLengthFaultReported(fault.DataStart) != true; } - private PdfStream Finish(PdfDictionary dictionary, int dataStart, int length, int bufferLength) + /// Reports what was found of the stream being read, whose data starts at . + private readonly void WarnLengthFault(string code, string message, int dataStart) => + _diagnostics!.Warn(code, message, _baseOffset + dataStart); + + private PdfStream Finish( + PdfDictionary dictionary, int dataStart, int length, int bufferLength, bool cutByGuard = false, bool endHidden = false) { length = Math.Max(length, 0); var data = _streamData is not null - ? _streamData.Create(_baseOffset + dataStart, length) + ? _streamData.Create(_baseOffset + dataStart, length, cutByGuard, endHidden) : PdfStreamData.FromMemory(_memory.Slice(dataStart, Math.Min(length, bufferLength - dataStart))); var afterData = (int)Math.Min((long)dataStart + length, bufferLength); @@ -934,99 +1281,6 @@ private PdfStream Finish(PdfDictionary dictionary, int dataStart, int length, in return new PdfStream(dictionary, data); } - /// - /// Determines whether the declared length is confirmed by an endstream keyword where the data - /// ends. When the keyword may lie across the end of the buffer, the file is asked rather than the buffer: - /// a stream whose data ends a few bytes short of the window's edge is not a truncated stream. - /// - private readonly bool ConfirmsLength(ReadOnlySpan span, int dataEnd) - { - if (IsEndStreamAt(span, dataEnd)) - { - return true; - } - - return _streamData is not null && - dataEnd + (long)EndStreamLookahead > span.Length && - _streamData.IsEndStreamAt(_baseOffset + dataEnd); - } - - /// - /// Determines whether endstream starts at , after at most - /// bytes of white space. - /// - internal static bool IsEndStreamAt(ReadOnlySpan span, long position) - { - if (position < 0 || position > span.Length) - { - return false; - } - - var index = (int)position; - - // Conforming files put an end-of-line before "endstream"; some put nothing, some put spaces. - var limit = Math.Min(index + MaxEndStreamGap, span.Length); - while (index < limit && PdfCharacters.IsWhitespace(span[index])) - { - index++; - } - - return span[index..].StartsWith(EndStreamKeyword); - } - - /// - /// Determines whether endstream starts at the start of , after at most - /// bytes of white space, and what follows the keyword: endobj, another token, - /// or nothing the bytes show — unless they reach the end of the file, which then follows it. - /// - /// Null when no endstream starts there. - internal static EndObjState? ReadStreamEnd(ReadOnlySpan bytes, bool reachesEndOfFile) - { - if (!IsEndStreamAt(bytes, 0)) - { - return null; - } - - // Read as the parser reads what follows a stream: "endstreamx" is no keyword, and is what follows the data. - var lexer = new PdfLexer(bytes); - var keyword = lexer.Read(); - var next = keyword.IsKeyword(EndStreamKeyword) ? lexer.Read() : keyword; - - if (!reachesEndOfFile && (next.Kind == PdfTokenKind.EndOfInput || next.End >= bytes.Length)) - { - return EndObjState.Unseen; - } - - return next.IsKeyword("endobj"u8) ? EndObjState.Present : EndObjState.Absent; - } - - /// - /// Finds where a stream really ends when its declared length is wrong, and returns its true length. - /// - private static int FindEndStream(ReadOnlySpan span, int dataStart, bool keepCarriageReturn = false) - { - var index = span[dataStart..].IndexOf(EndStreamKeyword); - if (index < 0) - { - return -1; - } - - var end = dataStart + index; - - // The end-of-line that precedes the keyword belongs to the syntax, not to the data. - if (end > dataStart && span[end - 1] == (byte)'\n') - { - end--; - } - - if (!keepCarriageReturn && end > dataStart && span[end - 1] == (byte)'\r') - { - end--; - } - - return end - dataStart; - } - /// /// Consumes a container whose contents are being discarded, keeping nesting balanced: the one that opens at /// , inside containers. @@ -1051,6 +1305,12 @@ private void SkipContainer(PdfTokenKind endKind, int openedAt, int enclosing) ReportCut(endKind == PdfTokenKind.ArrayEnd ? "an array" : "a dictionary", openedAt, enclosing, atEndObj: true); return; + case PdfTokenKind.Integer when StartsHeader(token): + _headerAt = token.Start; + _lexer.Position = token.Start; + ReportCut(endKind == PdfTokenKind.ArrayEnd ? "an array" : "a dictionary", openedAt, enclosing, atHeader: true); + return; + case PdfTokenKind.ArrayStart when endKind == PdfTokenKind.ArrayEnd: case PdfTokenKind.DictionaryStart when endKind == PdfTokenKind.DictionaryEnd: depth++; @@ -1209,9 +1469,11 @@ private void ReportIfUnterminated(PdfToken token, string what, int depth) /// The bytes a string took to the end of the data, or 0 for an array or a dictionary. /// Whether the cut came after a dictionary's key, before its value. /// Whether an endobj made the cut, rather than the end of the data. - private void ReportCut(string what, int openedAt, int enclosing, int swallowed = 0, bool valueMissing = false, bool atEndObj = false) + /// Whether the next object's header made the cut, rather than the end of the data (#230). + private void ReportCut( + string what, int openedAt, int enclosing, int swallowed = 0, bool valueMissing = false, bool atEndObj = false, bool atHeader = false) { - if ((!_endsData && !atEndObj) || _cutReported) + if ((!_endsData && !atEndObj && !atHeader) || _cutReported) { return; } @@ -1224,7 +1486,10 @@ private void ReportCut(string what, int openedAt, int enclosing, int swallowed = return; } - var data = atEndObj ? "An endobj ended the object" : End; + // A trailer, or a value read alone, is no object a header can end: the header came inside the construct left open. + var data = atEndObj ? "An endobj ended the object" + : atHeader ? (_inObject ? "An object header ended the object" : "An object header came") + : End; var content = swallowed > 0 ? string.Create(CultureInfo.InvariantCulture, $", which takes the {swallowed:N0} {(swallowed == 1 ? "byte" : "bytes")} from where it opens to that end") : valueMissing ? ", which was never closed, before the value of its last key" : ", which was never closed"; @@ -1254,9 +1519,107 @@ private void ReportCut(string what, int openedAt, int enclosing, int swallowed = /// /// Determines whether is endobj, and whole: one that reaches the end of a buffer that is - /// a window's edge may be the start of a longer keyword. + /// a window's edge may be the start of a longer keyword, which the file says when it backs the parser, so that an object + /// a window ends exactly with reads whole, as under a larger window, wherever its endobj comes — after its value, where a + /// container it left open is, or as all it holds (#174); a parser no file backs asks for a larger buffer. + /// + private readonly bool IsEndObj(PdfToken token) => IsWholeKeyword(token, "endobj"u8); + + /// + /// Determines whether is the keyword , and whole: one that reaches the + /// end of a buffer that is a window's edge may be the start of a longer keyword — endobjX, RG —, which the + /// file says when it backs the parser; a parser no file backs asks for a larger buffer. + /// + private readonly bool IsWholeKeyword(PdfToken token, ReadOnlySpan keyword) => + token.IsKeyword(keyword) && + (!AtWindowEdge(token) || (!PastWindowEdge && _fileEdge is { } file && file.TokenEndsAt(_baseOffset + token.End))); + + /// + /// Determines whether , read where a key is expected, starts the next object's header + /// (): the two tokens after it are read, and the lexer left where it was. + /// + private bool StartsHeader(PdfToken number) + { + if (number.Kind != PdfTokenKind.Integer || _member is not null) + { + return false; + } + + var saved = _lexer.Position; + var generation = _lexer.Read(); + var keyword = generation.Kind == PdfTokenKind.Integer ? _lexer.Read() : generation; + _lexer.Position = saved; + + // Tokens a window's edge leaves undecided may be a header a larger window shows: the parse met the edge, and what it + // reads from here — the number and the generation, as keys — is no fault of the file's. + if (HeaderUndecided(number, generation, keyword)) + { + _truncated = true; + } + + return IsHeader(number, generation, keyword); + } + + /// + /// Determines whether a window's edge leaves undecided whether a number and the two tokens after it are the next + /// object's header: the edge comes before a generation, before a keyword, or inside one that obj starts with, + /// and no whole token read rules a header out — a number that is no object number, a generation that is none + /// (). An object stream's member is followed by no header. + /// + private readonly bool HeaderUndecided(PdfToken number, PdfToken generation, PdfToken keyword) => + !_endsData && _member is null && MayBeginHeader(number, generation, _memory.Length) && + MayBeHeaderBytes(number, generation, keyword) && + (generation.Kind == PdfTokenKind.EndOfInput || + (generation.Kind == PdfTokenKind.Integer && + (keyword.Kind == PdfTokenKind.EndOfInput || + (keyword.Kind == PdfTokenKind.Keyword && keyword.End >= _memory.Length && "obj"u8.StartsWith(keyword.Text))))); + + /// + /// Determines whether three tokens are an object header as the bytes hold one, the rule the search for a stream's + /// endstream stops at (): N G obj, digits alone for a number from 1 to + /// 2,147,483,647 and a generation from 0 to 65,535, white space alone between them — no comment —, and obj a + /// whole keyword, which one a window's edge reaches may not be. An object stream's member is never followed by a + /// header: its members have none (#230). + /// + private readonly bool IsHeader(PdfToken number, PdfToken generation, PdfToken keyword) + { + if (_member is not null || number.Kind != PdfTokenKind.Integer || generation.Kind != PdfTokenKind.Integer || + !keyword.IsKeyword("obj"u8) || AtWindowEdge(keyword) || + number.Integer is < 1 or > PdfObjectId.MaxNumber || generation.Integer is < 0 or > PdfObjectId.MaxGeneration) + { + return false; + } + + var span = _memory.Span; + return !span[number.Start..number.End].ContainsAnyExceptInRange((byte)'0', (byte)'9') && + !span[generation.Start..generation.End].ContainsAnyExceptInRange((byte)'0', (byte)'9') && + !span[number.End..generation.Start].ContainsAnyExcept(PdfCharacters.Whitespace) && + !span[generation.End..keyword.Start].ContainsAnyExcept(PdfCharacters.Whitespace); + } + + /// + /// Determines whether the bytes read of what may be a header, up to the window's edge, keep to the header's byte rule + /// (): digits alone, white space alone between the tokens — no sign, no comment. /// - private readonly bool IsEndObj(PdfToken token) => token.IsKeyword("endobj"u8) && !AtWindowEdge(token); + private readonly bool MayBeHeaderBytes(PdfToken number, PdfToken generation, PdfToken keyword) + { + var span = _memory.Span; + + if (span[number.Start..number.End].ContainsAnyExceptInRange((byte)'0', (byte)'9')) + { + return false; + } + + if (generation.Kind != PdfTokenKind.Integer) + { + return !span[number.End..].ContainsAnyExcept(PdfCharacters.Whitespace); + } + + var upTo = keyword.Kind == PdfTokenKind.EndOfInput ? span.Length : keyword.Start; + return !span[generation.Start..generation.End].ContainsAnyExceptInRange((byte)'0', (byte)'9') && + !span[number.End..generation.Start].ContainsAnyExcept(PdfCharacters.Whitespace) && + !span[generation.End..upTo].ContainsAnyExcept(PdfCharacters.Whitespace); + } /// Gets what ended the buffer, as a report of what it cut short names it. private readonly string End => _member switch @@ -1372,6 +1735,15 @@ public void Report(string code, string message, long position) => /// The object it names, when it is a reference. /// What kind of value it holds, when that is not an integer. /// The real number, name or boolean it holds, as the file wrote it; null for any other value. + /// + /// The integer an object a guard cut held before the cut, when it is one a stream can have: where to look for an + /// endstream first, and no declared length (#224). + /// private readonly record struct DeclaredLength( - StreamLengthForm Form, long? Value = null, PdfObjectId? Reference = null, string? Kind = null, PdfObject? Written = null); + StreamLengthForm Form, + long? Value = null, + PdfObjectId? Reference = null, + string? Kind = null, + PdfObject? Written = null, + int? Hint = null); } diff --git a/src/AdCodicem.Pdf/IO/StreamEnd.cs b/src/AdCodicem.Pdf/IO/StreamEnd.cs new file mode 100644 index 00000000..b1c9cfb2 --- /dev/null +++ b/src/AdCodicem.Pdf/IO/StreamEnd.cs @@ -0,0 +1,14 @@ +namespace AdCodicem.Pdf.IO; + +/// What the bytes after a stream's data say of its end (). +/// Whether endstream follows the data, does not, or the bytes cannot say. +/// What follows the endstream, when one does. +/// Where the endstream token ends in the bytes, when one does. +internal readonly record struct StreamEnd(StreamEndKind Kind, EndObjState EndObj = EndObjState.Unseen, int KeywordEnd = 0) +{ + /// No endstream follows the data. + public static StreamEnd None => new(StreamEndKind.None); + + /// The bytes end in the white space, the comment or the keyword that may lead to endstream. + public static StreamEnd Undecided => new(StreamEndKind.Undecided); +} diff --git a/src/AdCodicem.Pdf/IO/StreamEndKind.cs b/src/AdCodicem.Pdf/IO/StreamEndKind.cs new file mode 100644 index 00000000..e02e7fc9 --- /dev/null +++ b/src/AdCodicem.Pdf/IO/StreamEndKind.cs @@ -0,0 +1,14 @@ +namespace AdCodicem.Pdf.IO; + +/// Whether the bytes after a stream's data hold its endstream. +internal enum StreamEndKind : byte +{ + /// Something else follows the data, after white space and comments. + None, + + /// endstream follows the data, after white space and comments. + EndStream, + + /// The bytes end before they can say: in white space, a comment, or what may be the keyword's start. + Undecided, +} diff --git a/src/AdCodicem.Pdf/IO/StreamEndSearch.cs b/src/AdCodicem.Pdf/IO/StreamEndSearch.cs index bfcb2f65..0d5debdd 100644 --- a/src/AdCodicem.Pdf/IO/StreamEndSearch.cs +++ b/src/AdCodicem.Pdf/IO/StreamEndSearch.cs @@ -15,7 +15,34 @@ namespace AdCodicem.Pdf.IO; /// is unknown. /// /// -/// Whether the file was searched at all: it is not once the document's searches have read as much of it as they may, -/// and then nothing was found. +/// Whether the file was searched as far as the search stops: past the bytes the parser holds, it is not once the +/// document's searches have read as much of it as they may, and then nothing past them was found. /// -internal readonly record struct StreamEndSearch(int? Length, long? NextObject, EndObjState EndObj, bool Searched = true); +internal readonly record struct StreamEndSearch(int? Length, long? NextObject, EndObjState EndObj, bool Searched = true) +{ + /// + /// Gets where the data ends, in the file, when no endstream was found and the object's own endobj, which + /// only white space separates from where the search stopped, ends it, the end-of-line before it left out; -1 when none + /// does. + /// + public long ObjectEnd { get; init; } = -1; + + /// + /// Gets a value indicating whether, no endstream found, an endobj follows the declared length, white + /// space and comments between, before where the search stopped. + /// + public bool EndObjAfterDeclared { get; init; } + + /// + /// Gets a value indicating whether, no endstream found, the bytes searched hold an endobj keyword + /// anywhere: the object may then end inside what is taken as its data, and whether it ends with endobj is + /// unknown. + /// + public bool EndObjSeen { get; init; } + + /// + /// Gets a value indicating whether, no endstream found, an object header the bytes hold stopped the search, which + /// then gives, rather than the next object the index places or the end of the file. + /// + public bool HeaderStopped { get; init; } +} diff --git a/src/AdCodicem.Pdf/IO/StreamEnds.cs b/src/AdCodicem.Pdf/IO/StreamEnds.cs new file mode 100644 index 00000000..258f82d4 --- /dev/null +++ b/src/AdCodicem.Pdf/IO/StreamEnds.cs @@ -0,0 +1,148 @@ +namespace AdCodicem.Pdf.IO; + +/// +/// The rules that say where a stream's data ends, read over bytes: whether endstream follows the data, where the +/// first one after its start lies, and whether an endobj ends the object. The parser applies them to the bytes it +/// holds, and the reader to the file past them (), so that a stream reads the same +/// wherever its window ends (#174). +/// +internal static class StreamEnds +{ + /// The keyword that ends a stream's data. + public static ReadOnlySpan EndStreamKeyword => "endstream"u8; + + /// The keyword that ends an object. + public static ReadOnlySpan EndObjKeyword => "endobj"u8; + + /// + /// Gives where the first byte at or after , where a stream's declared length ends, lies that is + /// neither white space nor part of a comment, or the length of when they run to its end. ISO + /// 32000-1 (7.2.3) makes a comment outside a string or a stream's data count as white space; one counts here only after + /// white space, since a % right where the length ends is the data's, a length too short, and the rest of its line + /// the rest of the data. + /// + public static int SkipGap(ReadOnlySpan bytes, int index) + { + if (index < bytes.Length && bytes[index] == (byte)'%') + { + return index; + } + + var lexer = new PdfLexer(bytes, index); + lexer.SkipWhitespaceAndComments(); + return lexer.Position; + } + + /// + /// Reads, at the start of , whether endstream follows a stream's data — after white + /// space and comments, as many as there are (#174), a comment only after white space () — and what + /// follows the keyword: endobj, another token, or nothing the bytes show. + /// + /// The bytes after the data. + /// Whether the bytes end where the file does, which then follows what they hold. + public static StreamEnd ReadEnd(ReadOnlySpan bytes, bool reachesEndOfFile) + { + var at = SkipGap(bytes, 0); + var lexer = new PdfLexer(bytes, at); + var rest = bytes[at..]; + + if (!rest.StartsWith(EndStreamKeyword)) + { + // White space, a comment or the start of the keyword that runs to the end of the bytes may lead to it past them. + return !reachesEndOfFile && EndStreamKeyword.StartsWith(rest) ? StreamEnd.Undecided : StreamEnd.None; + } + + // Read as the parser reads what follows a stream: "endstreamx" is no keyword, and is what follows the data. + var keyword = lexer.Read(); + + if (keyword.End >= bytes.Length && !reachesEndOfFile) + { + return StreamEnd.Undecided; + } + + if (!keyword.IsKeyword(EndStreamKeyword)) + { + return new StreamEnd(StreamEndKind.EndStream, EndObjState.Absent, keyword.End); + } + + var next = lexer.Read(); + + if (!reachesEndOfFile && (next.Kind == PdfTokenKind.EndOfInput || next.End >= bytes.Length)) + { + return new StreamEnd(StreamEndKind.EndStream, EndObjState.Unseen, keyword.End); + } + + return new StreamEnd( + StreamEndKind.EndStream, next.IsKeyword(EndObjKeyword) ? EndObjState.Present : EndObjState.Absent, keyword.End); + } + + /// + /// Determines whether endobj, a whole keyword, starts at , where a stream's declared length + /// ends, once white space and comments are skipped (); null when they, or the keyword, run to the + /// end of , which do not end the file. + /// + public static bool? IsEndObjAt(ReadOnlySpan bytes, int index, bool reachesEndOfFile) + { + var at = SkipGap(bytes, index); + var rest = bytes[at..]; + + if (rest.Length <= EndObjKeyword.Length) + { + if (!reachesEndOfFile && EndObjKeyword.StartsWith(rest)) + { + return null; + } + + return rest.SequenceEqual(EndObjKeyword); + } + + return rest.StartsWith(EndObjKeyword) && !PdfCharacters.IsRegular(rest[EndObjKeyword.Length]); + } + + /// + /// Determines whether the first bytes of hold an endobj + /// keyword: a read of a search that may end inside its object. + /// + public static bool HoldsEndObj(ReadOnlySpan bytes, int before) => + bytes[..Math.Clamp(before, 0, bytes.Length)].IndexOf(EndObjKeyword) >= 0; + + /// + /// Searches one read of a search for endstream — the parser's bytes, or a window of the file — for the first + /// endstream, and for the first object header before it: returns where the data ends in the file, the + /// end-of-line before the keyword left out, or -1 when the read holds no endstream or a header comes first, + /// whose start in the file is then , -1 otherwise. + /// + /// + /// The overlap between reads puts the end-of-line before a keyword first seen in a read inside it, unless the keyword + /// starts the data. A cross-reference stream whose /Length the chain could not read is searched as + /// : its rows may spell a header, and may end in a carriage return, so neither stops nor + /// trims it (#220). + /// + /// The bytes read, after what the reads before carry into them. + /// Whether they end where the search does: at the next object the index places, or the end of the file. + /// Whether the stream's /Length is one the chain could not read, which is no fault. + /// Where the header that stopped the search starts in the file, or -1. + public static long SearchStretch(in HeaderText text, bool endsRange, bool deferred, out long header) + { + var span = text.Bytes; + var keyword = span.IndexOf(EndStreamKeyword); + header = deferred ? -1 : text.FindHeader(keyword >= 0 ? keyword : span.Length, endsRange); + + if (keyword < 0 || header >= 0) + { + return -1; + } + + if (keyword > 0 && span[keyword - 1] == (byte)'\n') + { + keyword--; + } + + if (!deferred && keyword > 0 && span[keyword - 1] == (byte)'\r') + { + keyword--; + } + + return text.PositionOf(keyword); + } +} diff --git a/src/AdCodicem.Pdf/IO/StreamLengthFault.cs b/src/AdCodicem.Pdf/IO/StreamLengthFault.cs index 1b3f110e..6ca9a8d8 100644 --- a/src/AdCodicem.Pdf/IO/StreamLengthFault.cs +++ b/src/AdCodicem.Pdf/IO/StreamLengthFault.cs @@ -16,7 +16,10 @@ internal readonly record struct StreamLengthFault /// Gets how the /Length was written, or what the object it names holds. public required StreamLengthForm Form { get; init; } - /// Gets the object the /Length names, or null when it is written in the stream's dictionary. + /// + /// Gets the object the /Length names — for a length a guard cut or left out of the index, the one a chain of + /// references from it leads to, which the guard cut or left out —, or null when it is written in the stream's dictionary. + /// public PdfObjectId? Reference { get; init; } /// @@ -50,8 +53,9 @@ internal readonly record struct StreamLengthFault /// /// Gets the offset of the object the search for the endstream stopped at — where an index places it, or where - /// its header starts in the file's bytes —, when is - /// ; null otherwise. + /// its header starts in the file's bytes —, when it stopped at one: is then + /// , or when + /// the object's own endobj ended the data before it; null otherwise. /// public long? NextObject { get; init; } diff --git a/src/AdCodicem.Pdf/IO/StreamLengthForm.cs b/src/AdCodicem.Pdf/IO/StreamLengthForm.cs index 78cadac7..72151bb4 100644 --- a/src/AdCodicem.Pdf/IO/StreamLengthForm.cs +++ b/src/AdCodicem.Pdf/IO/StreamLengthForm.cs @@ -32,8 +32,21 @@ internal enum StreamLengthForm : byte Deferred, /// - /// A reference to an object one of the reader's guards left out of the index: whether the file holds it is unknown. The - /// data is taken up to its endstream, and nothing is reported: the guard's report says why. + /// A reference to an object one of the reader's guards left out of the index, or that a chain of references from it + /// leads to: whether the file holds it is unknown. The data ends where its endstream ends it, or as a stream with + /// none ends; nothing is reported of the length, whose fault is the guard's, which reported it, and a missing + /// endstream is still the file's (#224) — but where the search stops at an object header the data spells, + /// or at an endstream its object's endobj does not follow, which only the length could tell from the + /// end of the data: the data up to there is then the guard's. /// ObjectUnindexed, + + /// + /// A reference to an object one of the reader's guards cut, or that a chain of references from it leads to: what it + /// holds may lie past the cut. An integer read before the cut is only a place to look for endstream at, and + /// never kept as a length; the data is otherwise taken up to its endstream. Nothing is reported of the length, + /// whose fault is the guard's, which reported it; a missing endstream is still the file's (#224), but where + /// the search stops as says. + /// + ObjectCut, } diff --git a/src/AdCodicem.Pdf/IO/XRef/XRefSectionRecord.cs b/src/AdCodicem.Pdf/IO/XRef/XRefSectionRecord.cs index 75077127..06de2ebe 100644 --- a/src/AdCodicem.Pdf/IO/XRef/XRefSectionRecord.cs +++ b/src/AdCodicem.Pdf/IO/XRef/XRefSectionRecord.cs @@ -43,6 +43,12 @@ public XRefSectionRecord(string namedBy, long namedOffset, long namedFrom) /// Gets where the section was read, which is where it was named unless it was relocated. public long Offset { get; private set; } + /// + /// Gets a value indicating whether the section was found away from where it was named, whatever became of it there: + /// read, or kept from being read by a guard, which leaves its state . + /// + public bool WasRelocated => Offset != NamedOffset; + /// Gets what became of the section. public XRefSectionState State { get; set; } @@ -58,7 +64,11 @@ public XRefSectionRecord(string namedBy, long namedOffset, long namedFrom) /// public string? RefusedRow { get; set; } - /// Gets a value indicating whether a cross-reference stream holds fewer rows than it declares. + /// + /// Gets a value indicating whether a cross-reference stream holds fewer rows than it declares: the rows it holds place + /// objects, but where a value they are read with is one a guard cut, beside which the data holds too few whatever the + /// value holds. + /// public bool Incomplete { get; set; } /// @@ -67,12 +77,39 @@ public XRefSectionRecord(string namedBy, long namedOffset, long namedFrom) /// public bool CutByLimit { get; set; } + /// + /// Gets a value indicating whether what cut the section is a value it is read with — one its rows are read with, or its + /// /Length — that leads to an object one of the reader's guards cut: the section was found whole, and is the + /// guard's to keep from being read (#263). A guard that cut the section's own dictionary leaves it unknown, but for + /// what declared itself a section before the cut (, or a table's xref keyword). + /// + public bool CutByValue { get; set; } + + /// + /// Gets a value indicating whether the cross-reference stream's dictionary, as far as it was read, gives + /// /Type /XRef: what is there declared itself a section before any guard's cut. + /// + public bool TypedXRef { get; set; } + /// /// Gets a value indicating whether cut the section's /// trailer: what lay past the cut — an older section's /Prev, a key a newer trailer gives — is unknown. /// public bool TrailerCutByLimit { get; set; } + /// + /// Gets what the section's /Prev or /XRefStm is, when it leads to an object one of the reader's guards cut + /// — "its /Prev is the reference 9 0 R, which one of the reader's limits cut" —, which ends the chain here: the sections + /// it names are the guard's to hide (#263). Null when nothing did. + /// + public string? ChainHiddenByGuard { get; set; } + + /// + /// Gets a value indicating whether what says is the section's /XRefStm: what + /// the guard hides is then the cross-reference stream of the section's revision, and any section older than it. + /// + public bool ChainHiddenAtStream { get; set; } + /// Gets the section's own trailer — a cross-reference stream's dictionary —, or null when none was read. public PdfDictionary? Trailer { get; set; } @@ -117,7 +154,11 @@ public void RelocateTo(XRefSectionRecord attempt) RefusedRow = attempt.RefusedRow; Incomplete = attempt.Incomplete; CutByLimit = attempt.CutByLimit; + CutByValue = attempt.CutByValue; + TypedXRef = attempt.TypedXRef; TrailerCutByLimit = attempt.TrailerCutByLimit; + ChainHiddenByGuard = attempt.ChainHiddenByGuard; + ChainHiddenAtStream = attempt.ChainHiddenAtStream; Trailer = attempt.Trailer; TrailerFault = attempt.TrailerFault; TrailerPosition = attempt.TrailerPosition; diff --git a/src/AdCodicem.Pdf/Objects/PdfReference.cs b/src/AdCodicem.Pdf/Objects/PdfReference.cs index 34d01204..767f9b2b 100644 --- a/src/AdCodicem.Pdf/Objects/PdfReference.cs +++ b/src/AdCodicem.Pdf/Objects/PdfReference.cs @@ -52,13 +52,19 @@ public override PdfObject Resolve() /// Gives the object whose value gives: this reference's, or the last one a chain of objects whose /// values are references leads to, as far as follows it: a chain too long for it ends at its /// 31st object, the last whose value takes, so that no object past it is taken for the target. + /// It loads the objects before that one, which resolving the reference loads too. /// - internal PdfObjectId Target() + /// + /// How many objects the chain is followed through: one more for a value read from the object a reference names and then + /// resolved — a stream's /Length, a trailer's /Prev —, whose 32nd object gives it. + /// + internal PdfObjectId Target(int objects = MaxChainLength - 1) { var reference = this; - // Resolve gives the value of a chain's 31st object at most: that object is the furthest a chain is followed to. - for (var depth = 1; depth < MaxChainLength - 1; depth++) + // Resolve gives the value of a chain's 31st object at most: it loads a 32nd, whose value it drops (#173), and the 31st + // is the furthest whose value it takes. + for (var depth = 1; depth < objects; depth++) { if (reference.Source?.GetObject(reference.Id) is not PdfReference next) { diff --git a/src/AdCodicem.Pdf/Objects/PdfStream.cs b/src/AdCodicem.Pdf/Objects/PdfStream.cs index 6e93548e..2a1e8b07 100644 --- a/src/AdCodicem.Pdf/Objects/PdfStream.cs +++ b/src/AdCodicem.Pdf/Objects/PdfStream.cs @@ -29,7 +29,20 @@ public PdfStream(PdfDictionary dictionary, PdfStreamData data) /// Gets the number of encoded bytes. public int RawLength => Data.Length; - /// Returns the encoded bytes, exactly as they appear in the file. + /// Returns the encoded bytes, as they appear in the file, up to the reader's bound. + /// + /// A stream read from a document has its bytes read from the document's source each time they are asked for, and kept + /// by nothing: a document held in memory lends them where they lie — a view of that memory, which keeps it alive and + /// changes as it does —, and any other source is read again. No more than + /// bytes are read: past it, what lies within it comes back, and + /// is reported once for the stream. A stream built over + /// bytes in memory gives them back as they are. + /// + /// The stream's document is disposed. + /// + /// The data runs past , and the document was opened with + /// . + /// public ReadOnlyMemory GetRawBytes() => Data.GetBytes(); /// diff --git a/src/AdCodicem.Pdf/Objects/PdfStreamData.cs b/src/AdCodicem.Pdf/Objects/PdfStreamData.cs index d8db742b..cf3c94c5 100644 --- a/src/AdCodicem.Pdf/Objects/PdfStreamData.cs +++ b/src/AdCodicem.Pdf/Objects/PdfStreamData.cs @@ -31,7 +31,27 @@ public abstract class PdfStreamData /// internal virtual bool CutByGuard => false; - /// Returns the encoded bytes, exactly as they appear in the file. + /// + /// Gets a value indicating whether where the data ends is the guard's to hide, though the data is the file's: an + /// endstream proven the stream's own ended it under a /Length a guard hid, with white space, or comments, + /// before the keyword that only the length could say are the data's or not (#224). Its first filter decodes it up to + /// where that run starts, what lies before it the file's, its running out there saying nothing of the file, and the + /// end of what it may lack the guard's; Flate data whose deflate body ends in the data as taken is whole. + /// + internal virtual bool EndHidden => false; + + /// Returns the encoded bytes, as they appear in the file, up to the reader's bound. + /// + /// Data read from a document held in memory is a view of that memory, which keeps it alive and changes as it does. Data + /// read from a document gives no more than bytes: past it, + /// what lies within it comes back, fewer bytes than says, and + /// is reported once in the document's diagnostics. + /// + /// The data was read from a document that is disposed. + /// + /// The data runs past , and its document was opened with + /// . + /// public abstract ReadOnlyMemory GetBytes(); /// Creates stream data over bytes already in memory. diff --git a/src/AdCodicem.Pdf/Objects/PdfStreamDecodingExtensions.cs b/src/AdCodicem.Pdf/Objects/PdfStreamDecodingExtensions.cs index 69f252f0..2056d6d6 100644 --- a/src/AdCodicem.Pdf/Objects/PdfStreamDecodingExtensions.cs +++ b/src/AdCodicem.Pdf/Objects/PdfStreamDecodingExtensions.cs @@ -19,7 +19,10 @@ public static class PdfStreamDecodingExtensions /// /// /// Decoding stops at an image filter: a stream encoded with DCTDecode comes back as the JPEG it - /// already is, rather than as pixels nobody asked for. + /// already is, rather than as pixels nobody asked for. Data no filter changes comes back as + /// gives it — with no filter, or where decoding stops at the first: an image filter, + /// one the library does not know or could not decode the data with, the end of the chain —: from a document held in + /// memory, a view of that memory, which keeps it alive and changes as it does. /// /// /// Data that is damaged decodes as far as it can, and what decoded is kept: a Flate stream that lost its @@ -42,9 +45,16 @@ public static class PdfStreamDecodingExtensions /// reported once as ; the filters before the last share the /// bound for what they pass along together (). The /// filters after one the bound stopped decode what it kept, as data the reader cut: that it runs out is not - /// reported as damage. A /Filter array that names more filters than + /// reported as damage. So do they decode data that runs past + /// , of which the reader reads what lies within it, reported as + /// : once in the document's diagnostics, and in diagnostics of your + /// own each time the stream is decoded. A /Filter array that + /// names more filters than /// ends decoding before the first past them, reported as - /// when decoding reaches it. + /// when decoding reaches it. A /Filter, a /DecodeParms + /// or a parameter that leads to an object one of the document's guards cut, or left out of the index, ends decoding + /// the same way, at the filter it names or whose parameters it gives: the guard was reported in the document's + /// diagnostics where it was reached, as everything loading a value meets is, and nothing more is reported. /// /// /// The filters and the values of their parameters are read before any of the data is: a parameter held in an @@ -53,11 +63,14 @@ public static class PdfStreamDecodingExtensions /// /// /// - /// Decoding reaches one of its document's guards — the data decodes past - /// , or decoding reaches the filter past - /// —, and the document was opened with + /// Decoding reaches one of its document's guards — the encoded data runs past + /// , the data decodes past + /// , decoding reaches the filter past + /// , or loading a value the filters read reaches one, + /// among them —, and the document was opened with /// . /// + /// The stream was read from a document that is disposed. public static ReadOnlyMemory Decode(this PdfStream stream, PdfDiagnostics? diagnostics = null) { ArgumentNullException.ThrowIfNull(stream); @@ -68,12 +81,19 @@ public static ReadOnlyMemory Decode(this PdfStream stream, PdfDiagnostics? /// Determines whether the stream's data is an encoded image that leaves alone: whether its /// /Filter entry names an image filter before anything that ends decoding whatever the data holds — a filter /// the library does not know, an element that names no filter, a filter past - /// . + /// , a filter's value or parameter that leads to an object + /// one of the reader's guards cut or left out of the index. /// /// /// None of the data is read: whether a filter before the image filter fails on it, or whether the image is sound, - /// takes decoding it, and reports what it meets. + /// takes decoding it, and reports what it meets. The filters' parameters are read as decoding reads + /// them: one held in an object stream decodes that stream first. /// + /// + /// Loading a value the filters read reaches one of its document's guards, and the document was opened with + /// . + /// + /// A value the filters read is to be loaded from a document that is disposed. public static bool HasImageFilter(this PdfStream stream) { ArgumentNullException.ThrowIfNull(stream); diff --git a/src/AdCodicem.Pdf/Validation/PdfValidationRuleIds.cs b/src/AdCodicem.Pdf/Validation/PdfValidationRuleIds.cs index 9fa04f27..9d3427bd 100644 --- a/src/AdCodicem.Pdf/Validation/PdfValidationRuleIds.cs +++ b/src/AdCodicem.Pdf/Validation/PdfValidationRuleIds.cs @@ -72,8 +72,10 @@ public static class PdfValidationRuleIds /// /// A value a section's trailer — a classic table's, or a cross-reference stream's dictionary — gives the reader to read - /// the chain is written as the specification does not allow, and read all the same: a reference where it must be - /// direct, a real with no fractional part where it must be an integer. . + /// the chain is written as the specification does not allow: a reference where it must be direct, a real with no + /// fractional part where it must be an integer. The reader reads such a value all the same, or does not read it at all + /// — a hybrid file's stream's own /Prev, a /W element past the third —, which is then judged as written. + /// . /// public const string FileTrailerValueWrong = "file.trailer-value-wrong"; diff --git a/src/AdCodicem.Pdf/Validation/Rules/CheckedInPartRule.cs b/src/AdCodicem.Pdf/Validation/Rules/CheckedInPartRule.cs index 066b7919..ccc367a5 100644 --- a/src/AdCodicem.Pdf/Validation/Rules/CheckedInPartRule.cs +++ b/src/AdCodicem.Pdf/Validation/Rules/CheckedInPartRule.cs @@ -71,6 +71,23 @@ public void Check(ValidationContext context) $"One of the reader's limits stopped it reading the cross-reference section at offset {section.Offset} whole: what lies past the limit was not checked."), remedy: null); } + + // A hybrid file's /XRefStm names the cross-reference stream of the section's own revision, which a /Prev may + // not follow: what the guard hides is that stream, and whatever is older. + if (section.ChainHiddenByGuard is { } hidden) + { + context.Report( + this, + PdfValidationLocation.AtPosition(section.Offset), + section.ChainHiddenAtStream + ? string.Create( + CultureInfo.InvariantCulture, + $"The cross-reference section at offset {section.Offset} names the cross-reference stream of its revision, but {hidden}: that stream, and any section older than it, were not checked.") + : string.Create( + CultureInfo.InvariantCulture, + $"The cross-reference section at offset {section.Offset} names older sections, but {hidden}: they were not checked."), + remedy: null); + } } foreach (var finding in context.Probe.NotChecked) diff --git a/src/AdCodicem.Pdf/Validation/Rules/EndObjMissingRule.cs b/src/AdCodicem.Pdf/Validation/Rules/EndObjMissingRule.cs index 7bdf16c3..b2f2c651 100644 --- a/src/AdCodicem.Pdf/Validation/Rules/EndObjMissingRule.cs +++ b/src/AdCodicem.Pdf/Validation/Rules/EndObjMissingRule.cs @@ -11,9 +11,9 @@ namespace AdCodicem.Pdf.Validation.Rules; /// was evidently meant, a warning (ADR 45). An empty object, 2 0 obj endobj, has its endobj. /// /// -/// The objects checked are those the reader read, reached from the trailer; what follows a stream whose data runs past -/// the window the reader first reads through is not seen until #55 checks such a stream's length, nor what follows an -/// object one of the reader's limits cut. An object stream's members have no endobj of their own. +/// The objects checked are those the reader read, reached from the trailer; what follows an object one of the reader's +/// limits cut is not seen, nor what follows a stream whose declared length no endstream nor endobj follows, +/// kept with its object's end unknown. An object stream's members have no endobj of their own. /// /// internal sealed class EndObjMissingRule : IValidationRule diff --git a/src/AdCodicem.Pdf/Validation/Rules/PageTreePageOrphanedRule.cs b/src/AdCodicem.Pdf/Validation/Rules/PageTreePageOrphanedRule.cs index f7afb5c4..da2a2acc 100644 --- a/src/AdCodicem.Pdf/Validation/Rules/PageTreePageOrphanedRule.cs +++ b/src/AdCodicem.Pdf/Validation/Rules/PageTreePageOrphanedRule.cs @@ -92,7 +92,8 @@ private static bool IsIndexSound(ValidationContext context) foreach (var section in structure.Sections) { - if (section.State is not (XRefSectionState.Read or XRefSectionState.Relocated) || section.Incomplete || section.CutByLimit) + if (section.State is not (XRefSectionState.Read or XRefSectionState.Relocated) || section.Incomplete || section.CutByLimit || + section.ChainHiddenByGuard is not null) { return false; } diff --git a/src/AdCodicem.Pdf/Validation/Rules/SectionShiftedRule.cs b/src/AdCodicem.Pdf/Validation/Rules/SectionShiftedRule.cs index 7d4e05c5..9909013e 100644 --- a/src/AdCodicem.Pdf/Validation/Rules/SectionShiftedRule.cs +++ b/src/AdCodicem.Pdf/Validation/Rules/SectionShiftedRule.cs @@ -1,5 +1,4 @@ using System.Globalization; -using AdCodicem.Pdf.IO.XRef; namespace AdCodicem.Pdf.Validation.Rules; @@ -26,7 +25,8 @@ public void Check(ValidationContext context) { foreach (var section in context.Structure.Sections) { - if (section.State != XRefSectionState.Relocated) + // Found there and read, or found there and kept from being read by a guard, the section's own fault (#263). + if (!section.WasRelocated) { continue; } diff --git a/src/AdCodicem.Pdf/Validation/Rules/SizeWrongRule.cs b/src/AdCodicem.Pdf/Validation/Rules/SizeWrongRule.cs index cab66f1a..af207118 100644 --- a/src/AdCodicem.Pdf/Validation/Rules/SizeWrongRule.cs +++ b/src/AdCodicem.Pdf/Validation/Rules/SizeWrongRule.cs @@ -135,7 +135,8 @@ private static bool IsWhole(IReadOnlyList sections) { foreach (var section in sections) { - if (section.State is XRefSectionState.NotFound or XRefSectionState.Malformed || section.CutByLimit) + if (section.State is XRefSectionState.NotFound or XRefSectionState.Malformed || section.CutByLimit || + section.ChainHiddenByGuard is not null) { return false; } diff --git a/src/AdCodicem.Pdf/Validation/Rules/TrailerValueWrongRule.cs b/src/AdCodicem.Pdf/Validation/Rules/TrailerValueWrongRule.cs index c63832cb..0505d1c8 100644 --- a/src/AdCodicem.Pdf/Validation/Rules/TrailerValueWrongRule.cs +++ b/src/AdCodicem.Pdf/Validation/Rules/TrailerValueWrongRule.cs @@ -1,5 +1,6 @@ using System.Globalization; using AdCodicem.Pdf.IO; +using AdCodicem.Pdf.IO.Filters; using AdCodicem.Pdf.IO.XRef; using AdCodicem.Pdf.Objects; @@ -30,7 +31,14 @@ namespace AdCodicem.Pdf.Validation.Rules; /// /// Each trailer is judged as the file wrote it; a reference it holds is followed as any read follows it, through the index /// the validator had the reader settle on before its first rule — the chain's own, unless a read could still have rebuilt -/// it, in which case it was rebuilt then (#237) —, so that what was read before validating changes nothing. +/// it, in which case it was rebuilt then (#237) —, so that what was read before validating changes nothing. A value the +/// reader does not read is judged as the file wrote it, a reference it holds not followed: following it would read what +/// the reading never did, and could reach a guard it never met (#263). So are a hybrid file's stream's own /Prev, +/// which the reader does not follow, the chain going on through the table's; the /Prev and /XRefStm of a +/// table an /XRefStm names in a stream's place; the elements of a /W past the third; and what of a +/// cross-reference stream's /Filter and /DecodeParms decoding does not read — the parameters of a filter +/// that reads none, past its predictor where it has none, or past the element that ends the chain —, what decoding reads +/// judged as it read it, whatever the cache let go since. /// /// internal sealed class TrailerValueWrongRule : IValidationRule @@ -72,32 +80,41 @@ public void Check(ValidationContext context) judge.Integers(trailer, PdfName.W); judge.Integers(trailer, PdfName.Index); + // A hybrid file's stream gives a /Prev of its own, which the reader does not follow, the chain going on through + // the table's (7.5.8.4): judged as the file wrote it, a reference it holds not followed (#263). if (IsFollowed(sections, section, PdfName.Prev)) { - judge.Value("its /Prev", trailer.GetRaw(PdfName.Prev), Iso1, "Table 17"); + judge.Value("its /Prev", trailer.GetRaw(PdfName.Prev), Iso1, "Table 17", resolve: section.NamedBy != "/XRefStm"); } judge.Value("its /Length", trailer.GetRaw(PdfName.Length), pdf20 ? Iso2 : null, "Table 5"); - judge.Decoding(trailer, pdf20); + + // What decoding reads, followed as decoding follows it, through the bound on filters the reader read under. + var reads = new DecodingReads(); + FilterChain.TryFollow(trailer, context.Document.Reader.FilterChainBound, ref reads); + judge.Decoding(trailer, pdf20, reads); } else { + // A table an /XRefStm names gives values the reader does not follow either: judged as written (#263). + var followed = section.NamedBy != "/XRefStm"; + if (IsFollowed(sections, section, PdfName.Prev)) { - judge.Value("its /Prev", trailer.GetRaw(PdfName.Prev), "ISO 32000-1 makes it direct (Table 15)", "Table 15"); + judge.Value("its /Prev", trailer.GetRaw(PdfName.Prev), "ISO 32000-1 makes it direct (Table 15)", "Table 15", followed); } if (IsFollowed(sections, section, PdfName.XRefStm)) { - judge.Value("its /XRefStm", trailer.GetRaw(PdfName.XRefStm), direct: null, "Table 19"); + judge.Value("its /XRefStm", trailer.GetRaw(PdfName.XRefStm), direct: null, "Table 19", followed); } } } } /// - /// Determines whether the chain went on through the of : one that - /// names no section is 's to report. + /// Determines whether the of named a section the chain found, or none + /// it looked for: one that names no section is 's to report. /// private static bool IsFollowed(IReadOnlyList sections, XRefSectionRecord section, PdfName key) { @@ -130,10 +147,14 @@ private readonly struct Judge(ValidationContext context, IValidationRule rule, X /// The value as the file wrote it, or null when there is none. /// What makes the value direct, or null when nothing does. /// The table that makes the value an integer, or null when none does. - public void Value(string what, PdfObject? value, string? direct, string? integer) + /// + /// Whether a reference the value is written as is followed to judge what it leads to, which a value the reader does + /// not read is not: following it would read what the reading never did. + /// + public void Value(string what, PdfObject? value, string? direct, string? integer, bool resolve = true) { var reference = direct is not null ? value as PdfReference : null; - var real = integer is not null && value?.Resolve() is PdfReal written && written.AsInteger() is not null; + var real = integer is not null && (resolve ? value?.Resolve() : value) is PdfReal written && written.AsInteger() is not null; if (reference is null && !real) { @@ -172,17 +193,22 @@ public void Integers(PdfDictionary trailer, PdfName key) return; } + // The reader reads three elements of a /W, the widths of three fields, and every element of an /Index. for (var i = 0; i < array.Count; i++) { - Value(string.Create(CultureInfo.InvariantCulture, $"element {i} of its {name}"), array[i], Iso1, "Table 17"); + Value( + string.Create(CultureInfo.InvariantCulture, $"element {i} of its {name}"), array[i], Iso1, "Table 17", key != PdfName.W || i < 3); } } /// /// Reports a cross-reference stream's /Filter or /DecodeParms written as a reference, and, in a file - /// that declares PDF 2.0, each element and entry written as one; and the integers of Table 8 written as reals. + /// that declares PDF 2.0, each element and entry written as one; and the integers of Table 8 a /DecodeParms + /// gives written as reals — not a dictionary a /Filter gives, which is no decode parameters, Table 8 not + /// governing it, and ends decoding unread. A reference among them is followed where decoding follows it, as + /// says. /// - public void Decoding(PdfDictionary trailer, bool pdf20) + public void Decoding(PdfDictionary trailer, bool pdf20, DecodingReads reads) { var inside = pdf20 ? Iso2 : null; @@ -190,9 +216,10 @@ public void Decoding(PdfDictionary trailer, bool pdf20) { var name = FileQuote.Name(key); var value = trailer.GetRaw(key); + var decodeParms = key == PdfName.DecodeParms; Value("its " + name, value, Iso1, integer: null); - switch (value?.Resolve()) + switch (reads.Resolve(value)) { case PdfArray array: for (var i = 0; i < array.Count; i++) @@ -200,27 +227,59 @@ public void Decoding(PdfDictionary trailer, bool pdf20) var what = string.Create(CultureInfo.InvariantCulture, $"element {i} of its {name}"); Value(what, array[i], inside, integer: null); - if (array[i].Resolve() is PdfDictionary parameters) + if (reads.Resolve(array[i]) is PdfDictionary parameters) { - Parameters(parameters, what, inside); + Parameters(parameters, what, inside, reads, decodeParms); } } break; case PdfDictionary parameters: - Parameters(parameters, "its " + name, inside); + Parameters(parameters, "its " + name, inside, reads, decodeParms); break; } } } - private void Parameters(PdfDictionary parameters, string where, string? inside) + private void Parameters(PdfDictionary parameters, string where, string? inside, DecodingReads reads, bool decodeParms) { foreach (var (key, value) in parameters) { - Value(FileQuote.Name(key) + " of " + where, value, inside, Array.IndexOf(IntegerParameters, key) >= 0 ? "Table 8" : null); + var integer = decodeParms && Array.IndexOf(IntegerParameters, key) >= 0 ? "Table 8" : null; + Value(FileQuote.Name(key) + " of " + where, value, inside, integer, reads.Follows(value)); + } + } + } + + /// + /// The references among a cross-reference stream's /Filter and /DecodeParms that decoding follows, each as + /// the file wrote it, which gives in the order decoding reads them, with what it led + /// to (#263). The rule walks what they led to, not what a second read of them gives: the cache may have let an object go + /// since, and its references read again are other instances, which decoding did not follow. + /// + private readonly struct DecodingReads() : IFilterValueReader + { + private readonly Dictionary _followed = new(ReferenceEqualityComparer.Instance); + + /// Determines whether is no reference, or one decoding follows. + public bool Follows(PdfObject? written) => written is not PdfReference || _followed.ContainsKey(written); + + /// Gives what led to where decoding follows it, and itself otherwise. + public PdfObject? Resolve(PdfObject? written) => + written is PdfReference && _followed.TryGetValue(written, out var value) ? value : written; + + /// + public bool TryRead(PdfName key, PdfObject? written, bool entry, out PdfObject? value) + { + value = written?.Resolve(); + + if (written is PdfReference) + { + _followed[written] = value; } + + return true; } } } diff --git a/tests/AdCodicem.Pdf.IntegrationTests/CorpusOptions.cs b/tests/AdCodicem.Pdf.IntegrationTests/CorpusOptions.cs index a3253f01..36f3e2d7 100644 --- a/tests/AdCodicem.Pdf.IntegrationTests/CorpusOptions.cs +++ b/tests/AdCodicem.Pdf.IntegrationTests/CorpusOptions.cs @@ -18,6 +18,7 @@ public static PdfReaderOptions For(CorpusDocument entry) limits = limits with { MaxDecodedStreamLength = raised.MaxDecodedStreamLength ?? limits.MaxDecodedStreamLength, + MaxStreamLength = raised.MaxStreamLength ?? limits.MaxStreamLength, MaxObjectLength = raised.MaxObjectLength ?? limits.MaxObjectLength, MaxXRefSectionLength = raised.MaxXRefSectionLength ?? limits.MaxXRefSectionLength, MaxXRefSectionCount = raised.MaxXRefSectionCount ?? limits.MaxXRefSectionCount, diff --git a/tests/AdCodicem.Pdf.IntegrationTests/StreamLengthRefereeTests.cs b/tests/AdCodicem.Pdf.IntegrationTests/StreamLengthRefereeTests.cs index 8af1ec02..bb4bf40d 100644 --- a/tests/AdCodicem.Pdf.IntegrationTests/StreamLengthRefereeTests.cs +++ b/tests/AdCodicem.Pdf.IntegrationTests/StreamLengthRefereeTests.cs @@ -20,10 +20,8 @@ namespace AdCodicem.Pdf.IntegrationTests; /// qpdf reads the copy of an object its own index places, which is the reader's only when both indexes place the same /// one: a stream qpdf's index lacks — tiff2pdf's scans, whose chain qpdf does not follow —, or places elsewhere — the /// Atypon article's object 49, which the reader reads where its rebuilt index places it —, is another stream, and is not -/// compared. qpdf's search for endstream is not bounded by the next object, as the reader's past its window is; -/// where the two disagree for that reason, the stream is named below. Inside the window the reader does not bound its -/// search either, and the two agree even where both are wrong: PDFBox's zeroed object stream, object 417, takes object -/// 441's endstream, a zeroed stretch having erased its own. +/// compared. qpdf's search for endstream is not bounded by the next object, as the reader's is, in the bytes it +/// holds as past them (#174); where the two disagree for that reason, the stream is named below. /// /// [Collection(RefereeCollection.Name)] @@ -34,10 +32,15 @@ public class StreamLengthRefereeTests(RefereeContainer referee) /// planning report: a block of zeros erased the end of object 695's data, its endstream and the objects after /// it; qpdf takes the first endstream past the block, object 778's, and reads 3,292,376 bytes, where the reader /// stops at object 696, which the index as written places 21 bytes past the declared length, and keeps that length. + /// PDFBox's zeroed object stream: a 32 KB block of zeros erased object 417's endstream and the objects after it; + /// qpdf takes object 441's endstream, past the block, and reads 34,152 bytes, where the reader stops at the next + /// object the index places, inside the block, and keeps the declared 1,302 — it took 441's too until #174 bounded its + /// search within the bytes it holds as past them. /// private static readonly Dictionary KnownDisagreements = new(StringComparer.Ordinal) { ["remote/opf-format-corpus/jhove-hul-35-pdfium-urban-planning-report.pdf"] = [695], + ["remote/pdfbox/distiller6-zeroed-object-stream-pdfbox3947.pdf"] = [417], }; [Theory] diff --git a/tests/AdCodicem.Pdf.TestSupport/Corpus.cs b/tests/AdCodicem.Pdf.TestSupport/Corpus.cs index 5957349f..c1158844 100644 --- a/tests/AdCodicem.Pdf.TestSupport/Corpus.cs +++ b/tests/AdCodicem.Pdf.TestSupport/Corpus.cs @@ -110,6 +110,9 @@ public sealed record CorpusReaderLimits /// What one stream may decode to, in bytes. public int? MaxDecodedStreamLength { get; init; } + /// The most of one stream's encoded data read at once, in bytes. + public int? MaxStreamLength { get; init; } + /// The longest object, the data of a stream aside, in bytes. public int? MaxObjectLength { get; init; } diff --git a/tests/AdCodicem.Pdf.Tests/ChainFiles.cs b/tests/AdCodicem.Pdf.Tests/ChainFiles.cs index 5099e5e9..1ac7a6c8 100644 --- a/tests/AdCodicem.Pdf.Tests/ChainFiles.cs +++ b/tests/AdCodicem.Pdf.Tests/ChainFiles.cs @@ -116,93 +116,1684 @@ public static byte[] UnderAnUpdate( } /// - /// A file of one cross-reference stream, object 5, whose /Length names object 4, which only that stream - /// indexes: valid, since ISO 32000-1 lets a stream's /Length be indirect, and 7.5.8.2 does not except a - /// cross-reference stream's. Object 4 gives the data's length, plus ; with - /// , the rows of objects 6 and 7, of reserved types, spell endstream after - /// an end-of-line. With , a last row gives object 6 free with generation 13, so - /// that the data's last byte is a carriage return. The rows are compressed when , and the - /// data ends with before its endstream. + /// A file of one cross-reference stream, object 5, whose /Length names an object which only that stream + /// indexes — object 4, or with the one its last row places, object 4 then free —: valid, + /// since ISO 32000-1 lets a stream's /Length be indirect, and 7.5.8.2 does not except a cross-reference stream's. + /// The length gives the data's length, plus . After the rows of objects 0 to 5 come + /// free rows; with , a row that gives the next + /// object free with generation 13, so that it ends in a carriage return; then, with , + /// two rows of reserved types that spell endstream after an end-of-line, or with + /// an object header. The rows are compressed when , the + /// data loses its last bytes, and it ends with before its + /// endstream, which its endobj and the startxref follow. The object that holds the length holds + /// spaces before it. /// public static byte[] AloneWithIndirectLength( - int lengthOff = 0, bool spellingEndStream = false, bool compressed = false, bool endingInCarriageReturn = false, string endOfLine = "\n") + int lengthOff = 0, + bool spellingEndStream = false, + bool compressed = false, + bool endingInCarriageReturn = false, + string endOfLine = "\n", + int padding = 0, + bool lengthLast = false, + bool spellingHeader = false, + int dropped = 0, + int lengthPadding = 0) { using var file = new Writer("1.5"); file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); - var size = 6 + (spellingEndStream ? 2 : 0) + (endingInCarriageReturn ? 1 : 0); + var spelled = spellingEndStream ? "\nendstream\0\0\0\0"u8.ToArray() : spellingHeader ? "\n1 0 obj\0\0\0\0\0\0"u8.ToArray() : []; + var size = 6 + padding + (endingInCarriageReturn ? 1 : 0) + (spelled.Length / 7) + (lengthLast ? 1 : 0); + var lengthNumber = lengthLast ? size - 1 : 4; var encoding = compressed ? Rows.Flate : Rows.Raw; - // Uncompressed rows, or stored in a block of their own, take as many bytes whatever they hold. - var length = Encode(new (int, long, int)[size - (spellingEndStream ? 2 : 0)], encoding).Length + (spellingEndStream ? 14 : 0); - file.Object(4, (length + lengthOff).ToString(CultureInfo.InvariantCulture)); + // Uncompressed rows, or stored in blocks of their own, take as many bytes whatever they hold. + var length = Data(0, 0).Length + lengthOff; + file.Object(lengthNumber, new string(' ', lengthPadding) + length.ToString(CultureInfo.InvariantCulture)); var stream = file.Position; - (int Type, long Second, int Third)[] rows = - [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, file.OffsetOf(4), 0), (1, stream, 0)]; - var data = new List(Encode(endingInCarriageReturn ? [.. rows, (0, 0, 13)] : rows, encoding)); + var filter = compressed ? " /Filter /FlateDecode" : string.Empty; + file.Stream( + 5, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size {size} /W [1 4 2]{filter} /Root 1 0 R /Length {lengthNumber} 0 R"), + Data(file.OffsetOf(lengthNumber), stream), + endOfLine); + file.StartXRef(stream); + return file.ToArray(); + + byte[] Data(long lengthAt, long streamAt) + { + var rows = new List<(int Type, long Second, int Third)> + { + (0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), + lengthLast ? (0, 0, 0) : (1, lengthAt, 0), (1, streamAt, 0), + }; + + rows.AddRange(Enumerable.Repeat((0, 0L, 0), padding)); + + if (endingInCarriageReturn) + { + rows.Add((0, 0, 13)); + } + + // Types 10 and 101 or 106, reserved: the rows are ignored, and their bytes read "\nendstream", or "\n1 0 obj". The + // length's row past them is written as they are, uncompressed. + var data = new List(Encode([.. rows], encoding)); + data.AddRange(spelled); + + if (lengthLast) + { + data.AddRange(Encode([(1, lengthAt, 0)], Rows.Raw)); + } + + data.RemoveRange(data.Count - dropped, dropped); + return [.. data]; + } + } + + /// + /// A file of one cross-reference stream, object 5, whose /Length names object 4, which only that stream indexes, + /// in a row before two rows of reserved types that spell endstream after an end-of-line; the page's row comes + /// after them. With , the /Index numbers object 4 twice: first in a row of a + /// reserved type, which the specification says to ignore, then in the row that places it. With , + /// the data is compressed in stored blocks, which show the rows as written, times over, each + /// layer a Flate filter. The file is valid. + /// + public static byte[] AloneWithIndirectLengthBeforeSpelledRows(bool reservedRowFirst, bool stored = false, int layers = 1) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var reserved = reservedRowFirst ? [(3, 0L, 0)] : Array.Empty<(int, long, int)>(); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long four, long stream) + { + var data = new List(Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), .. reserved, (1, four, 0), (1, stream, 0)], Rows.Raw)); + data.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + data.AddRange(Encode([(1, file.OffsetOf(3), 0)], Rows.Raw)); + byte[] bytes = [.. data]; + + for (var layer = 0; stored && layer < layers; layer++) + { + bytes = Compress(bytes); + } + + return bytes; + } + + file.Object(4, Data(0, 0).Length.ToString(CultureInfo.InvariantCulture)); + var stream = file.Position; + var index = reservedRowFirst ? "0 3 4 1 4 2 6 2 3 1" : "0 3 4 2 6 2 3 1"; + var filter = stored ? " /Filter [" + string.Join(' ', Enumerable.Repeat("/FlateDecode", layers)) + "]" : string.Empty; + file.Stream(5, $"/Type /XRef /Size 8 /W [1 4 2] /Index [{index}]{filter} /Root 1 0 R /Length 4 0 R", Data(file.OffsetOf(4), stream)); + file.StartXRef(stream); + return file.ToArray(); + } + + /// + /// A file of one cross-reference stream, object 5, whose rows have no type field — /W [0 4 2], every row of type 1 + /// —, whose /Length names object 4, which only that stream indexes, and whose rows of objects 6 and 7 spell + /// endstream after an end-of-line, before the page's row. The file is valid. + /// + public static byte[] AloneWithIndirectLengthAndNoTypeField() + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(4, "42"); + var stream = file.Position; + var data = new List(); + + foreach (var offset in new[] { file.OffsetOf(1), file.OffsetOf(2), file.OffsetOf(4), stream }) + { + data.AddRange([(byte)(offset >> 24), (byte)(offset >> 16), (byte)(offset >> 8), (byte)offset, 0, 0]); + } + + data.AddRange("\nendstream\0\0"u8.ToArray()); + var page = file.OffsetOf(3); + data.AddRange([(byte)(page >> 24), (byte)(page >> 16), (byte)(page >> 8), (byte)page, 0, 0]); + file.Stream(5, "/Type /XRef /Size 8 /W [0 4 2] /Index [1 2 4 2 6 2 3 1] /Root 1 0 R /Length 4 0 R", [.. data]); + file.StartXRef(stream); + return file.ToArray(); + } + + /// + /// A file of two revisions. The first writes object 8 as (old) and indexes it in cross-reference stream 7. The + /// second writes it again as (new), and indexes it in cross-reference stream 10, whose /Length names + /// object 4, which only that stream indexes, and whose rows of objects 5 and 6, of reserved types, spell + /// endstream after an end-of-line when . With + /// , stream 7's /Length names object 9, which only stream 7 indexes, and a comment + /// sets the update further from it than a search for a section's misplaced offset reaches. Object 4 holds + /// , in which {length} stands for the length. The file is valid. With + /// , the update writes object 9 again, holding it, and stream 10 indexes it there. + /// + public static byte[] UpdateWithIndirectLength( + bool spellingEndStream, + bool olderIndirect = false, + string four = "{length}", + string? nine = null) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(8, "(old)"); + var olderRows = olderIndirect ? 10 : 9; + + if (olderIndirect) + { + file.Object(9, (olderRows * 7).ToString(CultureInfo.InvariantCulture)); + } + + var first = file.Position; + (int Type, long Second, int Third)[] older = + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (1, first, 0), (1, file.OffsetOf(8), 0)]; + var rows = Encode(olderIndirect ? [.. older, (1, file.OffsetOf(9), 0)] : older, Rows.Raw); + var olderLength = olderIndirect ? "9 0 R" : rows.Length.ToString(CultureInfo.InvariantCulture); + file.Stream(7, string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size {olderRows} /W [1 4 2] /Root 1 0 R /Length {olderLength}"), rows); + file.StartXRef(first); + + if (olderIndirect) + { + file.Text("%" + new string('x', 2 * AdCodicem.Pdf.IO.PdfFileReader.NearbySearchRadius) + "\n"); + } + + file.Object(8, "(new)"); + + if (nine is not null) + { + file.Object(9, nine); + } + + var spelled = spellingEndStream ? "\nendstream\0\0\0\0"u8.ToArray() : Encode([(0, 0, 0), (0, 0, 0)], Rows.Raw); + var after = nine is null ? 14 : 21; + file.Object(4, four.Replace("{length}", (7 + spelled.Length + after).ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal)); + var second = file.Position; + var data = new List(Encode([(1, file.OffsetOf(4), 0)], Rows.Raw)); + data.AddRange(spelled); + data.AddRange(Encode(nine is null ? [(1, file.OffsetOf(8), 0), (1, second, 0)] : [(1, file.OffsetOf(8), 0), (1, file.OffsetOf(9), 0), (1, second, 0)], Rows.Raw)); + var index = nine is null ? "4 3 8 1 10 1" : "4 3 8 3"; + file.Stream( + 10, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 11 /W [1 4 2] /Index [{index}] /Root 1 0 R /Prev {first} /Length 4 0 R"), + [.. data]); + file.StartXRef(second); + return file.ToArray(); + } + + /// + /// A file of two revisions. The first writes object 8 as (old) and indexes it in cross-reference stream 7, whose + /// /Length names object 4, which only that stream indexes; its rows of objects 5 and 6, of reserved types, spell + /// endstream after an end-of-line when . The second, set further from it than a + /// search for a section's misplaced offset reaches, writes object 8 again as (new) and indexes it in + /// cross-reference stream 9. The file is valid. + /// + public static byte[] OlderWithIndirectLength(bool spellingEndStream) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(8, "(old)"); + file.Object(4, "63"); + var first = file.Position; + var data = new List(Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, file.OffsetOf(4), 0)], Rows.Raw)); + data.AddRange(spellingEndStream ? "\nendstream\0\0\0\0"u8.ToArray() : Encode([(0, 0, 0), (0, 0, 0)], Rows.Raw)); + data.AddRange(Encode([(1, first, 0), (1, file.OffsetOf(8), 0)], Rows.Raw)); + file.Stream(7, "/Type /XRef /Size 9 /W [1 4 2] /Root 1 0 R /Length 4 0 R", [.. data]); + file.StartXRef(first); + file.Text("%" + new string('x', 2 * AdCodicem.Pdf.IO.PdfFileReader.NearbySearchRadius) + "\n"); + + file.Object(8, "(new)"); + var second = file.Position; + var rows = Encode([(1, file.OffsetOf(8), 0), (1, second, 0)], Rows.Raw); + file.Stream( + 9, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 10 /W [1 4 2] /Index [8 2] /Root 1 0 R /Prev {first} /Length {rows.Length}"), + rows); + file.StartXRef(second); + return file.ToArray(); + } + + /// + /// A file of revisions, each a cross-reference stream whose /Length names an object only + /// that stream indexes, written before it, and whose rows give that object, the stream, and + /// free objects after them; the first indexes the catalog, the page tree and the page as + /// well. The file is valid. + /// + public static byte[] UpdatesWithIndirectLengths(int count, int padding) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var previous = -1L; + + for (var i = 0; i < count; i++) + { + var length = 4 + (i * (padding + 2)); + var rows = new List<(int Type, long Second, int Third)>(); + + if (i == 0) + { + rows.AddRange([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0)]); + } + + file.Object(length, ((rows.Count + padding + 2) * 7).ToString(CultureInfo.InvariantCulture)); + var stream = file.Position; + rows.AddRange([(1, file.OffsetOf(length), 0), (1, stream, 0)]); + rows.AddRange(Enumerable.Repeat((0, 0L, 0), padding)); + var index = i == 0 ? $"0 {length + padding + 2}" : $"{length} {padding + 2}"; + var prev = previous < 0 ? string.Empty : string.Create(CultureInfo.InvariantCulture, $" /Prev {previous}"); + file.Stream( + length + 1, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size {length + padding + 2} /W [1 4 2] /Index [{index}] /Root 1 0 R{prev} /Length {length} 0 R"), + Encode([.. rows], Rows.Raw)); + file.StartXRef(stream); + previous = stream; + } + + return file.ToArray(); + } + + /// + /// A hybrid file: a classic table indexes the catalog, the page tree and object 6, which holds the offset of + /// cross-reference stream 7; its trailer's /XRefStm is , in which {offset} + /// stands for that offset. Only the stream indexes the page, object 3. Object 6's value comes after + /// spaces. + /// + public static byte[] HybridNamingItsStream(string xrefStm, int padding = 0) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The stream's offset is written in object 6 with a fixed width, so that it can be known before the stream is. + var six = file.Position; + var streamAt = six + padding + "6 0 obj\n0000000000\nendobj\n".Length; + file.Object(6, new string(' ', padding) + streamAt.ToString("D10", CultureInfo.InvariantCulture)); + var stream = file.Position; + var data = Encode([(1, file.OffsetOf(3), 0)], Rows.Raw); + file.Stream(7, string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 7 /W [1 4 2] /Index [3 1] /Length {data.Length}"), data); + var xref = file.Position; + var named = xrefStm.Replace("{offset}", stream.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal); + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 3\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n6 1\n{six:D10} 00000 n\r\ntrailer\n<< /Size 7 /Root 1 0 R /XRefStm {named} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of two classic tables: the first indexes the catalog, the page tree and the page; the second, an update, + /// indexes object 4 and object 9, which holds the first table's offset, and names that table through + /// , in which {offset} stands for the offset. Object 9's value comes after + /// spaces; with , it is written as a real with no fractional part. + /// + public static byte[] UpdateNamingItsPrevious(string prev, int padding = 0, bool real = false) + { + using var file = new Writer("1.4"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var first = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 4 /Root 1 0 R >>\nstartxref\n{first}\n%%EOF\n")); + file.Object(4, "(an update)"); + file.Object(9, new string(' ', padding) + first.ToString(CultureInfo.InvariantCulture) + (real ? ".0" : string.Empty)); + var second = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n4 1\n{file.OffsetOf(4):D10} 00000 n\r\n9 1\n{file.OffsetOf(9):D10} 00000 n\r\ntrailer\n<< /Size 10 /Root 1 0 R /Prev {prev.Replace("{offset}", first.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal)} >>\nstartxref\n{second}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of two revisions. The first holds the catalog, the page tree and the page, and cross-reference stream 7, whose + /// /Length names object 5, which holds the data's length after 3,000 spaces; or, when , + /// object 4, which holds 9 0 R, object 9 holding the length so. Its rows of objects 8 and 9 — 10 and 11 when + /// —, of reserved types, spell endstream after an end-of-line, and the page's row + /// comes after them. The second is a classic table that indexes object 5 — 9 when — again, + /// where it is, and names stream 7 through /Prev. The file is valid, unless is added + /// to the length; spaces come before it. With , the rows that + /// spell endstream come first: chained, stream 7's row of object 4 then lies past the data taken to that + /// endstream, and the file is read whole only by rebuilding its index. With , the + /// spaces come after the length instead. + /// + public static byte[] UpdatePlacingAPaddedLength( + bool chained, int padding = 3000, int lengthOff = 0, bool spelledFirst = false, bool lengthFirst = false) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var lengthNumber = chained ? 9 : 5; + var spelled = "\nendstream\0\0\0\0"u8.ToArray(); + var length = (6 * 7 + spelled.Length + lengthOff).ToString(CultureInfo.InvariantCulture); + file.Object(lengthNumber, lengthFirst ? length + new string(' ', padding) : new string(' ', padding) + length); + + if (chained) + { + file.Object(4, "9 0 R"); + } + + var stream = file.Position; + var rows = Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(chained ? 4 : 5), 0), (1, stream, 0)], + Rows.Raw); + var data = new List(spelledFirst ? spelled : rows); + data.AddRange(spelledFirst ? rows : spelled); + data.AddRange(Encode([(1, file.OffsetOf(3), 0)], Rows.Raw)); + var (spelledRows, lengthRow) = chained ? ("10 2", "4 1") : ("8 2", "5 1"); + var index = spelledFirst ? $"[{spelledRows} 0 3 {lengthRow} 7 1 3 1]" : $"[0 3 {lengthRow} 7 1 {spelledRows} 3 1]"; + file.Stream( + 7, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size {(chained ? 12 : 10)} /W [1 4 2] /Index {index} /Root 1 0 R /Length {(chained ? 4 : 5)} 0 R"), + [.. data]); + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n{lengthNumber} 1\n{file.OffsetOf(lengthNumber):D10} 00000 n\r\ntrailer\n<< /Size {(chained ? 12 : 10)} /Root 1 0 R /Prev {stream} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of two revisions. The first holds the catalog, the page tree and the page, object 5, which holds the data's + /// length, 210, after 1,014 spaces — a MaxObjectLength of 1,024 parts it after its 21 —, and cross-reference + /// stream 7, whose /Length names object 5 and whose rows spell endstream after an end-of-line twice, 7 and + /// 21 bytes in, before the rows of objects 1 to 3, 5 and 7 and free ones. With + /// , the rows are compressed in a stored block, which shows them as written 7 bytes in: the + /// data is 284 bytes with 29 free rows, which the same bound parts after its 28, where the second endstream is, + /// and 221 with 20, parted after its 22, where none is. The second is a classic table that indexes object 5 again, where + /// it is, and names stream 7 through /Prev. The file is valid. + /// + public static byte[] UpdatePlacingAPartedLength(int free = 20, bool compressed = false) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long five, long stream) + { + var spelled = "\nendstream\0\0\0\0"u8.ToArray(); + var data = new List(Encode([(0, 0, 65535)], Rows.Raw)); + data.AddRange(spelled); + data.AddRange(spelled); + data.AddRange(Encode([(1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, five, 0), (1, stream, 0)], Rows.Raw)); + data.AddRange(Encode([.. Enumerable.Repeat((0, 0L, 0), free)], Rows.Raw)); + return compressed ? Compress([.. data]) : [.. data]; + } + + file.Object(5, new string(' ', 1014) + Data(0, 0).Length.ToString(CultureInfo.InvariantCulture)); + var stream = file.Position; + var size = 30 + free; + file.Stream( + 7, + string.Create( + CultureInfo.InvariantCulture, + $"/Type /XRef /Size {size} /W [1 4 2] /Index [0 1 20 2 22 2 1 3 5 1 7 1 30 {free}]{(compressed ? " /Filter /FlateDecode" : string.Empty)} /Root 1 0 R /Length 5 0 R"), + Data(file.OffsetOf(5), stream)); + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n5 1\n{file.OffsetOf(5):D10} 00000 n\r\ntrailer\n<< /Size {size} /Root 1 0 R /Prev {stream} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of two revisions. The first holds the catalog, the page tree and the page, object 4, and cross-reference stream + /// 5, whose rows of objects 0 to 5, none spelling endstream, are compressed in a stored block and written in + /// hexadecimal after a stray x, which encodes nothing: /Filter [/ASCIIHexDecode /FlateDecode]. Its + /// /Length names object 4, which holds it and which only the stream indexes — or, unless + /// , is written direct. The second is a classic table that indexes objects 1 to 3 and names + /// stream 5 through /Prev. + /// + public static byte[] UpdateOverAStrayHexByte(bool deferred) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long four, long stream) => Encoding.ASCII.GetBytes( + "x" + Convert.ToHexString(Compress(Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, four, 0), (1, stream, 0)], + Rows.Raw))) + ">"); + + var length = Data(0, 0).Length.ToString(CultureInfo.InvariantCulture); + file.Object(4, length); + var at = file.Position; + file.Stream( + 5, + $"/Type /XRef /Size 6 /W [1 4 2] /Filter [/ASCIIHexDecode /FlateDecode] /Root 1 0 R /Length {(deferred ? "4 0 R" : length)}", + Data(file.OffsetOf(4), at)); + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 6 /Root 1 0 R /Prev {at} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of two revisions, as writes it, but stream 5's hexadecimal rows hold no + /// stray byte. With , the rows and zero bytes after them are + /// compressed, the zero bytes to a few, and written with no >: /Filter [/ASCIIHexDecode /FlateDecode]. Without, + /// the rows are written as they are, /Filter /ASCIIHexDecode, and an end-of-line, endstream and another + /// follow the digits of 14 bytes and one more: the deferred search stops there. + /// + public static byte[] UpdateOverHexRows(bool deferred, bool compressed, int padding = 0) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // Compressed, the data's length depends on the offsets its rows hold: written in four digits, it does not move them. + byte[] Data(long four, long stream) + { + var rows = Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, four, 0), (1, stream, 0)], + Rows.Raw); + + if (compressed) + { + using var output = new MemoryStream(); + + using (var zlib = new ZLibStream(output, CompressionLevel.Optimal, leaveOpen: true)) + { + zlib.Write([.. rows, .. new byte[padding]]); + } + + return Encoding.ASCII.GetBytes(Convert.ToHexString(output.ToArray())); + } + + var hex = Convert.ToHexString(rows); + return Encoding.ASCII.GetBytes(hex[..29] + "\nendstream\n" + hex[29..] + ">"); + } + + var four = file.Position; + var at = four + "4 0 obj\n0000\nendobj\n".Length; + var data = Data(four, at); + var length = data.Length.ToString("D4", CultureInfo.InvariantCulture); + file.Object(4, length); + file.Stream( + 5, + $"/Type /XRef /Size 6 /W [1 4 2] /Filter {(compressed ? "[/ASCIIHexDecode /FlateDecode]" : "/ASCIIHexDecode")} /Root 1 0 R /Length {(deferred ? "4 0 R" : length)}", + data); + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 6 /Root 1 0 R /Prev {at} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of two revisions, as writes it, but stream 5 is + /// /Filter [/RunLengthDecode /FlateDecode]: literal runs of RunLength, of 128 bytes but the last, hold the Flate + /// data, which decodes to its rows and 1,000 zero bytes, and a last one the bytes of an end-of-line, endstream, + /// another end-of-line and XYZ, where a deferred search stops. With , the Flate data + /// holds them in a stored block — 7 bytes of header, then the rows —, so that what a cut of it leaves does not depend on + /// how a platform's zlib compresses. Its /Length names object 4, which only the stream's own row places, unless + /// is false: written direct. The file is valid. + /// + public static byte[] UpdateOverRunsSpellingAnEndStream(bool deferred, bool stored = false) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // Compressed, the data's length depends on the offsets its rows hold: written in four digits, it does not move them. + byte[] Data(long four, long stream) + { + var rows = Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, four, 0), (1, stream, 0)], + Rows.Raw); + using var output = new MemoryStream(); + + using (var zlib = new ZLibStream(output, stored ? CompressionLevel.NoCompression : CompressionLevel.Optimal, leaveOpen: true)) + { + zlib.Write([.. rows, .. new byte[1000]]); + } + + var flate = output.ToArray(); + var data = new List(); + + for (var i = 0; i < flate.Length; i += 128) + { + var count = Math.Min(128, flate.Length - i); + data.Add((byte)(count - 1)); + data.AddRange(flate.AsSpan(i, count).ToArray()); + } + + var spelled = "\nendstream\nXYZ"u8.ToArray(); + data.Add((byte)(spelled.Length - 1)); + data.AddRange(spelled); + data.Add(128); + return [.. data]; + } + + var four = file.Position; + var at = four + "4 0 obj\n0000\nendobj\n".Length; + var data = Data(four, at); + var length = data.Length.ToString("D4", CultureInfo.InvariantCulture); + file.Object(4, length); + file.Stream( + 5, + $"/Type /XRef /Size 6 /W [1 4 2] /Filter [/RunLengthDecode /FlateDecode] /Root 1 0 R /Length {(deferred ? "4 0 R" : length)}", + data); + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 6 /Root 1 0 R /Prev {at} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of one cross-reference stream, object 5, whose /Length names object 4, which only the stream's own row + /// places and which holds the length after 1,100 spaces — a MaxObjectLength of 1,024 cuts it before its value. + /// Its rows, of objects 0 to 2, 4, 5 and 8, then 3 (/Index [0 3 4 2 8 1 3 1]), lose their last 3 bytes: the page's + /// row lacks, the file's fault. With , object 8, which its row places, follows the stream, + /// and the startxref otherwise. + /// + public static byte[] AloneWithACutLengthLackingItsLastRow(bool objectAfter) + { + // Every row and offset is as wide whatever it holds: where object 8 lies is read once, then written. + _ = Build(0, out var eight); + return Build(eight, out _); + + byte[] Build(long eightAt, out long eight) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var four = file.Position; + var at = four + ("4 0 obj\n" + new string(' ', 1100) + "46\nendobj\n").Length; + var rows = Encode( + [ + (0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, four, 0), (1, at, 0), (1, eightAt, 0), + (1, file.OffsetOf(3), 0), + ], + Rows.Raw); + file.Object(4, new string(' ', 1100) + "46"); + file.Stream(5, "/Type /XRef /Size 9 /W [1 4 2] /Index [0 3 4 2 8 1 3 1] /Root 1 0 R /Length 4 0 R", rows[..^3]); + eight = file.Position; + + if (objectAfter) + { + file.Object(8, "(eight)"); + } + + file.StartXRef(at); + return file.ToArray(); + } + } + + /// + /// A file as writes it, object 4 cut before its value and the page's + /// row lacking, but for objects 8, 9 and 10, which follow the stream in that order and which only its rows place, in the + /// order its /Index gives — 9, 8, 10 —: the nearest has the second of those rows. Without + /// , the rows have none — /W [0 4 2], every row of type 1 — and object 0 has no row. + /// + public static byte[] AloneWithACutLengthBeforeObjectsItsRowsPlace(bool typeField) + { + // Every row and offset is as wide whatever it holds: where objects 8 to 10 lie is read once, then written. + _ = Build([0, 0, 0], out var after); + return Build(after, out _); + + byte[] Build(long[] placed, out long[] written) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var four = file.Position; + var at = four + ("4 0 obj\n" + new string(' ', 1100) + "00\nendobj\n").Length; + (int Type, long Second, int Third)[] rows = + [ + (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, four, 0), (1, at, 0), (1, placed[1], 0), (1, placed[0], 0), + (1, placed[2], 0), (1, file.OffsetOf(3), 0), + ]; + var data = (typeField ? Encode([(0, 0, 65535), .. rows], Rows.Raw) : Untyped(rows))[..^3]; + file.Object(4, new string(' ', 1100) + data.Length.ToString("D2", CultureInfo.InvariantCulture)); + file.Stream( + 5, + $"/Type /XRef /Size 11 /W [{(typeField ? "1" : "0")} 4 2] /Index [{(typeField ? "0 3" : "1 2")} 4 2 9 1 8 1 10 1 3 1] /Root 1 0 R /Length 4 0 R", + data); + written = [file.Position, 0, 0]; + file.Object(8, "(eight)"); + written[1] = file.Position; + file.Object(9, "(nine)"); + written[2] = file.Position; + file.Object(10, "(ten)"); + file.StartXRef(at); + return file.ToArray(); + } + + static byte[] Untyped((int Type, long Second, int Third)[] rows) + { + var bytes = new List(); + + foreach (var (_, second, third) in rows) + { + bytes.AddRange([(byte)(second >> 24), (byte)(second >> 16), (byte)(second >> 8), (byte)second, (byte)(third >> 8), (byte)third]); + } + + return [.. bytes]; + } + } + + /// + /// A file of one cross-reference stream, object 7, in ASCII85, whose /Length names object 5, which only the + /// stream's own row places, holding the length after 1,015 spaces: a MaxObjectLength of 1,024 parts it after its + /// first digit. Its rows are those of objects 0, 5, 1 to 3, 20 and 21, of reserved types, and 7 (/Index). The text + /// spells an end-of-line and endstream four digits into the group that holds the page's row's last three bytes and + /// row 20's type, which is chosen so that the group's fifth digit is the e of endstream: one that a + /// u in its place carries into the page's row's generation where , and that it does not + /// otherwise. The file is valid. + /// + public static byte[] Ascii85RowsPartedBySpelledEndStream(bool carry) + { + for (var pad = 0; ; pad++) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Text("%" + new string('-', pad) + "\n"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var low = (int)(file.OffsetOf(3) & 0xFF); + + // The group [low 00 00 type] sums to low + type modulo 85, 2^24 being 1 there: its fifth digit is "e", 68, for one + // type in each range. A "u" in its place adds 16 to the group, which carries into the generation past 239. + var type = -1; + + for (var candidate = carry ? 240 : 200; candidate <= (carry ? 255 : 239) && type < 0; candidate++) + { + type = (low + candidate) % 85 == 68 ? candidate : -1; + } + + if (type < 0) + { + continue; + } + + // The digits "ndstr" and "eam!!" that follow the group are rows 20 and 21's bytes. + string Text(long five, long stream) => + Ascii85(Encode([(0, 0, 65535), (1, five, 0), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0)], Rows.Raw)) + + Ascii85([1, (byte)(file.OffsetOf(3) >> 24), (byte)(file.OffsetOf(3) >> 16), (byte)(file.OffsetOf(3) >> 8)]) + + Ascii85([(byte)low, 0, 0, (byte)type])[..4] + "\nendstr" + "eam\n!!" + Ascii85([.. new byte[5], .. Encode([(1, stream, 0)], Rows.Raw)]) + "~>"; + + file.Object(5, new string(' ', 1015) + Text(0, 0).Length.ToString(CultureInfo.InvariantCulture)); + var at = file.Position; + file.Stream( + 7, + "/Type /XRef /Size 22 /W [1 4 2] /Index [0 1 5 1 1 3 20 2 7 1] /Filter /ASCII85Decode /Root 1 0 R /Length 5 0 R", + Encoding.ASCII.GetBytes(Text(file.OffsetOf(5), at))); + file.StartXRef(at); + return file.ToArray(); + } + } + + /// + /// A file of two revisions. The first holds the catalog, the page tree, the page, object 5, which holds the length of + /// cross-reference stream 7 after 1,015 spaces — a MaxObjectLength of 1,024 parts it after its first digit —, and + /// stream 7, Flate in a stored block with a PNG predictor of columns, whose rows are the + /// catalog's, two of reserved types that spell endstream after an end-of-line, then the page tree's, the page's, + /// object 5's and its own (/Index [1 1 20 2 2 2 5 1 7 1]); with , the free row 0's, + /// the catalog's, the page tree's and the page's come first, then the two that spell endstream, then object 5's + /// and its own (/Index [0 4 20 2 5 1 7 1]). The second is a classic table that places object 5 and names stream 7 + /// through /Prev. The file is valid. + /// + public static byte[] PredictedRowsPartedBySpelledEndStream(int columns, bool pageBefore = false) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long five, long stream) + { + var rows = new List(pageBefore + ? Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0)], Rows.Raw) + : Encode([(1, file.OffsetOf(1), 0)], Rows.Raw)); + rows.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + rows.AddRange(pageBefore + ? Encode([(1, five, 0), (1, stream, 0)], Rows.Raw) + : Encode([(1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, five, 0), (1, stream, 0)], Rows.Raw)); + + while (rows.Count % columns != 0) + { + rows.Add(0); + } + + var predicted = new List(); + + for (var i = 0; i < rows.Count; i += columns) + { + predicted.Add(0); + predicted.AddRange(rows.GetRange(i, columns)); + } + + return Compress([.. predicted]); + } + + file.Object(5, new string(' ', 1015) + Data(0, 0).Length.ToString(CultureInfo.InvariantCulture)); + var at = file.Position; + file.Stream( + 7, + string.Create( + CultureInfo.InvariantCulture, + $"/Type /XRef /Size 22 /W [1 4 2] /Index {(pageBefore ? "[0 4 20 2 5 1 7 1]" : "[1 1 20 2 2 2 5 1 7 1]")} /Filter /FlateDecode /DecodeParms << /Predictor 12 /Columns {columns} >> /Root 1 0 R /Length 5 0 R"), + Data(file.OffsetOf(5), at)); + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n5 1\n{file.OffsetOf(5):D10} 00000 n\r\ntrailer\n<< /Size 22 /Root 1 0 R /Prev {at} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of one revision: the catalog, the page tree, the page, object 5, which holds the length of cross-reference + /// stream 7's data, and stream 7, Flate in a stored block with a PNG predictor of columns, + /// whose /Length names object 5, which only its own rows place: the free row 0's, the catalog's, the page tree's + /// and the page's, then object 5's, two of reserved types that spell endstream after an end-of-line, and its own + /// (/Index [0 4 5 1 19 2 7 1]). With 28 columns, the spelled endstream parts the second predicted row + /// right after object 5's row; with 49 or 56, the first, which object 5's row lies in. With 2, + /// the TIFF predictor differences each row in place of the PNG one. The file is valid. + /// + public static byte[] DeferredLengthRowPartedBySpelledEndStream(int columns, int predictor = 12) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long five, long stream) + { + var rows = new List(Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, five, 0)], + Rows.Raw)); + rows.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + rows.AddRange(Encode([(1, stream, 0)], Rows.Raw)); + + while (rows.Count % columns != 0) + { + rows.Add(0); + } + + var predicted = new List(); + + if (predictor == 2) + { + // Each byte but a row's first is written as its difference from the one before it: the two rows of reserved + // types hold the bytes whose differences spell the endstream, their types still past 2, which no reading + // takes. + var spelled = "\nendstream\0\0\0\0"u8; + + for (var i = 35; i < 49; i++) + { + rows[i] = (byte)((i % columns == 0 ? 0 : rows[i - 1]) + spelled[i - 35]); + } + + for (var i = 0; i < rows.Count; i++) + { + predicted.Add(i % columns == 0 ? rows[i] : (byte)(rows[i] - rows[i - 1])); + } + + return Compress([.. predicted]); + } + + for (var i = 0; i < rows.Count; i += columns) + { + predicted.Add(0); + predicted.AddRange(rows.GetRange(i, columns)); + } + + return Compress([.. predicted]); + } + + file.Object(5, Data(0, 0).Length.ToString(CultureInfo.InvariantCulture)); + var at = file.Position; + file.Stream( + 7, + string.Create( + CultureInfo.InvariantCulture, + $"/Type /XRef /Size 21 /W [1 4 2] /Index [0 4 5 1 19 2 7 1] /Filter /FlateDecode /DecodeParms << /Predictor {predictor} /Columns {columns} >> /Root 1 0 R /Length 5 0 R"), + Data(file.OffsetOf(5), at)); + file.StartXRef(at); + return file.ToArray(); + } + + /// + /// A file of one revision: the catalog, the page tree, the page, object 5, which holds the length of cross-reference + /// stream 7's data, and stream 7, raw, whose /Length names object 5, which only its own rows place. Its data + /// starts with zero bytes, fewer than a row's seven, then spells endstream after an + /// end-of-line, before its rows: the free row 0's, the catalog's, the page tree's, the page's, object 5's and its own + /// (/Index [19 2 0 4 5 1 7 1]). The file is valid. + /// + public static byte[] DeferredLengthEndStreamSpelledBeforeRows(int prefix) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long five, long stream) + { + var rows = new List(new byte[prefix]); + rows.AddRange("\nendstream\0"u8.ToArray()); + + while (rows.Count % 7 != 0) + { + rows.Add(0); + } + + rows.AddRange(Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, five, 0), (1, stream, 0)], + Rows.Raw)); + return [.. rows]; + } + + file.Object(5, Data(0, 0).Length.ToString(CultureInfo.InvariantCulture)); + var at = file.Position; + file.Stream(7, "/Type /XRef /Size 21 /W [1 4 2] /Index [19 2 0 4 5 1 7 1] /Root 1 0 R /Length 5 0 R", Data(file.OffsetOf(5), at)); + file.StartXRef(at); + return file.ToArray(); + } + + /// + /// A file of one revision: the catalog, the page tree, the page, object 5 and cross-reference stream 7, whose + /// /Length names object 5 — or, with , gives the length —, and whose data a RunLength + /// step decodes after a Flate one, in a stored block with a PNG predictor of 45 columns. The RunLength data: a literal run + /// of the rows of the free row 0, the catalog, the page tree and the page, 14 runs of 128 zeros — 256 free rows —, then + /// object 5's row and two of reserved types that spell endstream after an end-of-line — with + /// ; past them otherwise —, the stream's own row, and the end byte. The spelled + /// endstream parts the second predicted row, past the 1,052 bytes the data up to it decodes to, and object 5's row + /// lies 1,820 bytes or more into the rows. The file is valid. + /// + public static byte[] DeferredLengthRowPastRunLengthRuns(bool rowBefore, bool direct = false) + { + const int Columns = 45; + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long five, long stream) + { + var runs = new List { 27 }; + runs.AddRange(Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0)], Rows.Raw)); + + for (var i = 0; i < 14; i++) + { + runs.AddRange([0x81, 0x00]); + } + + if (rowBefore) + { + runs.Add(6); + runs.AddRange(Encode([(1, five, 0)], Rows.Raw)); + } + + runs.Add(13); + runs.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + runs.Add(rowBefore ? (byte)6 : (byte)13); + runs.AddRange(Encode(rowBefore ? [(1, stream, 0)] : [(1, five, 0), (1, stream, 0)], Rows.Raw)); + return PredictedRunLength(runs, Columns); + } + + var length = Data(0, 0).Length.ToString(CultureInfo.InvariantCulture); + file.Object(5, direct ? "0" : length); + var at = file.Position; + file.Stream( + 7, + string.Create( + CultureInfo.InvariantCulture, + $"/Type /XRef /Size 266 /W [1 4 2] /Index {(rowBefore ? "[0 4 8 256 5 1 264 2 7 1]" : "[0 4 8 256 264 2 5 1 7 1]")} /Filter [/FlateDecode /RunLengthDecode] /DecodeParms [<< /Predictor 12 /Columns {Columns} >> null] /Root 1 0 R /Length {(direct ? length : "5 0 R")}"), + Data(file.OffsetOf(5), at)); + file.StartXRef(at); + return file.ToArray(); + } + + /// + /// As , the rows past the endstream they spell, but stream 7 numbers no + /// row for object 5, which no section places (/Index [0 4 8 256 264 2 7 1]): its /Length can be read once + /// the chain is, and the rows past the endstream are the file's to lack, the data up to it decoding to 1,052 bytes. + /// + public static byte[] DeferredLengthNoRowNumbersPastRunLengthRuns() + { + const int Columns = 45; + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long stream) + { + var runs = new List { 27 }; + runs.AddRange(Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0)], Rows.Raw)); + + for (var i = 0; i < 14; i++) + { + runs.AddRange([0x81, 0x00]); + } + + runs.Add(13); + runs.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + runs.Add(6); + runs.AddRange(Encode([(1, stream, 0)], Rows.Raw)); + return PredictedRunLength(runs, Columns); + } + + file.Object(5, Data(0).Length.ToString(CultureInfo.InvariantCulture)); + var at = file.Position; + file.Stream( + 7, + string.Create( + CultureInfo.InvariantCulture, + $"/Type /XRef /Size 266 /W [1 4 2] /Index [0 4 8 256 264 2 7 1] /Filter [/FlateDecode /RunLengthDecode] /DecodeParms [<< /Predictor 12 /Columns {Columns} >> null] /Root 1 0 R /Length 5 0 R"), + Data(at)); + file.StartXRef(at); + return file.ToArray(); + } + + /// + /// A file of one revision: the catalog, the page tree, object 5, the page — object 155 — and cross-reference stream 7, + /// whose /Length names object 5 — or, with , gives the length —, and whose data a + /// RunLength step decodes after a Flate one, in a stored block with a PNG predictor of 45 columns. The first predicted + /// row decodes to the rows of objects 0 to 3 — 3 free — and 1,024 zeros; the second, which the endstream its rows spell + /// parts, to 5 zeros, the page's row, the 152nd, 1,057 bytes in, 630 zeros, then object 5's row, two of reserved types + /// that spell endstream after an end-of-line, and the stream's own row + /// (/Index [0 4 8 238 5 1 246 2 7 1]). The file is valid. + /// + public static byte[] DeferredLengthPageRowInPartedRunLengthRow(bool direct = false) + { + const int Columns = 45; + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [155 0 R] /Count 1 >>"); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long five, long page, long stream) + { + var runs = new List { 27 }; + runs.AddRange(Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (0, 0, 0)], Rows.Raw)); + + for (var i = 0; i < 8; i++) + { + runs.AddRange([0x81, 0x00]); + } + + runs.AddRange([0xFC, 0x00, 6]); + runs.AddRange(Encode([(1, page, 0)], Rows.Raw)); + + for (var i = 0; i < 5; i++) + { + runs.AddRange([0x83, 0x00]); + } + + runs.Add(6); + runs.AddRange(Encode([(1, five, 0)], Rows.Raw)); + runs.Add(13); + runs.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + runs.Add(6); + runs.AddRange(Encode([(1, stream, 0)], Rows.Raw)); + return PredictedRunLength(runs, Columns); + } + + var length = Data(0, 0, 0).Length.ToString(CultureInfo.InvariantCulture); + file.Object(5, direct ? "0" : length); + file.Object(155, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var at = file.Position; + file.Stream( + 7, + string.Create( + CultureInfo.InvariantCulture, + $"/Type /XRef /Size 248 /W [1 4 2] /Index [0 4 8 238 5 1 246 2 7 1] /Filter [/FlateDecode /RunLengthDecode] /DecodeParms [<< /Predictor 12 /Columns {Columns} >> null] /Root 1 0 R /Length {(direct ? length : "5 0 R")}"), + Data(file.OffsetOf(5), file.OffsetOf(155), at)); + file.StartXRef(at); + return file.ToArray(); + } - if (spellingEndStream) + /// + /// Pads RunLength data with no-op bytes to whole rows of bytes, gives each row the PNG + /// predictor's None tag, and stores the rows in a zlib stream. + /// + private static byte[] PredictedRunLength(List runs, int columns) + { + while (runs.Count % columns != 0) { - // Types 10 and 101, reserved: the rows are ignored, and their bytes read "\nendstream". - data.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + runs.Add(128); } - var filter = compressed ? " /Filter /FlateDecode" : string.Empty; - file.Stream(5, string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size {size} /W [1 4 2]{filter} /Root 1 0 R /Length 4 0 R"), [.. data], endOfLine); - file.StartXRef(stream); + var predicted = new List(); + + for (var i = 0; i < runs.Count; i += columns) + { + predicted.Add(0); + predicted.AddRange(runs.GetRange(i, columns)); + } + + return Compress([.. predicted]); + } + + /// + /// A file of one cross-reference stream, object 7, whose /Length names object 5, which only the stream's own row + /// places, before the rows of objects 1 and 2; two rows of reserved types spell endstream after an end-of-line + /// after them, 28 and 42 bytes in, before the rows of objects 3 and 7 and fifty free ones, 420 bytes in all. Object 5 + /// holds the length after 1,014 spaces — a MaxObjectLength of 1,024 parts it after its 42 —, or, with + /// , 6 0 R and 1,100 spaces after it — the same bound cuts it after the whole + /// reference —, object 6 holding the length, which a newer classic table indexes and names the stream through + /// /Prev from. The file is valid, unless : the stream's /Index declares one + /// row more than its data holds. + /// + public static byte[] OwnRowPlacingAPaddedLength(bool throughReference, bool lackingRow = false) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + const int Free = 50; + var spelled = "\nendstream\0\0\0\0"u8.ToArray(); + + // The data's length does not depend on the offsets its rows hold. + byte[] Data(long five, long stream) + { + var data = new List(Encode([(0, 0, 65535), (1, five, 0), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0)], Rows.Raw)); + data.AddRange(spelled); + data.AddRange(spelled); + data.AddRange(Encode([(1, file.OffsetOf(3), 0), (1, stream, 0), .. Enumerable.Repeat((0, 0L, 0), Free)], Rows.Raw)); + return [.. data]; + } + + var length = Data(0, 0).Length.ToString(CultureInfo.InvariantCulture); + + if (throughReference) + { + file.Object(6, length); + } + + file.Object(5, throughReference ? "6 0 R" + new string(' ', 1100) : new string(' ', 1014) + length); + var stream = file.Position; + var size = 30 + Free + (lackingRow ? 1 : 0); + file.Stream( + 7, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size {size} /W [1 4 2] /Index [0 1 5 1 1 2 20 2 22 2 3 1 7 1 30 {size - 30}] /Root 1 0 R /Length 5 0 R"), + Data(file.OffsetOf(5), stream)); + + if (!throughReference) + { + file.StartXRef(stream); + return file.ToArray(); + } + + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n6 1\n{file.OffsetOf(6):D10} 00000 n\r\ntrailer\n<< /Size {size} /Root 1 0 R /Prev {stream} >>\nstartxref\n{xref}\n%%EOF\n")); return file.ToArray(); } /// - /// A hybrid file: a classic table indexes the catalog, the page tree and object 6, which holds the offset of - /// cross-reference stream 7; its trailer's /XRefStm is , in which {offset} - /// stands for that offset. Only the stream indexes the page, object 3. + /// A file of three revisions. The first holds the catalog, the page tree and the page, object 5, which holds + /// 6 0 R and 1,100 spaces after it — a MaxObjectLength of 1,024 cuts it after the whole reference —, + /// object 6, which holds 49, and cross-reference stream 7, whose /Length names object 5 and whose rows of objects + /// 0 to 2 and 7 come before two rows of reserved types that spell endstream after an end-of-line, before the + /// page's row: 49 bytes. The second is cross-reference stream 20, of 49 bytes too, which indexes objects 6 and 20 to 25, + /// and names stream 7 through /Prev; its /Length is 49, or, with , + /// 5 0 R, which the chain then reads before any section places object 6. The third is a classic table that + /// indexes object 5 and names stream 20 through /Prev. The file is valid, unless : + /// stream 7's /Index declares one row more than its data holds. /// - public static byte[] HybridNamingItsStream(string xrefStm) + public static byte[] UpdatesReachingALengthThroughACutReference(bool bothNameIt, bool lackingRow = false) { using var file = new Writer("1.5"); file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(5, "6 0 R" + new string(' ', 1100)); + file.Object(6, "49"); + var older = file.Position; + var data = new List(Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, older, 0)], Rows.Raw)); + data.AddRange("\nendstream\0\0\0\0"u8.ToArray()); + data.AddRange(Encode([(1, file.OffsetOf(3), 0)], Rows.Raw)); + file.Stream( + 7, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size {(lackingRow ? 11 : 10)} /W [1 4 2] /Index [0 3 7 1 8 2 3 {(lackingRow ? 2 : 1)}] /Root 1 0 R /Length 5 0 R"), + [.. data]); + var newer = file.Position; + var rows = Encode([(1, file.OffsetOf(6), 0), (1, newer, 0), .. Enumerable.Repeat((0, 0L, 0), 5)], Rows.Raw); + file.Stream( + 20, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 26 /W [1 4 2] /Index [6 1 20 6] /Root 1 0 R /Prev {older} /Length {(bothNameIt ? "5 0 R" : "49")}"), + rows); + file.Text("%" + new string('-', 1200) + "\n"); + var xref = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n5 1\n{file.OffsetOf(5):D10} 00000 n\r\ntrailer\n<< /Size 26 /Root 1 0 R /Prev {newer} >>\nstartxref\n{xref}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file as writes it, the rows that spell endstream numbering objects 8 + /// and 9, but for a length a chain of objects gives: stream 7's /Length names object 5 + /// — or, with , object 4, which only stream 7's own row places —, which holds 100 0 R; + /// object 100 names object 101, and so on to the last, which holds the data's length after 3,000 spaces. The newer table + /// indexes object 5, unless , and objects 100 to the last, where they are. The file is valid. + /// + public static byte[] UpdatePlacingAPaddedLengthChain(int objects, bool ownRow) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var lengthNumber = ownRow ? 4 : 5; + var last = 100 + objects - 2; + var spelled = "\nendstream\0\0\0\0"u8.ToArray(); + file.Object(lengthNumber, "100 0 R"); + + for (var number = 100; number <= last; number++) + { + file.Object( + number, + number < last + ? string.Create(CultureInfo.InvariantCulture, $"{number + 1} 0 R") + : new string(' ', 3000) + (6 * 7 + spelled.Length).ToString(CultureInfo.InvariantCulture)); + } - // The stream's offset is written in object 6 with a fixed width, so that it can be known before the stream is. - var six = file.Position; - var streamAt = six + "6 0 obj\n0000000000\nendobj\n".Length; - file.Object(6, streamAt.ToString("D10", CultureInfo.InvariantCulture)); var stream = file.Position; - var data = Encode([(1, file.OffsetOf(3), 0)], Rows.Raw); - file.Stream(7, string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 7 /W [1 4 2] /Index [3 1] /Length {data.Length}"), data); + var data = new List(Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(lengthNumber), 0), (1, stream, 0)], + Rows.Raw)); + data.AddRange(spelled); + data.AddRange(Encode([(1, file.OffsetOf(3), 0)], Rows.Raw)); + file.Stream( + 7, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 10 /W [1 4 2] /Index [0 3 {lengthNumber} 1 7 1 8 2 3 1] /Root 1 0 R /Length {lengthNumber} 0 R"), + [.. data]); + file.Text("%" + new string('-', 1200) + "\n"); var xref = file.Position; - var named = xrefStm.Replace("{offset}", stream.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal); + var rows = new StringBuilder("xref\n0 1\n0000000000 65535 f\r\n"); + + if (!ownRow) + { + rows.Append(CultureInfo.InvariantCulture, $"5 1\n{file.OffsetOf(5):D10} 00000 n\r\n"); + } + + rows.Append(CultureInfo.InvariantCulture, $"100 {objects - 1}\n"); + + for (var number = 100; number <= last; number++) + { + rows.Append(CultureInfo.InvariantCulture, $"{file.OffsetOf(number):D10} 00000 n\r\n"); + } + + rows.Append(CultureInfo.InvariantCulture, $"trailer\n<< /Size {last + 1} /Root 1 0 R /Prev {stream} >>\nstartxref\n{xref}\n%%EOF\n"); + file.Text(rows.ToString()); + return file.ToArray(); + } + + /// + /// A hybrid update over a classic table that indexes the catalog, the page tree and the page. The update writes object + /// 9, which holds after spaces, and object 10, which cross-reference + /// stream 7 indexes; the stream's own /Prev is , which the reader does not follow. + /// The update's table indexes object 9, and its trailer names the older table through /Prev and the stream + /// through /XRefStm. + /// + public static byte[] HybridWithStreamPrev(string streamPrev, int padding = 3000, string nine = "123") + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var older = file.Position; file.Text(string.Create( CultureInfo.InvariantCulture, - $"xref\n0 3\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n6 1\n{six:D10} 00000 n\r\ntrailer\n<< /Size 7 /Root 1 0 R /XRefStm {named} >>\nstartxref\n{xref}\n%%EOF\n")); + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 4 /Root 1 0 R >>\nstartxref\n{older}\n%%EOF\n")); + file.Object(9, new string(' ', padding) + nine); + file.Object(10, "(ten)"); + var stream = file.Position; + var data = Encode([(1, file.OffsetOf(10), 0)], Rows.Raw); + file.Stream( + 7, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 11 /W [1 4 2] /Index [10 1] /Prev {streamPrev} /Length {data.Length}"), + data); + var table = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n9 1\n{file.OffsetOf(9):D10} 00000 n\r\ntrailer\n<< /Size 11 /Root 1 0 R /Prev {older} /XRefStm {stream} >>\nstartxref\n{table}\n%%EOF\n")); return file.ToArray(); } /// - /// A file of two classic tables: the first indexes the catalog, the page tree and the page; the second, an update, - /// indexes object 4 and object 9, which holds the first table's offset, and names that table through - /// , in which {offset} stands for the offset. + /// A file as writes it, but for what the update's table names through + /// /XRefStm: a classic table, where ISO 32000-1 names a cross-reference stream (Table 19), which indexes object + /// 10 and whose trailer gives as . + /// + public static byte[] HybridNamingATable(string key, string value, int padding = 3000) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var older = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 4 /Root 1 0 R >>\nstartxref\n{older}\n%%EOF\n")); + file.Object(9, new string(' ', padding) + "123"); + file.Object(10, "(ten)"); + var named = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n10 1\n{file.OffsetOf(10):D10} 00000 n\r\ntrailer\n<< /Size 11 /{key} {value} >>\n")); + var table = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n9 1\n{file.OffsetOf(9):D10} 00000 n\r\ntrailer\n<< /Size 11 /Root 1 0 R /Prev {older} /XRefStm {named} >>\nstartxref\n{table}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A classic table updated by a cross-reference stream. The table indexes the catalog, the page tree, the page and + /// object 12, the highest number; cross-reference stream 10, the update, indexes the first three with object 4, a page + /// the tree does not list, and object 9, which holds the table's offset after spaces — + /// written as a real with no fractional part with —, and names the table through + /// /Prev 9 0 R. The table and the stream both give /Size 13. + /// + public static byte[] StreamUpdateHidingItsHighestNumber(int padding, bool real = false) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(12, "(twelve)"); + var first = file.Position; + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\n12 1\n{file.OffsetOf(12):D10} 00000 n\r\ntrailer\n<< /Size 13 /Root 1 0 R >>\nstartxref\n{first}\n%%EOF\n")); + file.Object(4, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(9, new string(' ', padding) + first.ToString(CultureInfo.InvariantCulture) + (real ? ".0" : string.Empty)); + var stream = file.Position; + var data = Encode( + [(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, file.OffsetOf(4), 0), (1, file.OffsetOf(9), 0), (1, stream, 0)], + Rows.Raw); + file.Stream( + 10, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 13 /W [1 4 2] /Index [0 5 9 2] /Root 1 0 R /Prev 9 0 R /Length {data.Length}"), + data); + file.StartXRef(stream); + return file.ToArray(); + } + + /// + /// A file of two revisions. The first indexes the catalog, the page tree and the page, in a classic table or, with + /// , in cross-reference stream 4. The second writes object 5, 9 0 R followed by + /// 1,100 spaces, and object 9, 10, and indexes them in cross-reference stream 7, the newest section, whose + /// /Size names object 5 beside an /Index the rows are numbered by. A reader that reads object 5 whole finds + /// a valid file but for that indirect /Size. + /// + public static byte[] StreamSizedThroughAReferenceOverAnOlderSection(bool olderStream) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var first = file.Position; + + if (olderStream) + { + var rows = Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), (1, first, 0)], Rows.Raw); + file.Stream(4, string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 5 /W [1 4 2] /Root 1 0 R /Length {rows.Length}"), rows); + } + else + { + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 4 /Root 1 0 R >>\n")); + } + + file.StartXRef(first); + file.Object(5, "9 0 R" + new string(' ', 1100)); + file.Object(9, "10"); + var stream = file.Position; + var data = Encode([(1, file.OffsetOf(5), 0), (1, stream, 0), (1, file.OffsetOf(9), 0)], Rows.Raw); + file.Stream( + 7, + string.Create(CultureInfo.InvariantCulture, $"/Type /XRef /Size 5 0 R /Index [5 1 7 1 9 1] /W [1 4 2] /Root 1 0 R /Prev {first} /Length {data.Length}"), + data); + file.StartXRef(stream); + return file.ToArray(); + } + + /// + /// A file of one revision, whose cross-reference stream 5 indexes objects 0 to 39, the page object 39 last, under + /// /Filter [/ASCIIHexDecode /FlateDecode] and a /Length that names object 4, which only stream 5 indexes. Its + /// Flate data is rows 0 to 38 deflated, then a stored block that holds the page's row, with no checksum; its hex text + /// lacks its last digit, a 0 the page's row ends in, and its >. The rows decode to 280 bytes. Read whole, the + /// file is valid but for the hex text's end and the checksum. + /// + public static byte[] HexFlateRowsLackingTheirLastDigit() + { + using var output = new MemoryStream(); + var offsets = new Dictionary(); + + void Text(string text) => output.Write(Encoding.Latin1.GetBytes(text)); + + void Object(int number, string body) + { + offsets[number] = output.Position; + Text(string.Create(CultureInfo.InvariantCulture, $"{number} 0 obj\n{body}\nendobj\n")); + } + + static byte[] Row(int type, long second, int third) => + [(byte)type, (byte)(second >> 24), (byte)(second >> 16), (byte)(second >> 8), (byte)second, (byte)(third >> 8), (byte)third]; + + Text("%PDF-1.5\n"); + Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + Object(2, "<< /Type /Pages /Kids [39 0 R] /Count 1 >>"); + Object(39, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var four = output.Position; + var five = four + "4 0 obj\n000000\nendobj\n".Length; + var head = new List(); + + for (var i = 0; i < 39; i++) + { + head.AddRange(i switch + { + 0 => Row(0, 0, 65535), + 1 => Row(1, offsets[1], 0), + 2 => Row(1, offsets[2], 0), + 4 => Row(1, four, 0), + 5 => Row(1, five, 0), + _ => Row(0, 0, 0), + }); + } + + using var deflated = new MemoryStream(); + using var deflate = new DeflateStream(deflated, CompressionLevel.SmallestSize, leaveOpen: true); + deflate.Write([.. head]); + + // Flushed, not closed: the deflate data ends synced, with no final block, and the stored block after it is the last. + deflate.Flush(); + var zlib = new List { 0x78, 0xDA }; + zlib.AddRange(deflated.ToArray()); + var tail = Row(1, offsets[39], 0); + zlib.AddRange([0x01, (byte)tail.Length, 0, (byte)~tail.Length, 0xFF]); + zlib.AddRange(tail); + var hex = Convert.ToHexString([.. zlib])[..^1]; + + Object(4, hex.Length.ToString("D6", CultureInfo.InvariantCulture)); + Text(string.Create( + CultureInfo.InvariantCulture, + $"5 0 obj\n<< /Type /XRef /Size 40 /W [1 4 2] /Filter [/ASCIIHexDecode /FlateDecode] /Root 1 0 R /Length 4 0 R >>\nstream\n{hex}\nendstream\nendobj\nstartxref\n{five}\n%%EOF\n")); + return output.ToArray(); + } + + /// + /// A file of two classic tables. The first indexes the catalog, the page tree, the page and object 12, the highest + /// number; the second, an update, writes the catalog, the page tree and the page again, indexes them with object 4, a + /// page the tree does not list, and object 9, which holds the first table's offset after + /// spaces, and names that table through /Prev 9 0 R. Both trailers give /Size 13. /// - public static byte[] UpdateNamingItsPrevious(string prev) + public static byte[] UpdateHidingItsHighestNumber(int padding) { using var file = new Writer("1.4"); file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(12, "(twelve)"); var first = file.Position; file.Text(string.Create( CultureInfo.InvariantCulture, - $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\ntrailer\n<< /Size 4 /Root 1 0 R >>\nstartxref\n{first}\n%%EOF\n")); - file.Object(4, "(an update)"); - file.Object(9, first.ToString(CultureInfo.InvariantCulture)); + $"xref\n0 4\n0000000000 65535 f\r\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\n12 1\n{file.OffsetOf(12):D10} 00000 n\r\ntrailer\n<< /Size 13 /Root 1 0 R >>\nstartxref\n{first}\n%%EOF\n")); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(4, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + file.Object(9, new string(' ', padding) + first.ToString(CultureInfo.InvariantCulture)); var second = file.Position; file.Text(string.Create( CultureInfo.InvariantCulture, - $"xref\n4 1\n{file.OffsetOf(4):D10} 00000 n\r\n9 1\n{file.OffsetOf(9):D10} 00000 n\r\ntrailer\n<< /Size 10 /Root 1 0 R /Prev {prev.Replace("{offset}", first.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal)} >>\nstartxref\n{second}\n%%EOF\n")); + $"xref\n1 4\n{file.OffsetOf(1):D10} 00000 n\r\n{file.OffsetOf(2):D10} 00000 n\r\n{file.OffsetOf(3):D10} 00000 n\r\n{file.OffsetOf(4):D10} 00000 n\r\n9 1\n{file.OffsetOf(9):D10} 00000 n\r\ntrailer\n<< /Size 13 /Root 1 0 R /Prev 9 0 R >>\nstartxref\n{second}\n%%EOF\n")); + return file.ToArray(); + } + + /// + /// A file of one revision whose cross-reference stream 5 numbers 40 rows, 39 the page's, and takes its /Length + /// from object 4 when , which only it places. Its rows pass through three filters, the first + /// lacking its end marker and ending inside a unit: with , + /// /Filter [/ASCIIHexDecode /RunLengthDecode /FlateDecode], the hexadecimal digits short of their last; + /// otherwise /Filter [/ASCII85Decode /FlateDecode /RunLengthDecode], the last ASCII85 group partial. RunLength + /// writes literal runs of 128 bytes. + /// + public static byte[] RowsThroughThreeFilters(bool deferred, bool hex) => + RowsThroughFilters(deferred, hex ? ["AHx", "RL", "Fl"] : ["A85", "Fl", "RL"]); + + /// + /// A file as writes it, its rows passing through , as + /// /Filter names them: A85, AHx, Fl or RL. The first is ASCII85, its last group + /// partial and no ~>, or ASCIIHex short of its last digit and its >. + /// + public static byte[] RowsThroughFilters(bool deferred, string[] filters) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [39 0 R] /Count 1 >>"); + file.Object(39, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var four = file.Position; + var at = four + "4 0 obj\n000000\nendobj\n".Length; + var rows = new (int Type, long Second, int Third)[40]; + rows[0] = (0, 0, 65535); + rows[1] = (1, file.OffsetOf(1), 0); + rows[2] = (1, file.OffsetOf(2), 0); + rows[4] = (1, four, 0); + rows[5] = (1, at, 0); + rows[39] = (1, file.OffsetOf(39), 0); + var data = Encode(rows, Rows.Raw); + + for (var i = filters.Length - 1; i >= 0; i--) + { + data = filters[i] switch + { + "A85" => Encoding.ASCII.GetBytes(Ascii85(data)), + "AHx" => Encoding.ASCII.GetBytes(Convert.ToHexString(data)[..^1]), + "Fl" => Deflate(data), + "RL" => RunLength(data), + _ => throw new ArgumentOutOfRangeException(nameof(filters), filters[i], "No such filter."), + }; + } + + var names = string.Join(' ', Array.ConvertAll(filters, filter => filter switch + { + "A85" => "/ASCII85Decode", + "AHx" => "/ASCIIHexDecode", + "Fl" => "/FlateDecode", + _ => "/RunLengthDecode", + })); + var length = data.Length.ToString("D6", CultureInfo.InvariantCulture); + file.Object(4, length); + file.Stream(5, $"/Type /XRef /Size 40 /W [1 4 2] /Filter [{names}] /Root 1 0 R /Length {(deferred ? "4 0 R" : length)}", data); + file.StartXRef(at); + return file.ToArray(); + + static byte[] Deflate(byte[] bytes) + { + using var output = new MemoryStream(); + + using (var zlib = new ZLibStream(output, CompressionLevel.SmallestSize, leaveOpen: true)) + { + zlib.Write(bytes); + } + + return output.ToArray(); + } + + static byte[] RunLength(byte[] bytes) + { + var runs = new List(); + + for (var i = 0; i < bytes.Length; i += 128) + { + var count = Math.Min(128, bytes.Length - i); + runs.Add((byte)(count - 1)); + runs.AddRange(bytes.AsSpan(i, count)); + } + + runs.Add(128); + return [.. runs]; + } + } + + /// + /// A file of revisions, each a cross-reference stream whose /Length names an object only + /// its own rows place, its rows ending in before endstream, then + /// spaces before endobj: valid, its searches reading on through each run. + /// + public static byte[] UpdatesWithDeferredLengthsBeforeWhiteSpace(int count, int run, string endOfLine) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + var previous = -1L; + + for (var i = 0; i < count; i++) + { + var length = 4 + (2 * i); + var stream = length + 1; + file.Object(length, Invariant($"{(i == 0 ? 6 : 2) * 7}")); + var at = file.Position; + (int Type, long Second, int Third)[] own = [(1, file.OffsetOf(length), 0), (1, at, 0)]; + var rows = i == 0 + ? Encode([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0), (1, file.OffsetOf(3), 0), .. own], Rows.Raw) + : Encode(own, Rows.Raw); + file.Text(Invariant( + $"{stream} 0 obj\n<< /Type /XRef /Size {stream + 1} /W [1 4 2] /Index [{(i == 0 ? "0 6" : $"{length} 2")}] /Root 1 0 R{(previous < 0 ? string.Empty : $" /Prev {previous}")} /Length {length} 0 R >>\nstream\r\n")); + file.Bytes(rows); + file.Text(endOfLine + "endstream" + new string(' ', run) + "endobj\n"); + file.StartXRef(at); + previous = at; + } + return file.ToArray(); + + static string Invariant(FormattableString text) => text.ToString(CultureInfo.InvariantCulture); + } + + /// + /// A file of revisions, newest first in the file, each a cross-reference stream whose + /// /Length names an object only its own rows place, and each naming the next through /Prev, 1,024 spaces + /// past the rows of the one before — more than a search for a misplaced section spans —: the data of each runs over + /// the streams after it and spaces to the one endstream they share, at the end of the + /// file, as its length gives. Searched for it one after the other, past their first window, the streams read the file + /// about once each. With , the oldest revision comes first in the file, before them: stream + /// 5, whose /Length, object 4, which only its own rows place, holds the length after 1,100 spaces — a + /// MaxObjectLength of 1,024 cuts it before its value —, whose rows of objects 4, 5 and 3 lose their last 3 bytes, + /// the page's row lacking, and whose endstream 100 spaces part from its endobj, which object 20 follows — + /// or, with , 60 regular characters glued to it, which make it no keyword, then an end-of-line. + /// The newest stream indexes the catalog and the page tree, and the page where no such revision does. + /// + public static byte[] UpdatesSpendingTheSearches(int spenders, int padding, bool cutTarget, bool glued = false) + { + // Every offset and length is written ten digits wide: where each stream and its data lie is read once, then written. + var previous = new long[spenders]; + var lengths = new long[spenders]; + _ = Build(out var at, out var data, out var target, out var end); + + for (var i = 0; i < spenders; i++) + { + previous[i] = i + 1 < spenders ? at[i + 1] : target; + lengths[i] = end - data[i]; + } + + return Build(out _, out _, out _, out _); + + byte[] Build(out long[] streams, out long[] starts, out long oldest, out long shared) + { + using var file = new Writer("1.5"); + file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"); + oldest = -1; + + if (cutTarget) + { + file.Object(4, new string(' ', 1100) + "18"); + oldest = file.Position; + file.Text("5 0 obj\n<< /Type /XRef /Size 6 /W [1 4 2] /Index [4 2 3 1] /Root 1 0 R /Length 4 0 R >>\nstream\n"); + file.Bytes(Encode([(1, file.OffsetOf(4), 0), (1, oldest, 0), (1, file.OffsetOf(3), 0)], Rows.Raw)[..^3]); + file.Text("\nendstream" + (glued ? new string('x', 60) + "\n" : new string(' ', 100)) + "endobj\n"); + } + + for (var i = 0; i < spenders; i++) + { + file.Object(20 + (2 * i), lengths[i].ToString("D10", CultureInfo.InvariantCulture)); + } + + streams = new long[spenders]; + starts = new long[spenders]; + + for (var i = 0; i < spenders; i++) + { + // The data of the stream before takes in 1,024 spaces, more than a search for a misplaced section spans. + if (i > 0) + { + file.Text(new string(' ', 1024)); + } + + var length = 20 + (2 * i); + streams[i] = file.Position; + var rows = new List<(int Type, long Second, int Third)>(); + + if (i == 0) + { + rows.AddRange([(0, 0, 65535), (1, file.OffsetOf(1), 0), (1, file.OffsetOf(2), 0)]); + + if (!cutTarget) + { + rows.Add((1, file.OffsetOf(3), 0)); + } + } + + var index = string.Create(CultureInfo.InvariantCulture, $"{(i == 0 ? $"0 {rows.Count} " : string.Empty)}{length} 2"); + var prev = i + 1 < spenders || cutTarget + ? string.Create(CultureInfo.InvariantCulture, $" /Prev {previous[i]:D10}") + : string.Empty; + rows.AddRange([(1, file.OffsetOf(length), 0), (1, streams[i], 0)]); + file.Text(string.Create( + CultureInfo.InvariantCulture, + $"{length + 1} 0 obj\n<< /Type /XRef /Size 40 /W [1 4 2] /Index [{index}] /Root 1 0 R{prev} /Length {length} 0 R >>\nstream\n")); + starts[i] = file.Position; + file.Bytes(Encode([.. rows], Rows.Raw)); + } + + file.Text(new string(' ', padding)); + shared = file.Position; + file.Text("\nendstream\nendobj\n"); + file.StartXRef(spenders > 0 ? streams[0] : oldest); + return file.ToArray(); + } } private enum Rows @@ -227,16 +1818,45 @@ private static byte[] Encode((int Type, long Second, int Third)[] rows, Rows enc data.AddRange([(byte)type, (byte)(second >> 24), (byte)(second >> 16), (byte)(second >> 8), (byte)second, (byte)(third >> 8), (byte)third]); } - if (encoding == Rows.Raw) + return encoding == Rows.Raw ? [.. data] : Compress([.. data]); + } + + /// Writes in ASCII85, whole groups but the last, with no z and no ~>. + private static string Ascii85(ReadOnlySpan bytes) + { + var text = new StringBuilder(); + Span digits = stackalloc char[5]; + + for (var i = 0; i < bytes.Length; i += 4) { - return [.. data]; + var count = Math.Min(4, bytes.Length - i); + uint value = 0; + + for (var j = 0; j < 4; j++) + { + value = (value << 8) | (j < count ? bytes[i + j] : 0u); + } + + for (var j = 4; j >= 0; j--) + { + digits[j] = (char)('!' + (value % 85)); + value /= 85; + } + + text.Append(digits[..(count + 1)]); } + return text.ToString(); + } + + /// Compresses data in stored blocks, which show it as written. + private static byte[] Compress(byte[] data) + { using var output = new MemoryStream(); using (var zlib = new ZLibStream(output, CompressionLevel.NoCompression, leaveOpen: true)) { - zlib.Write([.. data]); + zlib.Write(data); } return output.ToArray(); @@ -256,6 +1876,8 @@ private sealed class Writer : IDisposable public void Text(string text) => _output.Write(Encoding.Latin1.GetBytes(text)); + public void Bytes(byte[] bytes) => _output.Write(bytes); + public void Object(int number, string body) { _offsets[number] = Position; diff --git a/tests/AdCodicem.Pdf.Tests/CorpusReadingTests.cs b/tests/AdCodicem.Pdf.Tests/CorpusReadingTests.cs index 3d50360a..645d07a1 100644 --- a/tests/AdCodicem.Pdf.Tests/CorpusReadingTests.cs +++ b/tests/AdCodicem.Pdf.Tests/CorpusReadingTests.cs @@ -248,9 +248,10 @@ public void A_document_holds_each_report_once_however_often_its_streams_are_deco [Fact] public void A_damaged_object_stream_s_fault_is_held_once_after_a_full_read_and_a_validation() { - // Object stream 417 of the PDFBox report turns corrupt where a block of zeros erased its data. Reading it serves its - // objects; reading object 5922 rebuilds the index, which decodes it again; the full read decodes it a third time, - // and the validator a fourth: its fault was held four times, and is held once (#144). + // Object stream 417 of the PDFBox report lost its end to a block of zeros: its data is the 1,302 bytes it declares, + // which no endstream follows before the next object (#174), and its Flate data ends before them. Reading it serves + // its objects; reading object 5922 rebuilds the index, which decodes it again; the full read decodes it a third + // time, and the validator a fourth: its fault was held four times, and is held once (#144). const string File = "remote/pdfbox/distiller6-zeroed-object-stream-pdfbox3947.pdf"; Assert.SkipUnless(Corpus.Paths.Contains(File), $"{File} is a remote document this machine did not fetch"); using var document = PdfDocument.Open(Corpus.PathOf(File), OptionsFor(Corpus.Get(File)) with { DiagnosticCapacity = int.MaxValue }); @@ -259,7 +260,7 @@ public void A_damaged_object_stream_s_fault_is_held_once_after_a_full_read_and_a new PdfValidator().Validate(document); document.Diagnostics.Where(d => d.Position == 1670741 && d.Code == PdfDiagnosticCodes.FilterFailed).Select(d => d.ToString()) - .Should().Equal("Warning filter.failed at 1670741: A Flate stream is corrupt at byte 33853 of its 34152; the 102785 bytes decoded before the fault was found were kept."); + .Should().Equal("Warning filter.failed at 1670741: A Flate stream ends before its data does; what decoded before the end was kept."); } [Theory] @@ -528,6 +529,7 @@ private static PdfReaderLimits LimitsFor(CorpusReaderLimits raised) return limits with { MaxDecodedStreamLength = raised.MaxDecodedStreamLength ?? limits.MaxDecodedStreamLength, + MaxStreamLength = raised.MaxStreamLength ?? limits.MaxStreamLength, MaxObjectLength = raised.MaxObjectLength ?? limits.MaxObjectLength, MaxXRefSectionLength = raised.MaxXRefSectionLength ?? limits.MaxXRefSectionLength, MaxXRefSectionCount = raised.MaxXRefSectionCount ?? limits.MaxXRefSectionCount, @@ -542,6 +544,7 @@ private static bool NamesARaisedLimit(string message, PdfReaderLimits raised) var defaults = PdfReaderLimits.Default; return (raised.MaxDecodedStreamLength != defaults.MaxDecodedStreamLength && message.Contains("PdfReaderLimits.MaxDecodedStreamLength", StringComparison.Ordinal)) + || (raised.MaxStreamLength != defaults.MaxStreamLength && message.Contains("PdfReaderLimits.MaxStreamLength", StringComparison.Ordinal)) || (raised.MaxObjectLength != defaults.MaxObjectLength && message.Contains("PdfReaderLimits.MaxObjectLength", StringComparison.Ordinal)) || (raised.MaxXRefSectionLength != defaults.MaxXRefSectionLength && message.Contains("PdfReaderLimits.MaxXRefSectionLength", StringComparison.Ordinal)) || (raised.MaxXRefSectionCount != defaults.MaxXRefSectionCount && message.Contains("PdfReaderLimits.MaxXRefSectionCount", StringComparison.Ordinal)) diff --git a/tests/AdCodicem.Pdf.Tests/CrossReferenceRuleTests.cs b/tests/AdCodicem.Pdf.Tests/CrossReferenceRuleTests.cs index c1e21ce7..5d873091 100644 --- a/tests/AdCodicem.Pdf.Tests/CrossReferenceRuleTests.cs +++ b/tests/AdCodicem.Pdf.Tests/CrossReferenceRuleTests.cs @@ -1307,6 +1307,102 @@ public void An_object_a_guard_left_out_of_the_index_is_no_reference_the_file_lac findings.Should().Contain(finding => finding.RuleId == PdfValidationRuleIds.XRefCheckedInPart); } + [Theory] + [InlineData(29)] + [InlineData(32)] + [InlineData(600)] + public void A_startxref_naming_white_space_far_before_a_classic_table_is_imprecise_and_the_table_read(int padding) + { + // Startxref names the start of white space before the xref keyword. The keyword lay past the first 32 bytes the + // reader looks at, or across them: the table was not read, the index was rebuilt, and startxref said to name neither + // the keyword nor an object, where a cross-reference stream is read past however much white space comes first. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"), + }; + file.Raw(new string(' ', padding)); + var table = file.Table(4, rows, "/Size 4 /Root 1 0 R"); + var bytes = Encoding.Latin1.GetBytes(Encoding.Latin1.GetString(file.Bytes) + .Replace( + string.Create(CultureInfo.InvariantCulture, $"startxref\n{table}\n"), + string.Create(CultureInfo.InvariantCulture, $"startxref\n{table - padding}\n"), + StringComparison.Ordinal)); + using var document = PdfDocument.Open(bytes); + + document.Diagnostics.Should().BeEmpty(); + Single(new PdfValidator().Validate(document), PdfValidationRuleIds.XRefOffsetImprecise).Message.Should().EndWith( + string.Create(CultureInfo.InvariantCulture, $"startxref gives offset {table - padding}, {padding} bytes before the xref keyword.")); + } + + [Theory(Timeout = 60_000)] + [InlineData(300, false)] + [InlineData(70_000, false)] + [InlineData(300, true)] + public async Task A_startxref_naming_more_white_space_than_a_trailer_s_window_before_a_classic_table_reaches_the_guard( + int padding, bool throwOnLimit) + { + // The reader reads on past white space for the xref keyword as far as MaxTrailerLength reaches, past which it would + // read the same window again and again: the guard is reached, the index rebuilt, or the guard thrown. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"), + }; + file.Raw(new string(' ', padding)); + var table = file.Table(4, rows, "/Size 4 /Root 1 0 R"); + var bytes = Encoding.Latin1.GetBytes(Encoding.Latin1.GetString(file.Bytes) + .Replace( + string.Create(CultureInfo.InvariantCulture, $"startxref\n{table}\n"), + string.Create(CultureInfo.InvariantCulture, $"startxref\n{table - padding}\n"), + StringComparison.Ordinal)); + var limits = padding > PdfReaderLimits.Default.MaxTrailerLength + ? PdfReaderLimits.Default + : PdfReaderLimits.Default with { MaxTrailerLength = 256 }; + var options = new PdfReaderOptions { Limits = limits, ThrowOnLimit = throwOnLimit }; + + if (throwOnLimit) + { + await Task.Run(() => FluentActions.Invoking(() => PdfDocument.Open(bytes, options)).Should().Throw(), TestContext.Current.CancellationToken); + return; + } + + using var document = await Task.Run(() => PdfDocument.Open(bytes, options), TestContext.Current.CancellationToken); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitTrailer, PdfDiagnosticCodes.XRefRebuilt); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Fact] + public void A_startxref_naming_white_space_that_runs_to_the_end_of_the_file_names_no_table() + { + // The white space runs out before the reader's window does: there is nothing more to read on to. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << >> >>"), + }; + var table = file.Table(4, rows, "/Size 4 /Root 1 0 R"); + var text = Encoding.Latin1.GetString(file.Bytes).Replace( + string.Create(CultureInfo.InvariantCulture, $"startxref\n{table}\n"), + "startxref\n##########\n", + StringComparison.Ordinal); + var bytes = Encoding.Latin1.GetBytes( + text.Replace("##########", text.Length.ToString("D10", CultureInfo.InvariantCulture), StringComparison.Ordinal) + + new string(' ', 100)); + using var document = PdfDocument.Open(bytes); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.XRefRebuilt); + Single(new PdfValidator().Validate(document), PdfValidationRuleIds.FileStartXRefWrong).Message.Should().EndWith( + "which holds neither the xref keyword nor an object."); + } + [Fact] public void A_startxref_naming_the_line_feed_before_a_cross_reference_stream_is_imprecise() { @@ -1611,6 +1707,21 @@ public void A_predictor_written_as_a_real_in_an_array_of_parameters_is_reported( "gives /Predictor of element 0 of its /DecodeParms as a real number, where Table 8 of ISO 32000-1 asks for an integer."); } + [Theory] + [InlineData("/Filter 5 0 R", "<< /Predictor 12.0 /Columns 7.0 >>")] + [InlineData("/Filter [5 0 R]", "<< /Predictor 12.0 /Columns 7.0 >>")] + [InlineData("/Filter << /Predictor 12.0 /Columns 7.0 >>", "0")] + [InlineData("/Filter [<< /Predictor 12.0 /Columns 7.0 >>]", "0")] + public void Parameters_a_filter_gives_as_a_dictionary_are_not_judged_by_what_decode_parameters_ask(string filter, string five) + { + // A /Filter, or an element of one, that is a dictionary ends decoding, the data left as it is: decoding reads nothing + // it holds, which Table 8, a /DecodeParms', does not govern. Its reals were reported as Table 8 asks (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] " + filter + " /Length {length}", five, compressed: false, predicted: false, placed: true); + + Validate(file).Findings.Select(finding => finding.Message).Should().NotContain(message => message.Contains("Table 8", StringComparison.Ordinal)); + } + [Fact] public void The_entry_a_deferred_length_found_off_its_object_is_judged_as_the_file_wrote_it() { diff --git a/tests/AdCodicem.Pdf.Tests/DiagnosticsTests.cs b/tests/AdCodicem.Pdf.Tests/DiagnosticsTests.cs index ce886667..f90b357f 100644 --- a/tests/AdCodicem.Pdf.Tests/DiagnosticsTests.cs +++ b/tests/AdCodicem.Pdf.Tests/DiagnosticsTests.cs @@ -149,6 +149,76 @@ public void A_copy_of_a_report_the_capacity_left_out_is_counted_again_each_time_ target.SuppressedCount.Should().Be(3); } + [Fact] + public void Rolling_back_beside_the_guards_keeps_their_reports_and_counts_those_the_capacity_left_out() + { + // A reading met two guards beside faults it is to make again, past a capacity of three: the guards stand, the one + // the capacity left no room for counted as left out still, and the faults are dropped, the one the capacity left + // out no longer counted. A mark that records no decoding reports, which the reader never rolls back to beside its + // guards, leaves which reports decodings moved in as it stands: a fault a decoding moved in before the mark is not + // moved in again. + var target = new PdfDiagnostics { Capacity = 3 }; + var decodings = target.Decodings; + var before = decodings.GetMark(); + decodings.Warn(PdfDiagnosticCodes.FilterFailed, "Moved in before the mark.", 3); + decodings.MoveDecodingReportsTo(target, before); + + var mark = target.GetMark(); + target.Warn("dropped", "Met beside the guards."); + target.ReachLimit(PdfDiagnosticCodes.LimitDecodedStream, "Kept.", 7); + target.Warn("left.out", "Past the capacity."); + target.ReachLimit(PdfDiagnosticCodes.LimitObject, "Past the capacity.", 9); + target.RollBackBesideLimits(mark); + + target.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.FilterFailed, PdfDiagnosticCodes.LimitDecodedStream); + target.LimitsReached.Should().Be(2); + target.SuppressedCount.Should().Be(1); + + var again = decodings.GetMark(); + decodings.Warn(PdfDiagnosticCodes.FilterFailed, "Moved in before the mark.", 3); + decodings.MoveDecodingReportsTo(target, again); + + target.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.FilterFailed, PdfDiagnosticCodes.LimitDecodedStream); + target.SuppressedCount.Should().Be(1); + } + + [Fact] + public void Rolling_back_beside_the_guards_holds_again_as_moved_in_only_the_guard_reports_a_decoding_moved_in() + { + // A guard reported outside any decoding before any decoding has reported — which the reader never rolls back beside + // to such a mark, a decoding having reported by then —, then a guard a decoding reached beside a fault, are each + // rolled back beside to a mark that records which reports decodings moved in. Every guard's report stands. The + // decoding's is held again as moved in, so that a decoding that meets it again does not report it twice; the fault + // dropped with the rollback is forgotten, so that a decoding that meets it again moves it in again (#144, #220). + var target = new PdfDiagnostics(); + var decodings = target.Decodings; + + var reading = target.GetMarkWithDecodingReports(); + target.ReachLimit(PdfDiagnosticCodes.LimitObject, "The reader's guard.", 9); + target.Warn("dropped", "Met beside the reader's guard."); + target.RollBackBesideLimits(reading); + + target.Select(d => d.Message).Should().Equal("The reader's guard."); + + var decoded = target.GetMarkWithDecodingReports(); + var first = decodings.GetMark(); + decodings.ReachLimit(PdfDiagnosticCodes.LimitDecodedStream, "The decoding's guard.", 7); + decodings.Warn(PdfDiagnosticCodes.FilterFailed, "Met beside the decoding's guard.", 7); + decodings.MoveDecodingReportsTo(target, first); + target.RollBackBesideLimits(decoded); + + target.Select(d => d.Message).Should().Equal("The reader's guard.", "The decoding's guard."); + + var again = decodings.GetMark(); + decodings.ReachLimit(PdfDiagnosticCodes.LimitDecodedStream, "The decoding's guard.", 7); + decodings.Warn(PdfDiagnosticCodes.FilterFailed, "Met beside the decoding's guard.", 7); + decodings.MoveDecodingReportsTo(target, again); + + target.Select(d => d.Message).Should().Equal("The reader's guard.", "The decoding's guard.", "Met beside the decoding's guard."); + target.LimitsReached.Should().Be(3); + target.SuppressedCount.Should().Be(0); + } + [Fact] public void A_nested_mark_keeps_or_drops_only_what_came_after_it() { diff --git a/tests/AdCodicem.Pdf.Tests/DocumentReaderTests.cs b/tests/AdCodicem.Pdf.Tests/DocumentReaderTests.cs index f69018d4..35dbd677 100644 --- a/tests/AdCodicem.Pdf.Tests/DocumentReaderTests.cs +++ b/tests/AdCodicem.Pdf.Tests/DocumentReaderTests.cs @@ -5,7 +5,9 @@ using AdCodicem.Pdf.Diagnostics; using AdCodicem.Pdf.Documents; using AdCodicem.Pdf.IO; +using AdCodicem.Pdf.IO.XRef; using AdCodicem.Pdf.Objects; +using AdCodicem.Pdf.Validation; namespace AdCodicem.Pdf.Tests; @@ -676,6 +678,55 @@ public void A_source_gives_an_empty_window_for_a_range_outside_itself_or_of_no_l window.Offset.Should().Be(0); } + [Theory] + [InlineData(-1, 2, false)] + [InlineData(0, -1, false)] + [InlineData(3, 2, false)] + [InlineData(5, 0, false)] + [InlineData(1, 3, true)] + [InlineData(4, 0, true)] + public void A_source_in_memory_lends_only_a_range_it_holds_whole(long offset, int length, bool lent) + { + // A stream's data is lent where it lies in a file in memory (#170). The reader asks only for data that starts in the + // file and whose length it clamped to the file, which no file can make otherwise: a range before the source, of a + // negative length, or running past its end is lent nothing, rather than sliced out of bytes the source does not hold. + // An empty range at its end is lent as empty. + var data = "%PDF"u8.ToArray(); + using var source = PdfFileSource.FromMemory(data); + + source.TryLend(offset, length, out var memory).Should().Be(lent); + memory.ToArray().Should().Equal(lent ? data.AsSpan((int)offset, length).ToArray() : []); + } + + [Fact] + public void A_stream_s_data_is_read_from_the_file_each_time_it_is_asked_for_and_kept_by_nothing() + { + // A stream's data was read once and kept with the stream for as long as the stream was held, by the object cache or + // by the caller (#170). It is read each time it is asked for: a file in memory lends it where it lies, and any other + // source is read again. + var file = new TestPdfBuilder() + .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") + .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") + .Stream(3, string.Empty, new string('A', 20_000)) + .BuildClassic(rootNumber: 1); + using var source = new StrictCountingSource(file); + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + var stream = document.GetObject(new PdfObjectId(3)).AsStream().Required(); + var before = source.BytesRead; + + var first = stream.GetRawBytes().ToArray(); + var second = stream.GetRawBytes().ToArray(); + + first.Should().Equal(second); + first.Should().HaveCount(20_000).And.OnlyContain(value => value == (byte)'A'); + (source.BytesRead - before).Should().Be(40_000, "the data is read each time it is asked for"); + + using var inMemory = PdfDocument.Open(file); + var lent = inMemory.GetObject(new PdfObjectId(3)).AsStream().Required().GetRawBytes(); + System.Runtime.InteropServices.MemoryMarshal.TryGetArray(lent, out var segment).Should().BeTrue(); + segment.Array.Should().BeSameAs(file, "a file in memory lends the data where it lies"); + } + [Fact] public void Disposing_a_document_twice_is_harmless() { @@ -1789,23 +1840,673 @@ public void Reads_every_row_of_a_cross_reference_stream_whose_data_a_deferred_le document.GetObject(new PdfObjectId(3)).AsDictionary().IsOfType(PdfName.Page).Should().BeTrue(); } + [Theory] + [InlineData(0)] + [InlineData(9_500)] + public void Rows_that_spell_endstream_after_the_length_s_own_row_are_read_as_the_length_gives(int padding) + { + // Two rows of reserved types spell an endstream after an end-of-line, a few bytes in or past the first 64 KB: the + // chain, which could not read the /Length, took the data up to it, and read the rows before it alone — once with a + // report, the section said to lack rows. The stream's own row for object 4 lies among them: the object it places + // gives the length, which the endstream after it confirms, and the data is taken again to it (#220). + var file = ChainFiles.AloneWithIndirectLength(spellingEndStream: true, padding: padding); + using var document = PdfDocument.Open(file); + + document.Diagnostics.Should().BeEmpty(); + document.WasRepaired.Should().BeFalse(); + document.Reader.Index.Count.Should().Be(6 + padding); + document.Reader.Structure.Sections.Should().ContainSingle().Which.Incomplete.Should().BeFalse(); + new PdfValidator().Validate(document).Findings.Should().BeEmpty(); + } + + [Fact] + public void Rows_that_spell_endstream_in_an_update_leave_none_of_its_objects_to_the_revision_it_supersedes() + { + // The update's stream indexes its /Length, object 4, and object 8 after rows that spell an endstream: read up to it, + // the stream held object 4's row alone, and object 8 read as the first revision wrote it, a valid file said to have + // a malformed section. The update's rows are read whole, and object 8 reads as it does in its direct twin (#220). + using var document = PdfDocument.Open(ChainFiles.UpdateWithIndirectLength(spellingEndStream: true)); + using var twin = PdfDocument.Open(ChainFiles.UpdateWithIndirectLength(spellingEndStream: false)); + + document.GetObject(new PdfObjectId(8)).Should().BeOfType().Which.ToText().Should().Be("new"); + twin.GetObject(new PdfObjectId(8)).Should().BeOfType().Which.ToText().Should().Be("new"); + document.Diagnostics.Should().BeEmpty(); + document.Reader.Index.Count.Should().Be(twin.Reader.Index.Count); + new PdfValidator().Validate(document).Findings.Should().BeEmpty(); + } + + [Fact] + public void A_deferred_length_is_taken_again_by_the_row_the_section_keeps_for_it() + { + // The stream's /Index numbers its /Length, object 4, first in a row of a reserved type, which reading the rows ignores, + // then in the row that places it, before rows that spell an endstream: the data was not taken again by the first, and + // the section was said to lack the rows past them, the page's among them (#220). + using var document = PdfDocument.Open(ChainFiles.AloneWithIndirectLengthBeforeSpelledRows(reservedRowFirst: true)); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.Structure.Sections.Should().ContainSingle().Which.Incomplete.Should().BeFalse(); + document.GetObject(new PdfObjectId(3)).AsDictionary().IsOfType(PdfName.Page).Should().BeTrue(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().BeFalse(); + } + + [Fact] + public void A_deferred_length_is_taken_again_by_a_row_with_no_type_field() + { + // With no type field, every row is of type 1: the section's row for its /Length, before rows that spell an endstream, + // places it, and the data is taken again to the length it gives (#220). + using var document = PdfDocument.Open(ChainFiles.AloneWithIndirectLengthAndNoTypeField()); + + document.Reader.Structure.Sections.Should().ContainSingle().Which.Incomplete.Should().BeFalse(); + document.GetObject(new PdfObjectId(3)).AsDictionary().IsOfType(PdfName.Page).Should().BeTrue(); + document.Diagnostics.Should().NotContain(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid); + } + + [Theory] + [InlineData(32)] + [InlineData(33)] + public void A_deferred_length_is_taken_again_through_as_many_references_as_the_parse_resolves_a_length_through_and_no_more(int objects) + { + // Stream 7's /Length names object 4, which only its own row places, before rows that spell an endstream; object 4 + // names object 100, which names the next, and so on, the newer table placing them, to the last, which holds the + // length. Through 32 objects, as far as the parse resolves a /Length, the data is taken again to that length, and + // every row is read. Through one more, the chain reaches no length, as a whole read of the stream reaches none: the + // data is left as the chain took it, up to the spelled endstream, and the rows past it are the file's to lack — the + // section incomplete, the length reported once the chain is read, the index rebuilt to find the page (#220). + var file = ChainFiles.UpdatePlacingAPaddedLengthChain(objects, ownRow: true); + using var document = PdfDocument.Open(file); + var section = document.Reader.Structure.Sections[1]; + + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + + if (objects == 32) + { + section.Incomplete.Should().BeFalse(); + document.Diagnostics.Should().BeEmpty(); + return; + } + + section.Incomplete.Should().BeTrue(); + section.Fault.Should().Be("it holds 5 rows where its /Index and /Size declare 8"); + document.WasRepaired.Should().BeTrue(); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which.Message.Should().Be( + "The /Length of cross-reference stream 7 names object 4 0, which holds null, not a length; the chain took its data up to its endstream, after 35 bytes."); + } + + [Fact] + public void A_deferred_length_a_newer_section_refused_takes_no_data_again() + { + // A third revision's table refuses the row it gives object 4, the update's /Length: no row of the update's stream + // stands for it then, and the data is left as the chain took it, up to the endstream its rows spell (#220). + var file = ChainFiles.UpdateWithIndirectLength(spellingEndStream: true); + var text = Encoding.Latin1.GetString(file); + var startxref = text.LastIndexOf("startxref\n", StringComparison.Ordinal) + "startxref\n".Length; + var second = text[startxref..text.IndexOf('\n', startxref)]; + var update = string.Create( + CultureInfo.InvariantCulture, + $"xref\n4 1\n0000000010 99999 n\r\ntrailer\n<< /Size 11 /Root 1 0 R /Prev {second} >>\nstartxref\n{file.Length}\n%%EOF\n"); + using var document = PdfDocument.Open(file.Concat(Encoding.ASCII.GetBytes(update)).ToArray()); + + var sections = document.Reader.Structure.Sections; + sections[0].RefusedRow.Should().NotBeNull(); + sections[1].Incomplete.Should().BeTrue(); + sections[1].Fault.Should().Be("it holds 1 row where its /Index and /Size declare 5"); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_deferred_length_is_not_checked_against_data_a_guard_cut(bool decoding) + { + // The stream's data, taken up to the endstream its rows spell, runs past MaxStreamLength: decoding cut it, and the + // length, read once the chain is, was checked against it, the guard's cut reported as the file's (ADR 34). Or the + // data, compressed in stored blocks that show the rows as written, decodes past MaxDecodedStreamLength, which kept + // it from being taken again to its length. + var file = ChainFiles.AloneWithIndirectLengthBeforeSpelledRows(reservedRowFirst: false, stored: decoding); + var limits = decoding + ? PdfReaderLimits.Default with { MaxDecodedStreamLength = 30 } + : PdfReaderLimits.Default with { MaxStreamLength = 30 }; + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = limits }); + using var whole = PdfDocument.Open(file); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + decoding ? PdfDiagnosticCodes.LimitDecodedStream : PdfDiagnosticCodes.LimitStreamLength); + document.Reader.Structure.Sections.Should().ContainSingle().Which.CutByLimit.Should().BeTrue(); + whole.Diagnostics.Should().BeEmpty(); + } + + [Fact] + public void A_filter_guard_that_keeps_a_deferred_length_s_data_from_being_taken_again_leaves_no_filter_fault() + { + // The stream's data, taken up to the endstream its rows spell, is compressed twice in stored blocks, and + // MaxFilterChainLength lets the first Flate alone run: over data cut short at the spelled endstream, it failed, and + // the guard, which kept the data from being taken again to its length, left that failure to the file (#220). + var file = ChainFiles.AloneWithIndirectLengthBeforeSpelledRows(reservedRowFirst: false, stored: true, layers: 2); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxFilterChainLength = 1 } }); + + whole.Diagnostics.Should().BeEmpty(); + document.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.LimitFilterChain); + document.Diagnostics.Should().NotContain(entry => entry.Code.StartsWith("filter.", StringComparison.Ordinal)); + + // The guard's report stands once, however often the stream is decoded again. + document.GetObject(new PdfObjectId(5)).AsStream().Required().Decode(); + new PdfValidator().Validate(document); + document.Diagnostics.Count(entry => entry.Code == PdfDiagnosticCodes.LimitFilterChain).Should().Be(1); + } + + [Theory] + [InlineData(1)] + [InlineData(0)] + public void A_guard_s_report_the_capacity_drops_beside_a_deferred_length_s_decoding_is_counted(int capacity) + { + // As above, under a capacity with room for one report, or none: the guard's report, which the capacity dropped, went + // with the decoding's other reports, neither kept nor counted (#220). + var file = ChainFiles.AloneWithIndirectLengthBeforeSpelledRows(reservedRowFirst: false, stored: true, layers: 2); + var limits = PdfReaderLimits.Default with { MaxFilterChainLength = 1 }; + using var full = PdfDocument.Open(file, new PdfReaderOptions { Limits = limits }); + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = limits, DiagnosticCapacity = capacity }); + + document.Diagnostics.Should().Equal(full.Diagnostics.Take(capacity)); + (document.Diagnostics.Count + document.Diagnostics.SuppressedCount).Should().Be(full.Diagnostics.Count); + } + + [Theory] + [InlineData(nameof(PdfReaderLimits.MaxFilterChainLength))] + [InlineData(nameof(PdfReaderLimits.MaxDecodedStreamLength))] + [InlineData(nameof(PdfReaderLimits.MaxStreamLength))] + public void A_fault_a_deferred_length_s_data_holds_before_a_guard_stops_it_is_the_file_s_as_for_a_direct_one(string limit) + { + // Stream 5's hexadecimal data starts with a stray x, which encodes nothing, and its rows spell no endstream. Read + // whole, the fault is the file's, whether the /Length names object 4, which only the stream places, or is written + // direct. Under a guard on decoding, or MaxStreamLength, only the guard's report stood when the /Length waited for + // the chain: but the fault lies before the guard stops, where the whole data meets it, and only how the data ends is + // the guard's to hide (#220). + var limits = limit switch + { + nameof(PdfReaderLimits.MaxFilterChainLength) => PdfReaderLimits.Default with { MaxFilterChainLength = 1 }, + nameof(PdfReaderLimits.MaxDecodedStreamLength) => PdfReaderLimits.Default with { MaxDecodedStreamLength = 20 }, + _ => PdfReaderLimits.Default with { MaxStreamLength = 40 }, + }; + using var direct = PdfDocument.Open(ChainFiles.UpdateOverAStrayHexByte(deferred: false), new PdfReaderOptions { Limits = limits }); + using var deferred = PdfDocument.Open(ChainFiles.UpdateOverAStrayHexByte(deferred: true), new PdfReaderOptions { Limits = limits }); + + Faults(direct).Should().ContainSingle().Which.Should().StartWith("An ASCIIHex stream holds a byte that encodes nothing"); + Faults(deferred).Should().Equal(Faults(direct)); + + static string[] Faults(PdfDocument document) => + [.. document.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.FilterFailed).Select(entry => entry.Message)]; + } + + [Fact] + public void A_deferred_length_s_data_whose_last_digit_alone_crosses_MaxDecodedStreamLength_is_cut_as_its_direct_twin_s() + { + // Stream 5's hexadecimal rows spell an endstream after the digits of 14 bytes and one more, where the deferred search + // stops: the lone digit makes a fifteenth byte, past MaxDecodedStreamLength. Decoded once more as data a guard cut, + // the digit was dropped and the data fit: the rows past the spelled endstream were charged to the file, the + // section rebuilt, where the direct twin's are the guard's. A hidden end silences what decoding says of the end, + // and changes nothing it decodes (#220, #224). + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 14 } }; + using var direct = PdfDocument.Open(ChainFiles.UpdateOverHexRows(deferred: false, compressed: false), options); + using var deferred = PdfDocument.Open(ChainFiles.UpdateOverHexRows(deferred: true, compressed: false), options); + + direct.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + deferred.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + deferred.WasRepaired.Should().BeFalse(); + deferred.Reader.Structure.Sections[^1].CutByLimit.Should().BeTrue(); + deferred.Reader.Structure.Sections[^1].Incomplete.Should().BeFalse(); + } + + [Theory] + [InlineData("\n")] + [InlineData("\r\n")] + public void Deferred_lengths_followed_by_long_white_space_have_it_read_on_once_however_their_data_ends(string endOfLine) + { + // Thirty revisions, each a cross-reference stream whose /Length only its own rows place, its endstream followed by + // 65,472 spaces before endobj. With a carriage return before the line feed, which a deferred length's data keeps, the + // search's end and the length's lie apart before one keyword, and each read on through the spaces: the file's + // searches read close to four times its length, the most they may, where twice is said (#174, #220). + var file = ChainFiles.UpdatesWithDeferredLengthsBeforeWhiteSpace(30, 65_472, endOfLine); + using var document = PdfDocument.Open(file); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.EndStreamSearchBytes.Should().BeLessThan(2L * file.Length + 1024); + } + + [Theory] + [InlineData("A85 RL RL RL")] + [InlineData("A85 RL RL Fl")] + [InlineData("A85 RL Fl RL")] + public void Data_whose_end_a_guard_hides_holds_no_more_rows_than_its_direct_twin_through_four_filters(string chain) + { + // As below, through four filters: the second, given less, wrote less, and left the third more room than the direct + // twin's decoding did, which read on, and the last read rows past where the direct twin stopped at the bound — the + // page indexed beside a guard, or rows and no rebuild where the direct twin rebuilds (#220). + var direct = ChainFiles.RowsThroughFilters(deferred: false, chain.Split(' ')); + var deferred = ChainFiles.RowsThroughFilters(deferred: true, chain.Split(' ')); + + for (var bound = 1; bound <= 1500; bound++) + { + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = bound } }; + using var directDocument = PdfDocument.Open(direct, options); + using var deferredDocument = PdfDocument.Open(deferred, options); + + Read(deferredDocument).Should().Be(Read(directDocument), $"MaxDecodedStreamLength is {bound}"); + } + + static string Read(PdfDocument document) => string.Join( + ' ', + document.WasRepaired, + document.Reader.Structure.Sections[^1].CutByLimit, + document.GetObject(new PdfObjectId(1)) is PdfDictionary, + document.GetObject(new PdfObjectId(39)) is PdfDictionary, + document.Reader.ChainIndex?.Count); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Data_whose_end_a_guard_hides_leaves_the_filters_after_a_dropped_group_the_room_its_direct_twin_s_have(bool hex) + { + // Stream 5's rows pass through three filters, the first lacking its end marker and ending inside a unit. Decoded once + // more knowing its end hidden, the first filter drops the unit, whose bytes counted against MaxDecodedStreamLength + // but not against what the filters before the last share: the filter after it was given more room than the direct + // twin's, and read a row past where it stopped — the page indexed beside a guard where the direct twin leaves it out, + // or a row and no rebuild where the direct twin rebuilds (#220). + var direct = ChainFiles.RowsThroughThreeFilters(deferred: false, hex); + var deferred = ChainFiles.RowsThroughThreeFilters(deferred: true, hex); + + for (var bound = 1; bound <= 400; bound++) + { + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = bound } }; + using var directDocument = PdfDocument.Open(direct, options); + using var deferredDocument = PdfDocument.Open(deferred, options); + + Read(deferredDocument).Should().Be(Read(directDocument), $"MaxDecodedStreamLength is {bound}"); + } + + static string Read(PdfDocument document) => string.Join( + ' ', + document.WasRepaired, + document.Reader.Structure.Sections[^1].CutByLimit, + document.GetObject(new PdfObjectId(1)) is PdfDictionary, + document.GetObject(new PdfObjectId(39)) is PdfDictionary, + document.Reader.ChainIndex?.Count); + } + + [Fact] + public void A_deferred_length_s_data_that_holds_every_row_under_a_guard_on_decoding_is_checked_against_its_length() + { + // Stream 5's RunLength runs hold its Flate rows and zero bytes, then the bytes of a spelled endstream, where the + // deferred search stops; MaxDecodedStreamLength stops the Flate output past every row. Nothing would take the data + // again, and the RunLength run the spelled endstream parts is reported as read whole, but the length, which says the + // chain took the data short, was not checked under the guard: the run-out stood as the file's fault alone (#220). + var file = ChainFiles.UpdateOverRunsSpellingAnEndStream(deferred: true); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 100 } }); + + whole.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid); + document.Diagnostics.Select(entry => entry.Code).Where(code => code != PdfDiagnosticCodes.LimitDecodedStream) + .Should().Equal(whole.Diagnostics.Select(entry => entry.Code)); + document.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.LimitDecodedStream); + } + + [Theory] + [InlineData(60, true)] + [InlineData(50, true)] + [InlineData(49, false)] + public void A_deferred_length_s_data_MaxStreamLength_cuts_past_every_row_is_checked_against_its_length(int bound, bool everyRow) + { + // As above, MaxStreamLength cutting the data past every row, from 50 bytes on — a RunLength byte, 7 of the stored + // block's header and 42 of rows —: the run the cut parts is the guard's, but the length, checked against where the + // search found the endstream, which no cut moves, says the chain took the data short, as read whole. It was left + // unchecked, as data a guard cut that lacks rows is, which the cut keeps from being taken again (#220). The Flate data + // is stored: compressed, where a cut parts it depended on the platform's zlib, which zlib-ng writes otherwise. + var file = ChainFiles.UpdateOverRunsSpellingAnEndStream(deferred: true, stored: true); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxStreamLength = bound } }); + var invalid = whole.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which; + + document.Diagnostics.Should().Equal(everyRow ? [document.Diagnostics[0], invalid] : [document.Diagnostics[0]]); + document.Diagnostics[0].Code.Should().Be(PdfDiagnosticCodes.LimitStreamLength); + document.Reader.Structure.Sections[^1].CutByLimit.Should().BeTrue(); + } + + [Fact] + public void A_deferred_length_s_data_whose_rows_a_guard_on_decoding_holds_all_of_reports_its_filters_ends() + { + // Stream 5's rows and 30 zero bytes, compressed, are written in hexadecimal with no ">", and MaxDecodedStreamLength + // stops the Flate output past every row the section declares: no row is missing, so nothing would take the data + // again, and the endstream it was taken to ends it whatever the guard does. Its missing ">" is the file's, as for + // the direct twin (#220). + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 50 } }; + using var direct = PdfDocument.Open(ChainFiles.UpdateOverHexRows(deferred: false, compressed: true, padding: 30), options); + using var deferred = PdfDocument.Open(ChainFiles.UpdateOverHexRows(deferred: true, compressed: true, padding: 30), options); + + direct.Diagnostics.Should().Contain(entry => entry.Message.StartsWith("An ASCIIHex stream ends without its > end marker", StringComparison.Ordinal)); + deferred.Diagnostics.Select(entry => entry.Message).Should().Equal(direct.Diagnostics.Select(entry => entry.Message)); + } + + [Fact] + public void A_deferred_length_whose_reference_leads_to_an_object_only_its_own_section_places_is_read_once_the_chain_is() + { + // The cross-reference stream's /Length, object 5, which the newer table places, holds a reference to object 6, which + // only the stream indexes: the chain read object 5 alone, and judged the length as holding null (#182). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", "6 0 R", compressed: false, predicted: false, placed: true, six: "56"); + using var document = PdfDocument.Open(file); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.Structure.Sections.Should().OnlyContain(section => section.State == IO.XRef.XRefSectionState.Read); + } + [Fact] public void Reports_the_rows_a_deferred_length_shows_the_chain_did_not_read() { - // The rows of objects 6 and 7, of reserved types, spell an endstream after an end-of-line: the chain, which could - // not read the /Length, took the data up to it. The length, read once the chain is, gives 14 bytes more, which the - // endstream after them confirms: what the chain left unread is reported. The rows are not read again. - var file = ChainFiles.AloneWithIndirectLength(spellingEndStream: true); + // The rows spell an endstream before the stream's own row for its /Length, the last: what the chain read holds no + // row to place it by. The length, read once the chain is, gives 21 bytes more, which the endstream after them + // confirms: what the chain left unread is reported, and the section's missing rows set off a rebuild. + var file = ChainFiles.AloneWithIndirectLength(spellingEndStream: true, lengthLast: true); using var document = PdfDocument.Open(file); - var report = document.Diagnostics.Should().ContainSingle().Which; - report.Code.Should().Be(PdfDiagnosticCodes.StreamLengthInvalid); + var report = document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which; report.Position.Should().Be(PdfTemplate.OffsetOf(file, "stream\n", 1) + "stream\n".Length); report.Message.Should().Be( - "The /Length of cross-reference stream 5 gives 56 bytes, which an endstream confirms; the chain, which could not read it, took the data up to an endstream 42 bytes in, and read no row past it."); + "The /Length of cross-reference stream 5 gives 63 bytes, which an endstream confirms; the chain, which could not read it, took the data up to an endstream 42 bytes in, and read no row past it."); + document.Reader.Structure.Sections.Should().ContainSingle().Which.Incomplete.Should().BeTrue(); document.GetObject(new PdfObjectId(3)).AsDictionary().IsOfType(PdfName.Page).Should().BeTrue(); } + [Theory] + [InlineData("generation")] + [InlineData("object stream")] + [InlineData("name")] + [InlineData("short")] + [InlineData("unconfirmed")] + [InlineData("unreadable")] + [InlineData("superseded")] + public void A_length_row_that_does_not_stand_leaves_the_data_up_to_the_endstream_the_rows_spell(string fault) + { + // The stream's row for object 4 places it under another generation, or in an object stream; object 4 holds a name; + // it gives no more than the data taken, or more that no endstream confirms; its header is misspelled; an update + // gives object 4 a place that holds nothing. None says where the data ends: the rows before the endstream they + // spell are all the section holds, as the chain took them (#220). + var file = ChainFiles.AloneWithIndirectLength( + spellingEndStream: true, lengthOff: fault switch { "short" => -14, "unconfirmed" => 3, _ => 0 }); + var row = (int)PdfTemplate.OffsetOf(file, "stream\n", 1) + "stream\n".Length + (4 * 7); + + switch (fault) + { + case "generation": + file[row + 6] = 1; + break; + case "object stream": + file[row] = 2; + break; + case "name": + case "unreadable": + var replaced = Encoding.Latin1.GetString(file).Replace( + "4 0 obj\n56\n", fault == "name" ? "4 0 obj\n/F\n" : "4 0 xbj\n56\n", StringComparison.Ordinal); + file = Encoding.Latin1.GetBytes(replaced); + break; + case "superseded": + file = WithUpdatePlacingFourOutside(file); + break; + } + + using var document = PdfDocument.Open(file); + + document.Reader.Structure.Sections[^1].Incomplete.Should().BeTrue(); + document.Reader.Structure.Sections[^1].Fault.Should().Be("it holds 6 rows where its /Index and /Size declare 8"); + + static byte[] WithUpdatePlacingFourOutside(byte[] file) + { + var stream = PdfTemplate.OffsetOf(file, "5 0 obj"); + var update = new StringBuilder("%").Append('x', 2 * PdfFileReader.NearbySearchRadius).Append('\n'); + var xref = file.Length + update.Length; + update.Append(CultureInfo.InvariantCulture, $"xref\n4 1\n0000099999 00000 n\r\ntrailer\n<< /Size 8 /Root 1 0 R /Prev {stream} >>\nstartxref\n{xref}\n%%EOF\n"); + return [.. file, .. Encoding.Latin1.GetBytes(update.ToString())]; + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Rows_that_spell_endstream_in_an_older_section_are_read_as_its_length_gives(bool twin) + { + // The older section's /Length names object 4, which only it indexes, and two of its rows spell an endstream: the + // search for it stops at the update's first object, past them. Its data is taken again to the length object 4 + // gives, as the newest section's is (#220). + var file = ChainFiles.OlderWithIndirectLength(spellingEndStream: !twin); + using var document = PdfDocument.Open(file); + + document.Diagnostics.Should().BeEmpty(); + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections.Should().HaveCount(2).And.OnlyContain(section => !section.Incomplete); + document.GetObject(new PdfObjectId(8)).Should().BeOfType().Which.ToText().Should().Be("new"); + new PdfValidator().Validate(document).Findings.Should().BeEmpty(); + } + + [Fact] + public void Updates_each_with_a_long_deferred_length_are_all_searched() + { + // Five revisions, each a cross-reference stream of 70,000 bytes of rows whose /Length only it indexes: each is + // searched for its endstream past the first window, up to the next object the newer sections place or the end of + // the file. Together they read about the file once, and one read past each: no search is left undone for want of + // what the document's searches may read (EndStreamSearchPasses). + var file = ChainFiles.UpdatesWithIndirectLengths(count: 5, padding: 10_000); + using var source = new StrictCountingSource(file); + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.Structure.Sections.Should().HaveCount(5).And.OnlyContain(section => section.State == XRefSectionState.Read && !section.Incomplete); + document.Reader.Index.Count.Should().Be(4 + (5 * 10_002)); + source.BytesRead.Should().BeLessThan(3L * file.Length); + } + + [Theory] + [InlineData(1)] + [InlineData(5)] + public void Reports_the_data_a_deferred_length_gives_past_the_chain_s_endstream_once_the_searches_have_read_all_they_may(int sections) + { + // Cross-reference streams written one after the other, with no endstream of their own, share 512 KB of data and the + // endstream after it, which four zero bytes and a second endstream follow; each one's /Length names an object only + // the oldest section, a classic table, places. Each is searched past its window up to the first endstream, and five + // such searches read more than the four times the file the document's searches may (EndStreamSearchPasses). The + // newest one's length, read once the chain is, gives the data up to the second endstream: what the chain did not read + // is reported, as for one such stream alone, though the file may no longer be read to find where the first starts. + var file = SectionsSharingTheirData(sections, 512 * 1024, newestExtra: 14); + using var document = PdfDocument.Open(file); + + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections.Should().HaveCount(sections + 1).And.OnlyContain(section => section.State == XRefSectionState.Read); + (document.Reader.EndStreamSearchBytes >= PdfFileReader.EndStreamSearchPasses * file.Length).Should().Be(sections == 5); + var report = document.Diagnostics.Should().ContainSingle().Which; + report.Code.Should().Be(PdfDiagnosticCodes.StreamLengthInvalid); + report.Position.Should().Be(PdfTemplate.OffsetOf(file, ">>\nstream\n", sections) + ">>\nstream\n".Length); + report.Message.Should().Be(string.Create( + CultureInfo.InvariantCulture, + $"The /Length of cross-reference stream {40 + sections} gives 524305 bytes, which an endstream confirms; the chain, which could not read it, took the data up to an endstream 524291 bytes in, and read no row past it.")); + } + + [Theory] + [InlineData(false, false, "\n")] + [InlineData(true, false, "\n")] + [InlineData(false, true, "\r\n")] + public void A_cross_reference_stream_whose_length_only_it_indexes_is_read_past_the_first_window_as_its_direct_twin( + bool compressed, bool endingInCarriageReturn, string endOfLine) + { + // 70,000 bytes of rows, past the 64 KB the chain's window first holds: the data, whose /Length object 4 gives and + // only the stream indexes, was searched for its endstream within the window alone, which grew to MaxTrailerLength, + // reported reached, and 9,350 of 10,000 rows were read. It is searched in the file, as any stream's whose length is + // not confirmed, and read whole — its last byte a carriage return too (#220). + var file = ChainFiles.AloneWithIndirectLength(padding: 10_000, compressed: compressed, endingInCarriageReturn: endingInCarriageReturn, endOfLine: endOfLine); + var options = new PdfReaderOptions { ThrowOnLimit = true }; + using var document = PdfDocument.Open(file, options); + + document.Diagnostics.Should().BeEmpty(); + document.WasRepaired.Should().BeFalse(); + document.Reader.Index.Count.Should().Be(10_006 + (endingInCarriageReturn ? 1 : 0)); + document.Reader.Structure.Sections.Should().ContainSingle().Which.CutByLimit.Should().BeFalse(); + } + + [Fact] + public void A_deferred_length_s_data_is_read_once_more_than_its_direct_twin_s() + { + // The search reads the data to its endstream, and the decoding reads it again: the file is read once more than its + // direct twin, whose /Length, written in place of "4 0 R", takes as many bytes, and not a byte past the next read. + var file = ChainFiles.AloneWithIndirectLength(padding: 10_000); + var text = Encoding.Latin1.GetString(file); + var twin = Encoding.Latin1.GetBytes(text.Replace("/Length 4 0 R", "/Length 70042", StringComparison.Ordinal)); + twin.Length.Should().Be(file.Length); + + var read = BytesReadOpening(file); + var twinRead = BytesReadOpening(twin); + + read.Should().BeInRange(twinRead, twinRead + 70_042 + PdfFileReader.EndStreamSearchChunk); + + static long BytesReadOpening(byte[] file) + { + using var source = new StrictCountingSource(file); + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + document.Diagnostics.Should().BeEmpty(); + return source.BytesRead; + } + } + + [Fact] + public void Rows_that_spell_an_object_header_do_not_end_a_deferred_length_s_data() + { + // Rows of reserved types spell "1 0 obj" past the first 64 KB: a header the bytes hold ends a search for a stream's + // endstream, but not one whose /Length the chain could not read, whose rows may spell anything (#220). + using var document = PdfDocument.Open(ChainFiles.AloneWithIndirectLength(padding: 10_000, spellingHeader: true)); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.Index.Count.Should().Be(10_006); + } + + [Fact] + public void A_deferred_length_s_data_with_no_endstream_before_the_end_of_the_file_makes_its_section_malformed() + { + // The stream's endstream is misspelled: its data, which no length the chain could read ends, ends nowhere, and the + // section is not read — once, the rest of the file was taken for its data (#220). The index is rebuilt. + var file = ChainFiles.AloneWithIndirectLength(padding: 10_000); + var at = (int)PdfTemplate.OffsetOf(file, "endstream"); + file[at] = (byte)'x'; + using var document = PdfDocument.Open(file); + + document.WasRepaired.Should().BeTrue(); + var section = document.Reader.Structure.Sections.Should().ContainSingle().Which; + section.State.Should().Be(XRefSectionState.Malformed); + section.Fault.Should().Be( + "its /Length is the reference 4 0 R, which no section read before it places where it can be read, and no endstream follows its data before the end of the file"); + document.Diagnostics.Select(entry => entry.Code).Should().NotContain(code => code.StartsWith("limit.", StringComparison.Ordinal)); + } + + [Fact] + public void A_deferred_length_s_data_with_no_endstream_before_the_next_object_makes_its_section_malformed() + { + // The older stream's /Length names object 9, which only it indexes, and its endstream is misspelled: the search for + // it stops at the update's first object the newer section places (#220). + var file = ChainFiles.UpdateWithIndirectLength(spellingEndStream: false, olderIndirect: true); + var at = (int)PdfTemplate.OffsetOf(file, "endstream"); + file[at] = (byte)'x'; + var next = PdfTemplate.OffsetOf(file, "8 0 obj\n(new)"); + using var document = PdfDocument.Open(file); + + document.Reader.Structure.Sections[1].Fault.Should().Be(string.Create( + CultureInfo.InvariantCulture, + $"its /Length is the reference 9 0 R, which no section read before it places where it can be read, and no endstream follows its data before the next object, at {next}")); + } + + [Fact] + public void A_deferred_length_s_data_the_searches_could_no_longer_read_makes_its_section_malformed_and_says_why() + { + // Six revisions, newest first in the file, each a cross-reference stream whose /Length only its own rows place, and + // whose data runs over the streams after it and 500,000 spaces to the endstream they share at the end of the file: + // the first five are each searched for it past their first window, and read the file about once each, more than + // four times in all — as much as the document's searches may (EndStreamSearchPasses). The sixth's data is searched + // in the bytes read with its object alone, which hold no endstream: it ends nowhere the section says, and the fault + // says the search was not made, not that no endstream follows the data (#220). + var file = ChainFiles.UpdatesSpendingTheSearches(spenders: 6, padding: 500_000, cutTarget: false); + using var document = PdfDocument.Open(file); + + var sections = document.Reader.Structure.Sections; + sections.Should().HaveCount(6); + sections.Take(5).Should().OnlyContain(section => section.State == XRefSectionState.Read && !section.Incomplete); + sections[5].State.Should().Be(XRefSectionState.Malformed); + sections[5].CutByLimit.Should().BeFalse(); + const string Fault = + "its /Length is the reference 30 0 R, which no section read before it places where it can be read, and its data was not searched for its endstream, the document's searches for endstream having read as much of the file as they may"; + sections[5].Fault.Should().Be(Fault); + document.Reader.EndStreamSearchBytes.Should().BeGreaterThanOrEqualTo(PdfFileReader.EndStreamSearchPasses * file.Length); + document.Diagnostics.Should().ContainSingle().Which.Should().Match(diagnostic => + diagnostic.Code == PdfDiagnosticCodes.XRefSectionUnreadable && diagnostic.Position == sections[5].Offset && + diagnostic.Message.Contains(Fault, StringComparison.Ordinal)); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Fact] + public void A_deferred_length_read_once_the_chain_is_says_what_its_chain_leads_to() + { + // Stream 7's /Length names object 5, which the newer table places, and which holds 6 0 R, which only stream 7 places: + // the length waits for the chain, then resolves to a name, which is said, as the parse says what it resolves to — not + // the reference object 5 holds (#220). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", "6 0 R", compressed: false, predicted: false, placed: true, six: "/Foo"); + using var document = PdfDocument.Open(file); + + document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( + "The /Length of cross-reference stream 7 names object 5 0, which holds the name /Foo, not a length; the chain took its data up to its endstream, after 56 bytes."); + } + + [Theory] + [InlineData(31, false)] + [InlineData(32, false)] + [InlineData(33, true)] + [InlineData(34, true)] + public void A_deferred_length_whose_chain_is_too_long_to_follow_is_reported_at_opening_as_a_later_load_reads_it(int objects, bool tooLong) + { + // Stream 7's /Length names object 5, which the newer table places, and which names object 100, the next, and so on; + // the last holds 56, and none is cut. The chain read follows as many objects as the parse resolves a /Length through, + // and reads a longer chain as null, as a later load of stream 7 does: the length is reported at opening, and the load + // adds nothing. Once, the parse resolved the reference the chain read gave back, took 56, and the report waited for + // the load (#220, #224). + var placed = new Dictionary(); + var last = 100 + objects - 2; + + for (var number = 100; number <= last; number++) + { + placed[number] = number < last ? string.Create(CultureInfo.InvariantCulture, $"{number + 1} 0 R") : "56"; + } + + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", "100 0 R", compressed: false, predicted: false, placed: true, placedToo: placed); + string[] expected = tooLong ? [PdfDiagnosticCodes.StreamLengthInvalid] : []; + using var document = PdfDocument.Open(file); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(expected); + + document.GetObject(new PdfObjectId(7)).AsStream().Required().GetRawBytes().Length.Should().Be(56); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(expected); + } + + [Fact] + public void A_deferred_length_s_fault_names_the_reference_at_the_end_of_its_chain() + { + // As above, the update placing object 9, which holds the reference 11 0 R, which no section places: the fault names + // the link the chain could not read, not the first, which it read (#220). + var file = ChainFiles.UpdateWithIndirectLength(spellingEndStream: false, olderIndirect: true, nine: "11 0 R"); + var at = (int)PdfTemplate.OffsetOf(file, "endstream"); + file[at] = (byte)'x'; + var next = PdfTemplate.OffsetOf(file, "8 0 obj\n(new)"); + using var document = PdfDocument.Open(file); + + document.Reader.Structure.Sections[1].Fault.Should().Be(string.Create( + CultureInfo.InvariantCulture, + $"its /Length holds the reference 11 0 R, which no section read before it places where it can be read, and no endstream follows its data before the next object, at {next}")); + } + [Theory] [MemberData(nameof(ChainFiles.Keys), MemberType = typeof(ChainFiles))] public void Changes_nothing_of_the_index_while_the_chain_is_read(string key) @@ -1884,11 +2585,13 @@ public void A_cross_reference_stream_whose_dictionary_the_trailer_guard_cuts_bef [Theory] [InlineData("<< /A 1 /B >>", 64 * 1024)] [InlineData("<< /Producer (an object longer than the trailer guard allows, which a place the section was moved to is only a guess at; it is read as itself when it is asked for, under the guard that bounds objects) >>", 128)] + [InlineData("<< /Type /Font /Subtype /CIDFontType2 /BaseFont /Foo /DW 1000 /W [0 [500] 1 [500] 2 [500] 3 [500] 4 [500] 5 [500] 6 [500] 7 [500] 8 [500] 9 [500] 10 [500] 11 [500] 12 [500] 13 [500] 14 [500] 15 [500] 16 [500] 17 [500] 18 [500] 19 [500] 20 [500] 21 [500] 22 [500] 23 [500] 24 [500] 25 [500] 26 [500] 27 [500] 28 [500] 29 [500]] >>", 128)] public void A_place_a_section_may_have_moved_to_that_holds_another_object_leaves_nothing_of_its_reading(string six, int maxTrailerLength) { // /Prev names a place 2 bytes into object 6, just before the older stream: object 6, nearer, is tried first, and is - // no section — a dictionary with a key short of a value, or one the trailer guard cuts. Neither its syntax fault - // nor the guard is reported: it was read as a section, and is read as itself when asked for. + // no section — a dictionary with a key short of a value, or one the trailer guard cuts, a CIDFont's whose /W gives + // its glyphs' widths among them. Neither its syntax fault nor the guard is reported: it was read as a section, and + // is read as itself when asked for. The font, once taken for the moved section, ended the chain as the guard's. var built = ChainFiles.UnderAnUpdate("/Type /XRef /Size 8 /W [1 4 2] /Length {length}", "0", compressed: false, predicted: false, placed: true, six: six); var stream = PdfTemplate.OffsetOf(built, "\n7 0 obj") + 1; var named = PdfTemplate.OffsetOf(built, "\n6 0 obj") + 3; @@ -1905,6 +2608,29 @@ public void A_place_a_section_may_have_moved_to_that_holds_another_object_leaves document.GetObject(new PdfObjectId(6)).Should().NotBeSameAs(PdfNull.Instance); } + [Fact] + public void A_place_a_section_may_have_moved_to_whose_dictionary_a_guard_cuts_is_no_section_the_chain_ends_at() + { + // Object 6 is a CIDFont that writes its /Type past a /W the trailer guard cuts: cut before it says what it is, the + // dictionary may be a section's, and the guard that hid it is reported. It is not the section all the same — only a + // value a section is read with that leads to an object a guard cut is (#263) —: the older stream, farther, is found. + var built = ChainFiles.UnderAnUpdate("/Type /XRef /Size 8 /W [1 4 2] /Length {length}", "0", compressed: false, predicted: false, placed: true, six: "<< /Subtype /CIDFontType2 /W [0 [500] 1 [500] 2 [500] 3 [500] 4 [500] 5 [500] 6 [500] 7 [500] 8 [500] 9 [500] 10 [500] 11 [500] 12 [500] 13 [500] 14 [500] 15 [500] 16 [500] 17 [500] 18 [500] 19 [500] 20 [500] 21 [500] 22 [500] 23 [500] 24 [500] 25 [500] 26 [500] 27 [500] 28 [500] 29 [500]] /Type /Font >>"); + var stream = PdfTemplate.OffsetOf(built, "\n7 0 obj") + 1; + var named = PdfTemplate.OffsetOf(built, "\n6 0 obj") + 3; + var file = Encoding.Latin1.GetBytes(Encoding.Latin1.GetString(built).Replace( + string.Create(CultureInfo.InvariantCulture, $"/Prev {stream}"), + string.Create(CultureInfo.InvariantCulture, $"/Prev {named}"), + StringComparison.Ordinal)); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxTrailerLength = 128 } }; + + using var document = PdfDocument.Open(file, options); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(diagnostic => diagnostic.Code).Should().Equal(PdfDiagnosticCodes.LimitTrailer, PdfDiagnosticCodes.XRefOffsetAdjusted); + document.Diagnostics[1].Message.Should().EndWith(string.Create(CultureInfo.InvariantCulture, $"was found {stream - named} bytes from there.")); + document.GetObject(new PdfObjectId(6)).AsDictionary().IsOfType(PdfName.Font).Should().BeTrue(); + } + [Fact] public void Reads_a_value_the_chain_refers_to_past_the_window_it_starts_with() { @@ -2180,15 +2906,21 @@ public void A_newest_size_far_past_the_room_made_so_far_is_not_made_room_for_at_ document.Reader.Index.Capacity.Should().BeLessThan(4000, "the map doubles from the newest section's 1,103 slots"); } - [Fact] - public void An_update_s_rows_and_the_original_s_make_room_for_the_objects_the_newest_size_numbers() + [Theory] + [InlineData("3000")] + [InlineData("3000.0")] + public void An_update_s_rows_and_the_original_s_make_room_for_the_objects_the_newest_size_numbers(string size) { // The update rewrites a thousand of the original's 3,000 objects: the original's rows for them supersede nothing, - // and room made for all its rows on top of the update's doubled the map. The newest /Size numbers every object. - var file = TestPdfBuilder.AppendIncrementalUpdate( + // and room made for all its rows on top of the update's doubled the map. The newest /Size numbers every object, + // written direct, as an integer or as a real with no fractional part, which the reader reads as the integer it is. + var built = TestPdfBuilder.AppendIncrementalUpdate( ChainOfSections(sections: 1, rows: 3000), rootNumber: 1, [.. Enumerable.Range(3, 1000).Select(number => (number, "(updated)"))]); + var text = Encoding.Latin1.GetString(built); + var newest = text.LastIndexOf("/Size 3000", StringComparison.Ordinal); + var file = Encoding.Latin1.GetBytes(string.Concat(text.AsSpan(0, newest), "/Size " + size, text.AsSpan(newest + "/Size 3000".Length))); using var document = PdfDocument.Open(file); @@ -2386,6 +3118,63 @@ private static byte[] ChainOfSections(int sections, int rows) return Encoding.ASCII.GetBytes(file.ToString()); } + /// + /// A classic table placing the catalog, an empty page tree and objects 31 onward, then + /// cross-reference streams, 41 onward, each naming the one before it, the oldest the table, written one after the other + /// with no endstream of its own: the data of each, its one free row first, runs through the streams after it and + /// bytes of x to the endstream they share, which four zero bytes and a second + /// endstream follow. Each stream's /Length is the object the table places for it, 31 onward, which gives + /// its data up to the first endstream; the newest one's gives bytes more. + /// + private static byte[] SectionsSharingTheirData(int count, int padding, int newestExtra) + { + // Each length is written in ten digits: the offsets the first pass finds hold for the second, which writes them. + var lengths = new long[count]; + var bytes = Array.Empty(); + + for (var pass = 0; pass < 2; pass++) + { + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + + for (var i = 0; i < count; i++) + { + rows[31 + i] = file.Object(31 + i, lengths[i].ToString("D10", CultureInfo.InvariantCulture)); + } + + var previous = file.Table(31 + count, rows, string.Create(CultureInfo.InvariantCulture, $"/Size {31 + count} /Root 1 0 R")); + var dataStarts = new long[count]; + + for (var i = 0; i < count; i++) + { + var at = file.Position; + file.Raw(string.Create( + CultureInfo.InvariantCulture, + $"{41 + i} 0 obj\n<< /Type /XRef /Size 60 /W [1 1 1] /Index [{51 + i} 1] /Root 1 0 R /Prev {previous} /Length {31 + i} 0 R >>\nstream\n")); + dataStarts[i] = file.Position; + file.Raw("\0\0\0"); + previous = at; + } + + file.Raw(new string('x', padding)); + var dataEnd = file.Position; + file.Raw("\nendstream\0\0\0\0\nendstream\nendobj\n"); + file.Raw(string.Create(CultureInfo.InvariantCulture, $"startxref\n{previous}\n%%EOF\n")); + bytes = file.Bytes; + + for (var i = 0; i < count; i++) + { + lengths[i] = dataEnd - dataStarts[i] + (i == count - 1 ? newestExtra : 0); + } + } + + return bytes; + } + /// Where the data of the file's object stream starts. private static long ObjectStreamDataStart(byte[] file) { @@ -2497,4 +3286,136 @@ protected override void Dispose(bool disposing) base.Dispose(disposing); } } + + [Theory] + [InlineData(28, 12)] + [InlineData(21, 12)] + [InlineData(49, 12)] + [InlineData(56, 12)] + [InlineData(21, 2)] + [InlineData(28, 2)] + public void Reads_a_cross_reference_stream_whose_length_row_a_spelled_endstream_parts_the_predicted_row_of(int columns, int predictor) + { + // Stream 7's /Length names object 5, which only its own row places, and its rows spell an endstream right after that + // row, which parts the predicted row it lies in: the data up to it decodes, as data whose end the file gives, to the + // rows before that one, the parted one dropped — or, the first parted, to data the PNG predictor leaves as decoded, + // its tag bytes in the rows; or, under the TIFF predictor, to that row still differenced —, and the row for the + // /Length was not found, or found garbled. The section was charged with the rows it lacked and the index rebuilt + // (#220). + var file = ChainFiles.DeferredLengthRowPartedBySpelledEndStream(columns, predictor); + + using var document = PdfDocument.Open(file); + + document.Diagnostics.Should().BeEmpty(); + document.WasRepaired.Should().BeFalse(); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Theory] + [InlineData(0)] + [InlineData(3)] + [InlineData(6)] + public void Reads_a_cross_reference_stream_whose_rows_a_length_it_places_runs_past_spell_an_endstream_before_the_first(int prefix) + { + // Stream 7's /Length names object 5, which only its own row places, and its data spells an endstream before its first + // row: the data up to it holds no whole row, which the row for the /Length cannot lie in, nor a predicted row undone, + // and the section lacks every row, the file's. Asked to look for that row again, the reader would divide by the + // length of a row it took to be none. + var file = ChainFiles.DeferredLengthEndStreamSpelledBeforeRows(prefix); + + using var document = PdfDocument.Open(file); + + document.WasRepaired.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.XRefRebuilt); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void A_decoding_guard_a_deferred_length_s_data_reaches_once_its_predicted_row_is_undone_is_reached_as_read_whole(bool rowBefore) + { + // Stream 7's /Length names object 5, which only its own row places, 1,820 bytes or more into the rows a RunLength step + // decodes after a predicted Flate one; its rows spell an endstream that parts a predicted row, past object 5's row or + // before it. MaxDecodedStreamLength, 1,100 bytes, stops the RunLength step read whole, the rows past it the guard's. + // The data up to the endstream, its end the file's, decodes within the bound; decoded again, its predicted row undone + // as far as it goes, to look for the /Length's row, it reached the bound, which the reader dropped with what that + // decoding reported: the section was charged with the rows it lacked and the index rebuilt, where ThrowOnLimit threw. + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 1100 } }; + + using (var whole = PdfDocument.Open(ChainFiles.DeferredLengthRowPastRunLengthRuns(rowBefore, direct: true), options)) + { + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + whole.WasRepaired.Should().BeFalse(); + } + + var file = ChainFiles.DeferredLengthRowPastRunLengthRuns(rowBefore); + + // Unbounded, the row before the endstream is found in the data decoded again, which is taken again to the length; one + // past it is not among the rows the data holds, which a length checked once the chain is read says (#220). + using (var unbounded = PdfDocument.Open(file)) + { + unbounded.WasRepaired.Should().Be(!rowBefore); + } + + using var document = PdfDocument.Open(file, options); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections[0].CutByLimit.Should().BeTrue(); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + + var throwing = () => PdfDocument.Open(file, options with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxDecodedStreamLength)); + } + + [Fact] + public void Rows_a_decoding_guard_stops_once_a_parted_predicted_row_is_undone_are_read_as_far_as_the_guard_lets_them() + { + // As above, but the page, object 155, has its row in the predicted row the spelled endstream parts, the 152nd, 1,057 + // bytes into the rows, within the 1,100 bytes MaxDecodedStreamLength lets the RunLength step decode, and object 5's + // row past them. Read whole, the rows past the bound are the guard's, the page's row read. The data decoded again, + // its predicted row undone as far as it goes, stops at the bound short of the /Length's row: where the rows end is + // the guard's, the rows it holds read — not only those the decoding before, which dropped the parted row, held. + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 1100 } }; + + using (var whole = PdfDocument.Open(ChainFiles.DeferredLengthPageRowInPartedRunLengthRow(direct: true), options)) + { + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + whole.GetObject(new PdfObjectId(155)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + var file = ChainFiles.DeferredLengthPageRowInPartedRunLengthRow(); + + using (var unbounded = PdfDocument.Open(file)) + { + unbounded.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, options); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + document.WasRepaired.Should().BeFalse(); + document.GetObject(new PdfObjectId(155)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Fact] + public void A_deferred_length_no_row_numbers_is_not_looked_for_in_data_decoded_again() + { + // Stream 7's /Length names object 5, which no section places and no subsection of its own numbers, and its rows spell + // an endstream that parts a predicted row. The data up to it, its end the file's, decodes within the 1,100 bytes + // MaxDecodedStreamLength lets the RunLength step decode; decoded again, its predicted row undone, it would pass them, + // but no row could give the length there: what is read is what an unbounded reading reads, no guard reached. + var file = ChainFiles.DeferredLengthNoRowNumbersPastRunLengthRuns(); + string[] unboundedCodes; + + using (var unbounded = PdfDocument.Open(file)) + { + unboundedCodes = [.. unbounded.Diagnostics.Select(entry => entry.Code)]; + } + + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 1100 } }); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(unboundedCodes).And.NotContain(PdfDiagnosticCodes.LimitDecodedStream); + } } diff --git a/tests/AdCodicem.Pdf.Tests/FilterDamageTests.cs b/tests/AdCodicem.Pdf.Tests/FilterDamageTests.cs index 64d4f684..06429a04 100644 --- a/tests/AdCodicem.Pdf.Tests/FilterDamageTests.cs +++ b/tests/AdCodicem.Pdf.Tests/FilterDamageTests.cs @@ -1233,9 +1233,10 @@ public void Reports_no_missing_end_marker_where_a_filter_stopped_at_the_bound_co [InlineData("own bound")] public void Reports_no_missing_greater_than_sign_where_the_reader_cut_ascii85_data_after_its_tilde(string cut) { - // The ~> closes the data, and the reader's cut falls between its two characters: the window MaxObjectLength allows - // ends on the ~, or the hex filter before stops at the bound there. Or the filter stops at its own bound in its - // last group, after the ~ was read. What lies past a cut is the guard's, and only the guard is reported (#199). + // The ~> closes the data, and the reader's cut falls between its two characters: the data MaxStreamLength lets the + // reader read ends on the ~, or the hex filter before stops at the bound there. Or the filter stops at its own bound + // in its last group, after the ~ was read. What lies past a cut is the guard's, and only the guard is reported + // (#199). const string Encoded = "87cURD]i,\"Ebo7~>"; var diagnostics = new PdfDiagnostics(); string[] expected; @@ -1245,16 +1246,18 @@ public void Reports_no_missing_greater_than_sign_where_the_reader_cut_ascii85_da var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") - .WithObject(3, $"<< /Filter /ASCII85Decode /Length 2 >>\nstream\n{Encoded}{new string(' ', 400)}\nendstream") + .Stream(3, "/Filter /ASCII85Decode", Encoded + new string(' ', 400)) .BuildClassic(rootNumber: 1); - var header = file.AsSpan().IndexOf("3 0 obj"u8); - var data = file.AsSpan().IndexOf(Encoding.ASCII.GetBytes(Encoded)); - var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = data + Encoded.Length - 1 - header } }; + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxStreamLength = Encoded.Length - 1 } }; using var document = PdfDocument.Open(file, options); Encoding.ASCII.GetString(document.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(diagnostics).Span).Should().Be("Hello World"); - expected = [PdfDiagnosticCodes.LimitObject]; - document.Diagnostics.Select(d => d.Code).Concat(diagnostics.Select(d => d.Code)).Should().Equal(expected); + + // The guard is reported once in the document's diagnostics, as the reader reads the data, and in the caller's own, + // as decoding meets the cut. + expected = [PdfDiagnosticCodes.LimitStreamLength]; + document.Diagnostics.Select(d => d.Code).Should().Equal(expected); + diagnostics.Select(d => d.Code).Should().Equal(expected); return; } @@ -1334,18 +1337,16 @@ public void Reports_no_end_marker_a_guard_of_the_reader_cut_off_and_a_fault_in_w var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") - .WithObject(3, $"<< /Filter [/ASCIIHexDecode /{filter}] /Length 10 >>\nstream\n{hex}\nendstream") + .Stream(3, $"/Filter [/ASCIIHexDecode /{filter}]", hex) .BuildClassic(rootNumber: 1); - var header = file.AsSpan().IndexOf("3 0 obj"u8); - var data = file.AsSpan().IndexOf(Encoding.ASCII.GetBytes(hex)); - var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = data + characters - header } }; + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxStreamLength = characters } }; using var document = PdfDocument.Open(file, options); document.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(document.Diagnostics); string[] expected = stray && filter == "ASCII85Decode" - ? [PdfDiagnosticCodes.LimitObject, PdfDiagnosticCodes.FilterFailed] - : [PdfDiagnosticCodes.LimitObject]; + ? [PdfDiagnosticCodes.LimitStreamLength, PdfDiagnosticCodes.FilterFailed] + : [PdfDiagnosticCodes.LimitStreamLength]; document.Diagnostics.Select(d => d.Code).Should().Equal(expected); } @@ -1429,56 +1430,49 @@ public void Counts_the_bytes_a_damaged_stream_kept_before_its_predictor_takes_th [Fact] public void Reports_a_fault_in_the_part_of_a_stream_a_guard_of_the_reader_kept_and_counts_that_part() { - // A /Length far too short sends the reader looking for endstream, past the largest window MaxObjectLength - // allows, which ends 10,000 bytes into the stream's data, well past the byte its Flate data turns corrupt - // at. The guard is reported, and so is the fault, which lies in bytes the reader read and is the file's: - // counted in the data the Flate filter was given, which is what the window kept. + // MaxStreamLength lets the reader read 10,000 bytes of the stream's hex data, well past the byte its Flate data + // turns corrupt at. The guard is reported, and so is the fault, which lies in bytes the reader read and is the + // file's: counted in the data the Flate filter was given, which is what the guard kept. const int Kept = 10_000; var plain = ContentStream(200); var hex = Convert.ToHexString(CorruptAfter(plain, out var faultAt, after: RandomBytes(60_000))) + ">"; var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") - .WithObject(3, $"<< /Filter [/ASCIIHexDecode /FlateDecode] /Length 10 >>\nstream\n{hex}\nendstream") + .Stream(3, "/Filter [/ASCIIHexDecode /FlateDecode]", hex) .BuildClassic(rootNumber: 1); - var header = file.AsSpan().IndexOf("3 0 obj"u8); - var dataStart = file.AsSpan().IndexOf(Encoding.ASCII.GetBytes(hex)); - var options = PdfReaderOptions.Default with - { - Limits = PdfReaderLimits.Default with { MaxObjectLength = dataStart + (2 * Kept) - header }, - }; + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxStreamLength = 2 * Kept } }; faultAt.Should().BeLessThan(Kept); using var document = PdfDocument.Open(file, options); var decoded = document.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(document.Diagnostics); decoded.ToArray().Should().Equal(plain); - document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitObject, PdfDiagnosticCodes.FilterFailed); + document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitStreamLength, PdfDiagnosticCodes.FilterFailed); document.Diagnostics[^1].Message.Should().Be(CorruptAt(faultAt, Kept, plain.Length)); } [Fact] public void Says_nothing_of_the_tail_a_guard_of_the_reader_cut_off() { - // A /Length far too short sends the reader looking for endstream, which lies past the largest window - // MaxObjectLength allows: the object is kept as far as the window went, and the guard is reported. - // That the Flate data then runs out is the reader's doing, not the file's. + // The stream's data runs past what MaxStreamLength lets the reader read: the data is kept as far as the guard + // allows, and the guard is reported. That the Flate data then runs out is the reader's doing, not the file's. var compressed = Compress(RandomBytes(100_000), CompressionLevel.Optimal); var hex = Convert.ToHexString(compressed) + ">"; var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") - .WithObject(3, $"<< /Filter [/ASCIIHexDecode /FlateDecode] /Length 10 >>\nstream\n{hex}\nendstream") + .Stream(3, "/Filter [/ASCIIHexDecode /FlateDecode]", hex) .BuildClassic(rootNumber: 1); - var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = 64 * 1024 } }; + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxStreamLength = 64 * 1024 } }; using var cut = PdfDocument.Open(file, options); using var whole = PdfDocument.Open(file); cut.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(cut.Diagnostics); whole.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(whole.Diagnostics).Length.Should().Be(100_000); - cut.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); - whole.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.StreamLengthInvalid); + cut.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitStreamLength); + whole.Diagnostics.Should().BeEmpty(); } [Theory] @@ -1562,27 +1556,24 @@ public void Drops_the_unit_a_filter_before_it_parted_at_the_bound_rather_than_co [InlineData("ASCII85Decode", 40)] public void Drops_the_unit_a_guard_of_the_reader_parted_rather_than_complete_it(string filter, int characters) { - // A /Length far too short sends the reader looking for endstream past the largest window MaxObjectLength - // allows, which ends after the given number of characters of the data: a digit without its pair, or a group - // short of five, is one the guard parted, and what comes back is the start of the data, with the guard alone + // MaxStreamLength lets the reader read the given number of characters of the data: a digit without its pair, or a + // group short of five, is one the guard parted, and what comes back is the start of the data, with the guard alone // reported. var plain = Encoding.ASCII.GetBytes(string.Concat(Enumerable.Repeat("The quick brown fox. ", 20))); var encoded = Encoding.ASCII.GetString(Encode(plain, filter)); var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") - .WithObject(3, $"<< /Filter /{filter} /Length 10 >>\nstream\n{encoded}\nendstream") + .Stream(3, $"/Filter /{filter}", encoded) .BuildClassic(rootNumber: 1); - var header = file.AsSpan().IndexOf("3 0 obj"u8); - var data = file.AsSpan().IndexOf(Encoding.ASCII.GetBytes(encoded)); - var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = data + characters - header } }; + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxStreamLength = characters } }; using var document = PdfDocument.Open(file, options); var decoded = document.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(document.Diagnostics); var whole = filter == "ASCIIHexDecode" ? characters / 2 : characters / 5 * 4; decoded.ToArray().Should().Equal(plain.AsSpan(0, whole).ToArray()); - document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitStreamLength); } [Theory] @@ -1590,23 +1581,21 @@ public void Drops_the_unit_a_guard_of_the_reader_parted_rather_than_complete_it( [InlineData("ASCII85Decode", "87cURD]i,\"Ebo7~>", "Hello World")] public void Completes_the_unit_its_end_marker_closes_however_the_reader_cut_the_data_after_it(string filter, string encoded, string expected) { - // The window MaxObjectLength allows ends a few characters past the end marker: the digit, or the short group, - // before the marker is the file's last, which the marker closes, and is completed as in data nothing cut. + // The data MaxStreamLength lets the reader read ends a few characters past the end marker: the digit, or the short + // group, before the marker is the file's last, which the marker closes, and is completed as in data nothing cut. var data = encoded + new string(' ', 400); var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") - .WithObject(3, $"<< /Filter /{filter} /Length 2 >>\nstream\n{data}\nendstream") + .Stream(3, $"/Filter /{filter}", data) .BuildClassic(rootNumber: 1); - var header = file.AsSpan().IndexOf("3 0 obj"u8); - var start = file.AsSpan().IndexOf(Encoding.ASCII.GetBytes(data)); - var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = start + encoded.Length + 5 - header } }; + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxStreamLength = encoded.Length + 5 } }; using var document = PdfDocument.Open(file, options); var decoded = document.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(document.Diagnostics); Encoding.ASCII.GetString(decoded.Span).Should().Be(expected); - document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitStreamLength); } [Fact] @@ -1678,7 +1667,7 @@ public void Says_nothing_of_the_end_of_data_a_filter_before_it_stopped_at_the_bo [InlineData(100)] public void Says_nothing_of_a_guard_s_cut_that_falls_in_the_checksum_or_after_the_data(int bytesPastTheBody) { - // The largest window MaxObjectLength allows ends 2 bytes into the checksum, or in the white space after + // The data MaxStreamLength lets the reader read ends 2 bytes into the checksum, or in the white space after // the whole zlib stream: a checksum the reader cut is no repair of the file's, and data that ended // before the cut is whole. Only the guard is reported. var compressed = Compress(ContentStream(200), CompressionLevel.Optimal); @@ -1686,19 +1675,17 @@ public void Says_nothing_of_a_guard_s_cut_that_falls_in_the_checksum_or_after_th var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") - .WithObject(3, $"<< /Filter [/ASCIIHexDecode /FlateDecode] /Length 10 >>\nstream\n{hex}{new string(' ', 400)}>\nendstream") + .Stream(3, "/Filter [/ASCIIHexDecode /FlateDecode]", hex + new string(' ', 400) + ">") .BuildClassic(rootNumber: 1); - var header = file.AsSpan().IndexOf("3 0 obj"u8); - var data = file.AsSpan().IndexOf(Encoding.ASCII.GetBytes(hex)); - var bodyEnd = data + (2 * (compressed.Length - ChecksumLength)); - var cut = bytesPastTheBody < 0 ? bodyEnd + (2 * -bytesPastTheBody) : data + hex.Length + bytesPastTheBody; - var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = cut - header } }; + var bodyEnd = 2 * (compressed.Length - ChecksumLength); + var cut = bytesPastTheBody < 0 ? bodyEnd + (2 * -bytesPastTheBody) : hex.Length + bytesPastTheBody; + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxStreamLength = cut } }; using var document = PdfDocument.Open(file, options); var decoded = document.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(document.Diagnostics); decoded.Length.Should().Be(ContentStream(200).Length); - document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Diagnostics.Select(d => d.Code).Should().Equal(PdfDiagnosticCodes.LimitStreamLength); } [Theory] @@ -1951,10 +1938,11 @@ private static byte[] CompressRaw(byte[] data) } /// - /// Encodes as the PDF variant of LZW does: a clear first, the end-of-data code last, and the code width - /// growing one code early when is 1. + /// Encodes as the PDF variant of LZW does: a clear first, the end-of-data code last unless + /// says not to, and the code width growing one code early when + /// is 1. /// - private static byte[] LzwEncode(byte[] data, int earlyChange = 1) + internal static byte[] LzwEncode(byte[] data, int earlyChange = 1, bool endCode = true) { var table = new Dictionary(StringComparer.Ordinal); for (var i = 0; i < 256; i++) @@ -1989,12 +1977,17 @@ private static byte[] LzwEncode(byte[] data, int earlyChange = 1) } codes.Add((table[current], bits)); - codes.Add((257, bits)); + + if (endCode) + { + codes.Add((257, bits)); + } + return Pack(codes); } /// Packs LZW codes nine bits each, most significant bit first, as the filter reads them. - private static byte[] NineBitCodes(params int[] codes) => Pack([.. codes.Select(code => (code, 9))]); + internal static byte[] NineBitCodes(params int[] codes) => Pack([.. codes.Select(code => (code, 9))]); private static byte[] Pack(List<(int Code, int Bits)> codes) { diff --git a/tests/AdCodicem.Pdf.Tests/FilterTests.cs b/tests/AdCodicem.Pdf.Tests/FilterTests.cs index e31d6c6e..4e1fd2ff 100644 --- a/tests/AdCodicem.Pdf.Tests/FilterTests.cs +++ b/tests/AdCodicem.Pdf.Tests/FilterTests.cs @@ -26,6 +26,37 @@ public class FilterTests 0x49, 0x2D, 0x2A, 0x56, 0xF4, 0x20, 0x4A, 0x08, 0x00, ]; + [Theory] + [InlineData(4, true)] + [InlineData(5, true)] + [InlineData(6, false)] + public void A_group_ascii85_data_that_ends_where_the_file_s_may_not_parts_is_dropped_and_counted_against_the_bound(int maxLength, bool limited) + { + // "5sdq," is ABCD, and "77I" the part of the group EF's: data the reader cut, or whose end a guard hides, may end + // inside the group, whose bytes the missing digits decide. They are not written, but the whole data holds them, + // and a bound they do not fit within is reached (#224): with no room for them, with room for one of the two, not + // with room for both. + var decoded = Ascii85Filter.Decode("5sdq,77I"u8, out var reached, out _, maxLength, cut: true); + + decoded.Should().Equal("ABCD"u8.ToArray()); + reached.Should().Be(limited); + Ascii85Filter.Decode("5sdq,77I"u8, out _, out _, 6).Should().Equal("ABCDEF"u8.ToArray()); + } + + [Theory] + [InlineData(2, true)] + [InlineData(3, false)] + public void A_digit_asciihex_data_that_ends_where_the_file_s_may_not_leaves_alone_is_dropped_and_counted_against_the_bound(int maxLength, bool limited) + { + // The last digit's byte, which the next digit decides, is not written, but the whole data holds it, and a bound it + // does not fit within is reached (#224). + var decoded = AsciiHexFilter.Decode("41424"u8, out var reached, out _, maxLength, cut: true); + + decoded.Should().Equal("AB"u8.ToArray()); + reached.Should().Be(limited); + AsciiHexFilter.Decode("41424"u8, out _, out _, 3).Should().Equal([0x41, 0x42, 0x40]); + } + [Fact] public void Decodes_a_flate_stream() { @@ -1197,25 +1228,27 @@ public void Gives_a_dictionary_of_parameters_beside_a_chain_to_each_of_its_steps } [Theory] - [InlineData("/Filter /DCTDecode", true)] - [InlineData("/Filter [/ASCIIHexDecode /DCTDecode]", true)] - [InlineData("/Filter [/ASCIIHexDecode /JPXDecode /FlateDecode]", true)] - [InlineData("/Filter [/MadeUpDecode /DCTDecode]", false)] - [InlineData("/Filter [5 /DCTDecode]", false)] - [InlineData("/Filter (DCTDecode)", false)] - [InlineData("/Filter [/ASCIIHexDecode /FlateDecode]", false)] - [InlineData("/Filter null", false)] - [InlineData("", false)] - public void Says_a_stream_is_an_image_only_when_decoding_it_stops_at_an_image_filter(string entries, bool image) + [InlineData("/Filter /DCTDecode", true, false)] + [InlineData("/Filter [/ASCIIHexDecode /DCTDecode]", true, false)] + [InlineData("/Filter [/ASCIIHexDecode /JPXDecode /FlateDecode]", true, false)] + [InlineData("/Filter [/FlateDecode /DCTDecode]", true, true)] + [InlineData("/Filter [/MadeUpDecode /DCTDecode]", false, false)] + [InlineData("/Filter [5 /DCTDecode]", false, false)] + [InlineData("/Filter (DCTDecode)", false, false)] + [InlineData("/Filter [/ASCIIHexDecode /FlateDecode]", false, true)] + [InlineData("/Filter null", false, false)] + [InlineData("", false, false)] + public void Says_a_stream_is_an_image_only_when_decoding_it_stops_at_an_image_filter(string entries, bool image, bool readsParameters) { // Decoding stops at the first filter it does not know, or at an element that names none: the data that comes - // back is not the image a filter after it names. Telling it reads no parameter. + // back is not the image a filter after it names. Telling it reads what decoding reads before it, and no more: the + // parameters of a Flate or LZW step before the chain ends, a value of which a guard cut ends it there (#263). var source = Substitute.For(); var dictionary = new PdfObjectParser(Encoding.ASCII.GetBytes($"<< {entries} /DecodeParms 9 0 R >>"), source: source).ParseObject().AsDictionary().Required(); var stream = new PdfStream(dictionary, PdfStreamData.FromMemory("00>"u8.ToArray())); stream.HasImageFilter().Should().Be(image); - source.ReceivedCalls().Should().BeEmpty(); + source.ReceivedCalls().Should().HaveCount(readsParameters ? 1 : 0); } [Fact] diff --git a/tests/AdCodicem.Pdf.Tests/GuardCutValueTests.cs b/tests/AdCodicem.Pdf.Tests/GuardCutValueTests.cs new file mode 100644 index 00000000..c1bebbb6 --- /dev/null +++ b/tests/AdCodicem.Pdf.Tests/GuardCutValueTests.cs @@ -0,0 +1,4098 @@ +using System.Globalization; +using System.IO.Compression; +using System.Text; +using AdCodicem.Pdf.Diagnostics; +using AdCodicem.Pdf.Documents; +using AdCodicem.Pdf.IO; +using AdCodicem.Pdf.IO.Filters; +using AdCodicem.Pdf.IO.XRef; +using AdCodicem.Pdf.Objects; +using AdCodicem.Pdf.Validation; +using AdCodicem.Pdf.Validation.Rules; + +namespace AdCodicem.Pdf.Tests; + +/// +/// A stream's /Length, /Filter or /DecodeParms that leads to an object one of the reader's guards cut, +/// or left out of the index: what the cut left of it — null, a token it parted, a whole value it may yet go on past — is +/// the guard's, which reported it, and not a value the file gave (#224, #263). +/// +public class GuardCutValueTests +{ + private const string Page = "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595 842] /Contents 4 0 R /Resources << >> >>"; + + private static readonly PdfReaderOptions ObjectGuard = new() { Limits = PdfReaderLimits.Default with { MaxObjectLength = 1024 } }; + + [Theory] + [InlineData("before the value", "0 0 m")] + [InlineData("after the value", "0 0 m")] + [InlineData("after a wrong value", "0 0 m")] + [InlineData("inside the value", "0 0 m 10 10 l S")] + public void A_length_a_guard_cut_is_not_judged_and_the_data_is_found_by_its_endstream(string cut, string data) + { + // Object 20, the /Length, runs past MaxObjectLength: before its value, which read as null; after it, or after a + // wrong one; or inside "15", which read as 1. Each was reported as the file's fault but the one after the right + // value — "holds null", "declared 4 bytes", "declared 1 bytes" (#224). The data is found by its endstream, and only + // the guard is reported. + var twenty = cut switch + { + "before the value" => new string(' ', 3000) + "5", + "after the value" => "5" + new string(' ', 3000), + "after a wrong value" => "4" + new string(' ', 3000), + _ => new string(' ', 1024 - "20 0 obj\n".Length - 1) + "15" + new string(' ', 100), + }; + var file = ContentLengthIn(twenty, data); + using var document = PdfDocument.Open(file, ObjectGuard); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.GetRawBytes().ToArray().Should().Equal(Encoding.ASCII.GetBytes(data)); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.TryGetStreamLengthFault(4, out _).Should().BeFalse(); + document.Reader.IsCutAtLimit(20).Should().BeTrue(); + + // Read whole, a wrong value is the file's. + using var control = PdfDocument.Open(file); + control.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal(Encoding.ASCII.GetBytes(data)); + control.Diagnostics.Select(entry => entry.Message).Should().Equal( + cut == "after a wrong value" ? ["The stream declared 4 bytes but ended after 5."] : []); + } + + [Fact] + public void A_length_a_guard_cut_leaves_a_missing_endstream_the_file_s_fault() + { + // The stream lost its endstream: its own endobj ends its data, which is the file's to answer for, whatever its + // length — the guard cut that, and is not judged. + var file = ContentLengthIn(new string(' ', 3000) + "5", "0 0 m", endStream: false); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal("0 0 m"u8.ToArray()); + var report = document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated).Which; + report.Message.Should().StartWith( + "The stream's /Length names object 20 0, which one of the reader's limits cut, and no endstream follows its data"); + report.Message.Should().EndWith("its object's endobj ends it, after 5 bytes."); + } + + [Fact] + public void A_length_an_object_stream_s_guard_left_unread_is_not_judged() + { + // Object 20, the /Length, is a member of object stream 11 past what MaxDecodedStreamLength lets it decode to: it + // read as null, and was reported as one that could not be read (#263). It is the guard's. + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [4] = (1, file.Object(4, "<< /Length 20 0 R >>\nstream\n0 0 m\nendstream"), 0), + [11] = (1, FlateObjectStream(file, 11, [(21, "(" + new string('x', 300) + ")"), (20, "5")]), 0), + [21] = (2, 11, 0), + [20] = (2, 11, 1), + }; + file.CrossReferenceStream(12, 22, rows, "/Root 1 0 R"); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 100 } }; + using var document = PdfDocument.Open(file.Bytes, options); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal("0 0 m"u8.ToArray()); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + document.Reader.TryGetStreamLengthFault(4, out _).Should().BeFalse(); + } + + [Fact] + public void A_length_whose_reference_leads_to_an_object_a_guard_left_out_of_the_index_is_not_judged() + { + // Object 20, the /Length, holds a reference to object 21, which only an older section places, which + // MaxXRefSectionCount leaves unread — the newer table numbers no row 21: 21 is unknown, and 20 led to null, + // reported as no length. + var file = new HandWrittenFile(); + var older = new Dictionary { [21] = file.Object(21, "5") }; + var first = file.Table(22, older, "/Size 22"); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [4] = file.Object(4, "<< /Length 20 0 R >>\nstream\n0 0 m\nendstream"), + [20] = file.Object(20, "21 0 R"), + }; + file.Table(21, rows, string.Create(CultureInfo.InvariantCulture, $"/Size 22 /Root 1 0 R /Prev {first}")); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxXRefSectionCount = 1 } }; + using var document = PdfDocument.Open(file.Bytes, options); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal("0 0 m"u8.ToArray()); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitXRefSectionCount); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_deferred_length_a_guard_cut_leaves_a_stream_with_no_endstream_the_guard_s(bool chained) + { + // Stream 7 lost its endstream, and its /Length, object 5 — or object 6, which object 5 refers to —, which the newer + // table places, runs past MaxObjectLength: the section was called malformed because no section placed the length, + // and the index rebuilt. Without the guard the length is read, and the section with it: the section is the guard's. + var padded = new string(' ', 3000) + "56"; + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", + chained ? "6 0 R" : padded, + compressed: false, + predicted: false, + placed: true, + placedToo: chained ? new Dictionary { [6] = padded } : null); + var text = Encoding.Latin1.GetString(file); + var at = text.IndexOf("endstream", StringComparison.Ordinal); + Encoding.ASCII.GetBytes("endstrXam").CopyTo(file, at); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[1].CutByLimit.Should().BeTrue(); + document.Reader.Structure.Sections[1].Fault.Should().EndWith( + string.Create(CultureInfo.InvariantCulture, $"the reference {(chained ? 6 : 5)} 0 R, which one of the reader's limits cut")); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().BeFalse(); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData("21 0 R")] + [InlineData("20 0 R")] + public void A_missing_endstream_is_the_file_s_fault_when_a_guard_left_the_length_out_of_the_index(string length) + { + // Object 4's /Length leads to object 21, which only an older section places, which MaxXRefSectionCount leaves unread: + // the length is unknown, and not judged, but the endstream the stream lacks was not reported either (#224). + var file = new HandWrittenFile(); + var older = new Dictionary { [21] = file.Object(21, "5") }; + var first = file.Table(22, older, "/Size 22"); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [4] = file.Object(4, "<< /Length " + length + " >>\nstream\n0 0 m"), + [20] = file.Object(20, "21 0 R"), + }; + file.Table(21, rows, string.Create(CultureInfo.InvariantCulture, $"/Size 22 /Root 1 0 R /Prev {first}")); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxXRefSectionCount = 1 } }; + using var document = PdfDocument.Open(file.Bytes, options); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal("0 0 m"u8.ToArray()); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated).Which.Message.Should().StartWith( + "The stream's /Length names object 21 0, which one of the reader's limits left out of the index, and no endstream follows its data"); + document.Reader.TryGetStreamLengthFault(4, out var fault).Should().BeTrue(); + fault.Form.Should().Be(StreamLengthForm.ObjectUnindexed); + fault.Reference.Should().Be(new PdfObjectId(21)); + } + + [Theory] + [InlineData("cut", "endstream")] + [InlineData("left out", "endstream")] + [InlineData("cut", "inner stream")] + [InlineData("left out", "inner stream")] + [InlineData("cut", "none")] + [InlineData("left out", "none")] + public void Data_a_header_stops_where_a_guard_hid_the_length_is_the_guard_s(string guard, string end) + { + // Object 4 embeds a PDF file, whose data spells "1 0 obj" — and holds a stream of its own, whose endstream comes + // after —, and its /Length, object 10, is one MaxObjectLength cuts or MaxIndexEntries leaves out of the index. The + // search for its endstream stops at the header the data spells: the file was said to lack the endstream it holds, + // or the data was taken to the embedded stream's. What decides is the length the guard hid: the data up to the + // header is the guard's, cut by it, the object's endobj unjudged, and nothing said of the file, whatever follows + // (#224). + var embedded = "%PDF-1.4\n1 0 obj\n<< /Type /Catalog >>\nendobj\n" + + (end == "inner stream" ? "2 0 obj\n<< /Length 5 >>\nstream\nhello\nendstream\nendobj\n" : string.Empty) + + "trailer\n<< /Root 1 0 R >>\n%%EOF"; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, "<< /Type /EmbeddedFile /Length 10 0 R >>\nstream\n" + embedded + (end == "none" ? string.Empty : "\nendstream")), + [6] = file.Object(6, "(six)"), + [10] = file.Object(10, (guard == "cut" ? new string(' ', 2000) : string.Empty) + embedded.Length.ToString(CultureInfo.InvariantCulture)), + }; + file.Table(11, rows, "/Size 11 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 8 }; + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.GetRawBytes().ToArray().Should().Equal("%PDF-1.4\n"u8.ToArray()); + stream.Data.CutByGuard.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + document.Reader.TryGetStreamLengthFault(4, out _).Should().BeFalse(); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectEndObjMissing).Should().BeFalse(); + } + + [Theory] + [InlineData("cut", "next object")] + [InlineData("left out", "next object")] + [InlineData("cut", "end of file")] + [InlineData("left out", "end of file")] + public void A_missing_endstream_no_header_stops_the_search_at_is_the_file_s_where_a_guard_hid_the_length(string guard, string stop) + { + // Object 4, whose /Length a guard hid, lost its endstream and its endobj, and its data spells no header: the search + // stops at the next object the index places, or at the end of the file. Nothing the guard hides would say where the + // data ends otherwise: the missing endstream is the file's, as read whole (#224). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + + if (stop == "end of file") + { + WriteOthers(); + } + + rows[4] = file.Position; + file.Raw("4 0 obj\n<< /Length 10 0 R >>\nstream\nABC\n"); + + if (stop == "next object") + { + WriteOthers(); + } + + file.Table(11, rows, "/Size 11 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 8 }; + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Data.CutByGuard.Should().BeFalse(); + document.Reader.TryGetStreamLengthFault(4, out var fault).Should().BeTrue(); + fault.Form.Should().Be(guard == "cut" ? StreamLengthForm.ObjectCut : StreamLengthForm.ObjectUnindexed); + var code = stop == "next object" ? PdfDiagnosticCodes.StreamLengthInvalid : PdfDiagnosticCodes.StreamTruncated; + document.Diagnostics.Should().ContainSingle(entry => entry.Code == code).Which.Message.Should().StartWith( + $"The stream's /Length names object 10 0, which one of the reader's limits {(guard == "cut" ? "cut" : "left out of the index")}, " + + $"and no endstream follows its data before the {(stop == "next object" ? "next object" : "end of the file")}"); + + if (stop == "next object") + { + stream.GetRawBytes().ToArray().Should().Equal("ABC\n"u8.ToArray()); + } + + void WriteOthers() + { + rows[6] = file.Object(6, "(six)"); + rows[10] = file.Object(10, (guard == "cut" ? new string(' ', 2000) : string.Empty) + "3"); + } + } + + [Theory] + [InlineData("cut")] + [InlineData("left out")] + public void A_stream_a_guard_hid_the_length_of_ends_at_its_own_endobj_before_a_header(string guard) + { + // Object 4, whose /Length a guard hid, lost its endstream; its own endobj, then an object no row places, a stream + // with an endstream, follow: the endobj, which comes before that object's header, ends the data before any header + // can, and the missing endstream is the file's (#224). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, "<< /Length 10 0 R >>\nstream\nABC"), + }; + file.Object(30, "<< /Length 3 >>\nstream\nxyz\nendstream"); + rows[6] = file.Object(6, "(six)"); + rows[10] = file.Object(10, (guard == "cut" ? new string(' ', 2000) : string.Empty) + "3"); + file.Table(11, rows, "/Size 11 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 8 }; + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.GetRawBytes().ToArray().Should().Equal("ABC"u8.ToArray()); + stream.Data.CutByGuard.Should().BeFalse(); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_deferred_length_a_guard_cut_is_not_judged_once_the_chain_is_read(bool inside) + { + // The cross-reference stream's /Length, object 5, which the newer table places, runs past MaxObjectLength before its + // value, or inside "56": read once the chain is, it was said to hold null, or to declare 5 bytes (#224). + var five = inside ? new string(' ', 1024 - "5 0 obj\n".Length - 1) + "56" + new string(' ', 100) : new string(' ', 3000) + "56"; + var file = ChainFiles.UnderAnUpdate("/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", five, compressed: false, predicted: false, placed: true); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections.Should().OnlyContain(section => section.State == IO.XRef.XRefSectionState.Read); + } + + /// The entries of object stream 10 that lead to object 20, and what object 20 holds. + private static readonly (string Entries, string Twenty)[] Values = + [ + ("/Filter 20 0 R", "/FlateDecode"), + ("/Filter [20 0 R]", "/FlateDecode"), + ("/Filter /FlateDecode /DecodeParms 20 0 R", "<< /Predictor 1 >>"), + ("/Filter /FlateDecode /DecodeParms [20 0 R]", "<< /Predictor 1 >>"), + ("/Filter /FlateDecode /DecodeParms << /Predictor 20 0 R >>", "1"), + ]; + + public static TheoryData Cuts + { + get + { + var data = new TheoryData(); + + foreach (var (entries, twenty) in Values) + { + foreach (var cut in new[] { "before the value", "inside the value", "member past the data", "left out of the index" }) + { + data.Add(entries, twenty, cut); + } + } + + return data; + } + } + + [Theory] + [MemberData(nameof(Cuts))] + public void An_object_stream_whose_filter_values_a_guard_cut_is_hidden_and_its_members_taken_as_read(string entries, string twenty, string cut) + { + // The catalog is packed in object stream 10, whose /Filter, /DecodeParms or a value of its parameters leads to object + // 20, which a guard cut — before its value, or inside it: "/Flate" of "/FlateDecode" —, left past the data of an + // object stream it decoded, or out of the index. The stream decoded as no filter, or as "/Flate", unsupported, and + // its header read as stray bytes: the catalog was looked for elsewhere, and a sound file judged broken (#263). The + // filters end at the guard's value, the stream is left encoded and hidden, and its members are the guard's. + var (file, options) = PackedCatalog(entries, twenty, cut); + using var document = PdfDocument.Open(file, options); + + document.Reader.Structure.RootCutByGuard.Should().BeTrue(); + document.Diagnostics.Should().NotContain(entry => + entry.Code == PdfDiagnosticCodes.ObjectStreamUnreadable || entry.Code == PdfDiagnosticCodes.FilterUnsupported || + entry.Code == PdfDiagnosticCodes.TrailerRootRecovered); + var report = new PdfValidator().Validate(document); + report.Contains(PdfValidationRuleIds.FileRootInvalid).Should().BeFalse(); + report.Contains(PdfValidationRuleIds.XRefObjectStreamBroken).Should().BeFalse(); + report.Contains(PdfValidationRuleIds.XRefCheckedInPart).Should().BeTrue(); + + using var control = PdfDocument.Open(file); + control.Catalog.Should().NotBeNull(); + control.Diagnostics.Should().BeEmpty(); + } + + [Fact] + public void A_content_stream_whose_filter_a_guard_cut_comes_back_as_the_filters_before_it_decode_it() + { + // The page's content stream decodes through /ASCIIHexDecode, then object 20, which MaxObjectLength cuts before + // "/ASCIIHexDecode": its data comes back as the first filter decodes it, as past MaxFilterChainLength, and nothing is + // said but the guard — once, it decoded as if the chain ended there, in silence (#263). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [4] = file.Object(4, "<< /Filter [/ASCIIHexDecode 20 0 R] /Length 23 >>\nstream\n333032303330323036443E>\nendstream"), + [20] = file.Object(20, new string(' ', 3000) + "/ASCIIHexDecode"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + Encoding.ASCII.GetString(stream.Decode().Span).Should().Be("302030206D>"); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + + using var control = PdfDocument.Open(file.Bytes); + Encoding.ASCII.GetString(control.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().Span).Should().Be("0 0 m"); + control.Diagnostics.Should().BeEmpty(); + } + + [Fact] + public void A_guard_a_filter_value_reaches_as_it_loads_is_not_the_stream_s_own_cut() + { + // Object stream 10's /Filter is object 20, read whole from object stream 11, whose own decoding reaches + // MaxDecodedStreamLength on a later member; 10's /First lies past its data. The guard 20's load reached was taken for + // 10's: its fault went unreported, and its catalog was taken as cut, though the validator, reading 20 from the cache, + // judged it broken (#263). The guard is 11's: 10's fault is the file's, reported by both. + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595 842] /Resources << >> >>"), 0), + [11] = (1, FlateObjectStream(file, 11, [(20, "/FlateDecode"), (21, "(" + new string('x', 300) + ")")]), 0), + [20] = (2, 11, 0), + [21] = (2, 11, 1), + [10] = (1, FlateObjectStream(file, 10, [(1, "<< /Type /Catalog /Pages 2 0 R >>")], "/First 999 ", filter: "20 0 R"), 0), + [1] = (2, 10, 0), + }; + file.CrossReferenceStream(12, 22, rows, "/Root 1 0 R"); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 100 } }; + using var document = PdfDocument.Open(file.Bytes, options); + + document.Reader.Structure.RootCutByGuard.Should().BeFalse(); + document.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.ObjectStreamUnreadable); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefObjectStreamBroken).Should().BeTrue(); + } + + [Theory] + [InlineData("Filter")] + [InlineData("DecodeParms")] + [InlineData("Columns")] + [InlineData("W")] + [InlineData("W element")] + [InlineData("Index")] + [InlineData("Index element")] + [InlineData("Size")] + public void A_cross_reference_stream_value_a_guard_cut_ends_the_chain_as_the_guard_s(string key) + { + // Stream 7's /Filter, /DecodeParms, /W or /Index is object 5, which the newer table places and MaxObjectLength cuts + // after its value. The chain said no section placed it, rebuilt the index and judged the section malformed. The + // section is the guard's: the chain ends there, nothing is rebuilt, and the guard is reached once the chain is read + // — thrown at opening when the caller asks for it (#263). + var file = ChainFiles.UnderAnUpdate(key, placed: true, padding: 3000); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[1].CutByLimit.Should().BeTrue(); + document.Reader.Structure.Sections[1].Fault.Should().EndWith("the reference 5 0 R, which one of the reader's limits cut"); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().BeFalse(); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + /// + /// A page whose content stream, object 4, holds BT ET, its /Length object 20, which holds 5 after 2,000 + /// spaces where is cut; its endstream, and its endobj where + /// , followed by : object 6, which the table places, or which no row + /// places — at its header, or on the end-of-line before it —; the table; the startxref after the table, which + /// comes before object 4; or the end of the file, past the one the table's %%EOF marks. + /// + private static byte[] StreamFollowedBy(string guard, string follower, bool endObj) + { + var four = "4 0 obj\n<< /Length 20 0 R >>\nstream\nBT ET\nendstream\n" + (endObj ? "endobj\n" : string.Empty); + var late = follower is "startxref" or "end of file"; + + if (!late) + { + return Encoding.Latin1.GetBytes(Text(0)); + } + + // The table's row for object 4 gives where it lies after the table: an offset as many digits long wherever it lies. + var first = Text(0); + var at = follower == "startxref" ? first.LastIndexOf("startxref", StringComparison.Ordinal) : first.Length; + return Encoding.Latin1.GetBytes(Text(at).Insert(at, four)); + + string Text(long at) + { + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + "5"), + }; + + if (late) + { + rows[4] = at; + } + else + { + rows[4] = file.Position; + file.Raw(four); + + if (follower != "xref") + { + var six = file.Object(6, "(six)"); + + // A row may name the white space before the header, where the reader reads the object all the same. + if (follower is "next object" or "next object past white space") + { + rows[6] = follower == "next object" ? six : six - 1; + } + } + } + + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + return Encoding.Latin1.GetString(file.Bytes); + } + } + + /// + /// A page whose content stream, object 4, holds , its /Length object 20, which holds + /// ; with no endstream unless . + /// + private static byte[] ContentLengthIn(string twenty, string data, bool endStream = true) + { + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [4] = file.Object(4, "<< /Length 20 0 R >>\nstream\n" + data + (endStream ? "\nendstream" : string.Empty)), + [20] = file.Object(20, twenty), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + return file.Bytes; + } + + /// + /// The catalog, packed in Flate object stream 10, whose dictionary holds , which lead to object + /// 20, holding ; and the reader's options under which a guard cuts object 20 as + /// says. + /// + private static (byte[] File, PdfReaderOptions Options) PackedCatalog(string entries, string twenty, string cut) + { + var file = new HandWrittenFile(version: "1.5"); + var older = -1L; + + if (cut == "left out of the index") + { + older = file.Table(23, new Dictionary { [20] = file.Object(20, twenty) }, "/Size 23"); + } + + var rows = new Dictionary + { + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595 842] /Resources << >> >>"), 0), + [1] = (2, 10, 0), + }; + var catalog = "<< /Type /Catalog /Pages 2 0 R >>"; + var data = HostileInputTests.Compress(Encoding.ASCII.GetBytes("1 0 " + catalog)); + rows[10] = (1, file.Object( + 10, + string.Create(CultureInfo.InvariantCulture, $"<< /Type /ObjStm /N 1 /First 4 {entries} /Length {data.Length} >>\nstream\n{Encoding.Latin1.GetString(data)}\nendstream")), + 0); + PdfReaderLimits limits; + + switch (cut) + { + case "before the value": + rows[20] = (1, file.Object(20, new string(' ', 3000) + twenty), 0); + limits = PdfReaderLimits.Default with { MaxObjectLength = 1024 }; + break; + + case "inside the value": + rows[20] = (1, file.Object(20, new string(' ', 1024 - "20 0 obj\n".Length - (twenty.Length / 2)) + twenty + new string(' ', 100)), 0); + limits = PdfReaderLimits.Default with { MaxObjectLength = 1024 }; + break; + + case "member past the data": + rows[11] = (1, FlateObjectStream(file, 11, [(21, "(" + new string('x', 300) + ")"), (20, twenty)]), 0); + rows[21] = (2, 11, 0); + rows[20] = (2, 11, 1); + limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 100 }; + break; + + default: + limits = PdfReaderLimits.Default with { MaxXRefSectionCount = 1 }; + break; + } + + // Left out of the index, object 20 is numbered by no row of the newest section, which writes a row for every number + // below its /Size. + var size = older < 0 ? 23 : 13; + file.CrossReferenceStream(12, size, rows, "/Root 1 0 R" + (older < 0 ? string.Empty : string.Create(CultureInfo.InvariantCulture, $" /Prev {older}"))); + return (file.Bytes, new PdfReaderOptions { Limits = limits }); + } + + /// + /// Writes object stream , holding compressed with Flate, or with the + /// filter names, its dictionary holding before its own. + /// + private static long FlateObjectStream( + HandWrittenFile file, int number, (int Number, string Body)[] members, string entries = "", string filter = "/FlateDecode") + { + var header = new StringBuilder(); + var contents = new StringBuilder(); + + foreach (var (member, body) in members) + { + header.Append(CultureInfo.InvariantCulture, $"{member} {contents.Length} "); + contents.Append(body).Append(' '); + } + + var data = HostileInputTests.Compress(Encoding.ASCII.GetBytes(header.ToString() + contents)); + var first = entries.Contains("/First", StringComparison.Ordinal) ? string.Empty : string.Create(CultureInfo.InvariantCulture, $" /First {header.Length}"); + return file.Object( + number, + string.Create( + CultureInfo.InvariantCulture, + $"<< /Type /ObjStm /N {members.Length}{first} {entries}/Filter {filter} /Length {data.Length} >>\nstream\n{Encoding.Latin1.GetString(data)}\nendstream")); + } + + [Fact] + public void A_missing_endstream_names_the_object_a_guard_cut_at_the_end_of_the_length_s_chain_and_keeps_no_length() + { + // Object 4's /Length, object 20, holds a reference to object 21, which MaxObjectLength cuts inside "15": the report + // of its missing endstream named object 20, read whole, and the fault kept the parted 1 as the declared length. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [4] = file.Object(4, "<< /Length 20 0 R >>\nstream\n0 0 m 10 10 l S"), + [20] = file.Object(20, "21 0 R"), + [21] = file.Object(21, new string(' ', 1024 - "21 0 obj\n".Length - 1) + "15" + new string(' ', 100)), + }; + file.Table(22, rows, "/Size 22 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal("0 0 m 10 10 l S"u8.ToArray()); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated) + .Which.Message.Should().StartWith("The stream's /Length names object 21 0, which one of the reader's limits cut"); + document.Reader.TryGetStreamLengthFault(4, out var fault).Should().BeTrue(); + fault.Form.Should().Be(StreamLengthForm.ObjectCut); + fault.Reference.Should().Be(new PdfObjectId(21)); + fault.Declared.Should().BeNull(); + } + + [Fact] + public void A_deferred_length_whose_reference_leads_to_an_object_a_guard_cut_is_not_judged_once_the_chain_is_read() + { + // The cross-reference stream's /Length, object 5, holds a reference to object 6, which the newer table places and + // MaxObjectLength cuts: the chain read object 5 alone, judged the length as holding null, and never reached the guard + // (#224). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", + "6 0 R", + compressed: false, + predicted: false, + placed: true, + placedToo: new Dictionary { [6] = new string(' ', 3000) + "56" }); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections.Should().OnlyContain(section => section.State == IO.XRef.XRefSectionState.Read); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Fact] + public void The_integer_a_guard_cut_a_deferred_length_after_is_where_rows_that_spell_an_endstream_are_taken_again_to() + { + // The update's stream indexes its /Length, object 4, and object 8 after rows that spell an endstream, and + // MaxObjectLength cuts object 4 after its 35: the section was said to lack the rows past the endstream, and object 8 + // read as the first revision wrote it. The 35 the cut left is where to look for the endstream, which confirms it + // (#224). + var file = ChainFiles.UpdateWithIndirectLength(spellingEndStream: true, four: "{length}" + new string(' ', 3000)); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.GetObject(new PdfObjectId(8)).Should().BeOfType().Which.ToText().Should().Be("new"); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.WasRepaired.Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().BeFalse(); + } + + [Theory] + [InlineData(false, 0)] + [InlineData(true, 0)] + [InlineData(false, 3)] + [InlineData(true, 3)] + public void The_integer_a_guard_cut_a_length_a_newer_section_places_after_is_where_spelled_rows_are_taken_again_to( + bool chained, int lengthOff) + { + // Stream 7's /Length leads to an object the newer table places — object 5 itself, or object 9 through object 4, + // which only stream 7's own row places — and MaxObjectLength cuts it after its value. The value the cut left is where + // to look for the endstream, as for an object the section's own row places: confirmed, the data is taken again to it + // and every row read; otherwise the rows past the endstream they spell are the guard's (#224). + var file = ChainFiles.UpdatePlacingAPaddedLength(chained, lengthOff: lengthOff, lengthFirst: true); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[1].CutByLimit.Should().Be(lengthOff != 0); + document.Reader.Structure.Sections[1].Incomplete.Should().BeFalse(); + (document.GetObject(new PdfObjectId(3)).AsDictionary() is not null).Should().Be(lengthOff == 0); + } + + [Fact] + public void Rows_data_taken_again_to_an_integer_a_guard_parted_still_lacks_are_the_guard_s() + { + // MaxObjectLength parts stream 7's /Length, 210, after its 21, which the rows' second spelled endstream confirms: the + // data taken again to it holds 3 rows of the 30 the section declares, and the 27 it lacks were the file's, the + // section incomplete and the index rebuilt. The integer is only a place to look at, the length that decides the + // guard's: what it leaves unread is the guard's (#224). + var file = ChainFiles.UpdatePlacingAPartedLength(); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + whole.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[1].CutByLimit.Should().BeTrue(); + document.Reader.Structure.Sections[1].Incomplete.Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().BeFalse(); + } + + [Theory] + [InlineData("cut before the value")] + [InlineData("cut inside the value")] + [InlineData("left out")] + public void An_endstream_the_data_spells_where_a_guard_hid_the_length_ends_data_that_is_the_guard_s(string guard) + { + // The content stream's data spells endstream inside a string, which nothing that would prove it the stream's own + // follows — an object the index places, the end of the file, xref or startxref —; its own endstream and endobj come + // after the 40 bytes. Read whole, the length passes the spelled one. Where a guard cut the + // /Length, or left it out of the index, the search stops there, and the object was said to lack its endobj: the + // length that tells the two apart is the guard's, and so is the data up to it (#224). + const string Data = "BT /F1 12 Tf 72 712 Td (endstream) Tj ET"; + var (twenty, options) = guard switch + { + "cut before the value" => (new string(' ', 2000) + "40", ObjectGuard), + "cut inside the value" => (new string(' ', 1024 - "20 0 obj\n".Length - 1) + "40" + new string(' ', 100), ObjectGuard), + _ => ("40", new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxIndexEntries = 20 } }), + }; + var file = ContentLengthIn(twenty, Data); + + using (var whole = PdfDocument.Open(file)) + { + whole.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal(Encoding.ASCII.GetBytes(Data)); + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, options); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.GetRawBytes().ToArray().Should().Equal(Encoding.ASCII.GetBytes("BT /F1 12 Tf 72 712 Td (")); + stream.Data.CutByGuard.Should().BeTrue(); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + document.Reader.TryGetStreamLengthFault(4, out _).Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "left out" ? PdfDiagnosticCodes.LimitIndexEntries : PdfDiagnosticCodes.LimitObject); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectEndObjMissing).Should().BeFalse(); + } + + [Theory] + [InlineData("cut", false, true)] + [InlineData("left out", false, true)] + [InlineData("cut", true, true)] + [InlineData("left out", true, true)] + [InlineData("cut", true, false)] + [InlineData("left out", true, false)] + public void An_endstream_the_next_object_follows_where_a_guard_hid_the_length_lacks_its_endobj_as_the_file_s_fault(string guard, bool flate, bool endOfLine) + { + // Object 4's endstream lost its endobj, and object 5, which the table places, follows it directly. No endstream the + // data spells can be followed by an object the index places, so the one found is the stream's own, whatever the + // guard hid: what the object lacks is the file's, as read whole. So is the end of its Flate data, which lost its + // last bytes, where the endstream follows it directly; past an end-of-line, which only the hidden length says is + // the data's or not, its end is the guard's to hide, and the run-out says nothing of the file (#224). + var data = "BT /F1 12 Tf 72 712 Td (Hello) Tj ET"; + + if (flate) + { + using var output = new MemoryStream(); + + using (var zlib = new ZLibStream(output, CompressionLevel.NoCompression, leaveOpen: true)) + { + zlib.Write(Encoding.ASCII.GetBytes(data)); + } + + data = Encoding.Latin1.GetString(output.ToArray()[..(endOfLine ? ^6 : ^4)]); + } + + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + data.Length.ToString(CultureInfo.InvariantCulture)), + [4] = file.Position, + }; + file.Raw($"4 0 obj\n<< /Length 20 0 R{(flate ? " /Filter /FlateDecode" : string.Empty)} >>\nstream\n{data}{(endOfLine ? "\n" : string.Empty)}endstream\n"); + rows[5] = file.Object(5, "(five)"); + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + _ = stream.Decode(); + + stream.GetRawBytes().ToArray().Should().Equal(Encoding.Latin1.GetBytes(data)); + stream.Data.CutByGuard.Should().BeFalse(); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + flate && !endOfLine + ? [guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries, PdfDiagnosticCodes.FilterFailed] + : [guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries]); + stream.Data.EndHidden.Should().Be(endOfLine); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectEndObjMissing).Should().BeTrue(); + } + + [Theory] + [InlineData("cut", "\n")] + [InlineData("left out", "\n")] + [InlineData("cut", "\r\n")] + public void Flate_data_whose_last_byte_is_a_carriage_return_before_a_proven_endstream_is_no_fault_of_the_file(string guard, string endOfLine) + { + // Content stream 4's Flate data ends in 0x0D, its checksum's last byte, then an end-of-line and its endstream, which the + // next object proves its own. The search leaves a carriage return out before a line feed, the length that would + // say it is the data's hidden by the guard: the data lacked its last byte, and its Flate data, short of its checksum, + // was reported as the file's fault, which read whole it is not (#224). + byte[] zlib = []; + + for (var i = 0; zlib.Length == 0 || zlib[^1] != 0x0D; i++) + { + using var output = new MemoryStream(); + + using (var stream = new ZLibStream(output, CompressionLevel.Optimal, leaveOpen: true)) + { + stream.Write(Encoding.ASCII.GetBytes(Invariant($"BT /F1 12 Tf ({i}) Tj ET"))); + } + + zlib = output.ToArray(); + } + + var data = Encoding.Latin1.GetString(zlib); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + data.Length.ToString(CultureInfo.InvariantCulture)), + [4] = file.Object(4, $"<< /Length 20 0 R /Filter /FlateDecode >>\nstream\n{data}{endOfLine}endstream"), + [5] = file.Object(5, "(five)"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + + using (var whole = PdfDocument.Open(file.Bytes)) + { + _ = whole.GetObject(new PdfObjectId(4)).AsStream().Required().Decode(); + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + var content = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + content.Decode().ToArray().Should().Equal(ZLibDecompress(zlib)); + content.Data.CutByGuard.Should().BeFalse(); + content.Data.EndHidden.Should().BeTrue(); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + } + + [Theory] + [InlineData("cut", true)] + [InlineData("left out", true)] + [InlineData("cut", false)] + public void Data_that_ends_in_white_space_right_before_a_proven_endstream_ends_where_the_guard_hides(string guard, bool space) + { + // Content stream 4's hexadecimal data lacks its > and ends in a space, the endstream right after it, which the next + // object proves its own: whether the space is the data's only the hidden length says, and the missing > is no fault + // the file shows where the data ends. With no space, the data ends where the file's does, and the > it lacks is the + // file's fault (#224). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + (space ? "11" : "10")), + [4] = file.Object(4, $"<< /Length 20 0 R /Filter /ASCIIHexDecode >>\nstream\n48656C6C6F{(space ? " " : string.Empty)}endstream"), + [5] = file.Object(5, "(five)"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + + using (var whole = PdfDocument.Open(file.Bytes)) + { + _ = whole.GetObject(new PdfObjectId(4)).AsStream().Required().Decode(); + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.FilterFailed); + } + + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + var content = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + var reached = guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries; + + content.Decode().ToArray().Should().Equal("Hello"u8.ToArray()); + content.Data.CutByGuard.Should().BeFalse(); + content.Data.EndHidden.Should().Be(space); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(space ? [reached] : [reached, PdfDiagnosticCodes.FilterFailed]); + } + + [Theory] + [InlineData("cut", "(world)", "world")] + [InlineData("left out", "(world)", "world")] + [InlineData("cut", "(w)", "w")] + [InlineData("left out", "(w)", "w")] + public void An_ascii85_object_stream_that_lacks_its_end_marker_before_a_proven_endstream_decodes_as_read_whole( + string guard, string last, string text) + { + // Object stream 10's ASCII85 data lacks its ~> and ends inside a group, then an end-of-line and its endstream, which + // the next object proves its own, under a /Length a guard hid. White space parts no group: the data decodes as read + // whole. Its last group was dropped, as one a cut parts, and the member it shortened charged to the file — (wor), or + // (w) past the 16 bytes left, the object stream broken (#224). + var encoded = Ascii85Text(Encoding.Latin1.GetBytes("6 0 7 8 (hello) " + last)); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [20] = (1, file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + encoded.Length.ToString(CultureInfo.InvariantCulture)), 0), + [10] = (1, file.Object(10, $"<< /Type /ObjStm /N 2 /First 8 /Length 20 0 R /Filter /ASCII85Decode >>\nstream\n{encoded}\nendstream"), 0), + [11] = (1, file.Object(11, "(eleven)"), 0), + [6] = (2, 10, 0), + [7] = (2, 10, 1), + }; + file.CrossReferenceStream(21, 22, rows, "/Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.GetObject(new PdfObjectId(7)).Should().BeOfType().Which.ToText().Should().Be(text); + } + + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + + document.GetObject(new PdfObjectId(7)).Should().BeOfType().Which.ToText().Should().Be(text); + document.GetObject(new PdfObjectId(10)).AsStream().Required().Data.EndHidden.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefObjectStreamBroken).Should().BeFalse(); + } + + /// Writes in ASCII85, whole groups but the last, with no z and no ~>. + private static string Ascii85Text(ReadOnlySpan bytes) + { + var text = new StringBuilder(); + Span digits = stackalloc char[5]; + + for (var i = 0; i < bytes.Length; i += 4) + { + var count = Math.Min(4, bytes.Length - i); + uint value = 0; + + for (var j = 0; j < 4; j++) + { + value = (value << 8) | (j < count ? bytes[i + j] : 0u); + } + + for (var j = 4; j >= 0; j--) + { + digits[j] = (char)('!' + (value % 85)); + value /= 85; + } + + text.Append(digits[..(count + 1)]); + } + + return text.ToString(); + } + + [Theory] + [InlineData("cut", 12)] + [InlineData("left out", 12)] + [InlineData("cut", 2)] + public void Predicted_rows_a_hidden_end_leaves_a_byte_short_are_undone_as_far_as_they_go(string guard, int predictor) + { + // Content stream 4's Flate data, raw deflate with no zlib header nor checksum, stores two predicted rows of 40 bytes + // whose last byte is 0x0D, then an end-of-line and its endstream, which the next object proves its own. The search + // leaves the carriage return out with the line feed, which only the hidden length says is the data's or not: the + // stored block runs out a byte short, which says nothing of the file, and the row it parts is the guard's, undone as + // far as it goes. It was dropped, or left predicted (#224). + var content = new List(); + + for (var row = 0; row < 2; row++) + { + var bytes = Encoding.ASCII.GetBytes(Invariant($"Row {row} of predicted data, padded to forty.")[..40]); + bytes[^1] = 0x0D; + + if (predictor != 2) + { + content.Add((byte)(row == 0 ? 0 : 2)); + } + + content.AddRange(bytes); + } + + var length = content.Count; + byte[] raw = [0x01, (byte)length, (byte)(length >> 8), (byte)~length, (byte)(~length >> 8), .. content]; + var data = Encoding.Latin1.GetString(raw); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + raw.Length.ToString(CultureInfo.InvariantCulture)), + [4] = file.Object(4, Invariant($"<< /Length 20 0 R /Filter /FlateDecode /DecodeParms << /Predictor {predictor} /Columns 40 >> >>\nstream\n{data}\nendstream")), + [5] = file.Object(5, "(five)"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + byte[] whole; + + using (var document = PdfDocument.Open(file.Bytes)) + { + whole = document.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray(); + whole.Should().HaveCount(80); + } + + using var guarded = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Decode().ToArray().Should().Equal(whole[..^1]); + stream.Data.EndHidden.Should().BeTrue(); + guarded.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.FilterFailed).Select(entry => entry.Message) + .Should().Equal("A Flate stream was not valid zlib data."); + } + + [Theory] + [InlineData("cut")] + [InlineData("left out")] + public void Lzw_data_that_lacks_its_end_code_before_a_proven_endstream_past_an_end_of_line_is_no_fault_of_the_file(string guard) + { + // Content stream 4's LZW data, the codes 256 and 113, "q", lacks its end-of-data code, then an end-of-line and its + // endstream, which the next object proves its own: whether the end-of-line is the data's only the hidden length says, + // and that the codes run out there says nothing of the file (#224). + var data = Encoding.Latin1.GetString([0x80, 0x1C, 0x40]); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + "3"), + [4] = file.Object(4, $"<< /Length 20 0 R /Filter /LZWDecode >>\nstream\n{data}\nendstream"), + [5] = file.Object(5, "(five)"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray().Should().Equal("q"u8.ToArray()); + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.FilterFailed); + } + + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Decode().ToArray().Should().Equal("q"u8.ToArray()); + stream.Data.EndHidden.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + } + + [Theory] + [InlineData("cut", " ")] + [InlineData("left out", " ")] + [InlineData("cut", "\n\n")] + [InlineData("cut", "\r\n%c\r\n")] + [InlineData("left out", " %c\n")] + public void A_checksum_short_of_a_byte_white_space_before_a_proven_endstream_completes_is_no_fault_of_the_file( + string guard, string tail) + { + // Content stream 4's Flate data lacks its checksum's last byte, then white space before its endstream, which the next + // object proves its own. The search took a byte of white space in as data, which only the hidden length says is the + // data's or not, and which the inflater reads as the checksum's last byte: a checksum that disagrees was reported, a + // warning, where read whole the checksum is only short (#224). + byte[] zlib; + + using (var output = new MemoryStream()) + { + using (var stream = new ZLibStream(output, CompressionLevel.Optimal, leaveOpen: true)) + { + stream.Write("BT (Hello) Tj ET"u8); + } + + zlib = output.ToArray()[..^1]; + } + + var data = Encoding.Latin1.GetString(zlib); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + zlib.Length.ToString(CultureInfo.InvariantCulture)), + [4] = file.Object(4, $"<< /Length 20 0 R /Filter /FlateDecode >>\nstream\n{data}{tail}endstream"), + [5] = file.Object(5, "(five)"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + var content = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + content.Decode().ToArray().Should().Equal("BT (Hello) Tj ET"u8.ToArray()); + content.Data.EndHidden.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + } + + [Theory] + [InlineData("cut", "\n")] + [InlineData("cut", "\n\n")] + [InlineData("left out", " \n")] + [InlineData("cut", "\r\n\r\n")] + public void A_checksum_that_disagrees_before_the_white_space_a_hidden_length_leaves_is_the_file_s(string guard, string tail) + { + // Content stream 4's Flate data is whole, its checksum's first byte wrong, then white space before its endstream, + // which the next object proves its own. Its last byte is no white space, nor completed by any: the checksum disagrees + // whatever the hidden length holds, as read whole. It said nothing where the data taken ended in white space (#224). + var zlib = Zlib("BT (Hello) Tj ET"u8); + zlib[^4] ^= 0x55; + var file = HiddenLengthFile(guard, "/Filter /FlateDecode", zlib, tail); + + using (var whole = PdfDocument.Open(file)) + { + whole.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray().Should().Equal("BT (Hello) Tj ET"u8.ToArray()); + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.FilterChecksumMismatch); + } + + using var document = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Decode().ToArray().Should().Equal("BT (Hello) Tj ET"u8.ToArray()); + stream.Data.EndHidden.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(GuardCode(guard), PdfDiagnosticCodes.FilterChecksumMismatch); + } + + [Theory] + [InlineData("cut", 12, 60)] + [InlineData("left out", 12, 60)] + [InlineData("cut", 2, 60)] + [InlineData("cut", 12, 30)] + public void A_filter_after_one_that_ends_where_its_format_says_decodes_as_read_whole_whatever_hides_the_end( + string guard, int predictor, int kept) + { + // Content stream 4's ASCIIHex data, its > end marker in place before the end-of-line and the endstream the next object + // proves its own, decodes to Flate data that stores two predicted rows of 41 bytes, cut short after so many. The + // ASCIIHex data ends where its format says, before the white space: the Flate data it decodes to is whole, and lost + // its tail as read whole, the file's. It was decoded as data the hidden end ran short, its parted row undone, or a + // predictor's warning left unsaid (#224). + var rows = new byte[82]; + + for (var i = 0; i < rows.Length; i++) + { + rows[i] = i % 41 == 0 ? (byte)0 : (byte)('a' + (i % 26)); + } + + byte[] stored = [0x78, 0x01, 0x01, 82, 0, unchecked((byte)~82), 0xFF, .. rows]; + var hex = Convert.ToHexString(stored[..(7 + kept)]) + ">"; + var file = HiddenLengthFile( + guard, + Invariant($"/Filter [/ASCIIHexDecode /FlateDecode] /DecodeParms [null << /Predictor {predictor} /Columns 40 >>]"), + Encoding.ASCII.GetBytes(hex), + "\n"); + byte[] whole; + string[] faults; + + using (var document = PdfDocument.Open(file)) + { + whole = document.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray(); + faults = [.. document.Diagnostics.Select(entry => entry.Message)]; + faults.Should().NotBeEmpty(); + } + + using var guarded = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Decode().ToArray().Should().Equal(whole); + stream.Data.EndHidden.Should().BeTrue(); + guarded.Diagnostics.Skip(1).Select(entry => entry.Message).Should().Equal(faults); + } + + [Theory] + [InlineData("cut", 2, 5)] + [InlineData("left out", 2, 5)] + [InlineData("cut", 12, 4)] + [InlineData("left out", 12, 14)] + public void Lzw_rows_a_hidden_end_leaves_a_byte_short_are_undone_as_far_as_they_go(string guard, int predictor, int columns) + { + // Content stream 4's LZW data, a clear code and fifteen literal codes, lacks its end-of-data code: its last byte, + // 0x0D, ends the last code, then an end-of-line and the endstream the next object proves its own. The search leaves + // the carriage return out with the line feed, which only the hidden length says is the data's or not: the codes run + // out a byte short, which says nothing of the file, and the row it parts is the guard's, undone as far as it goes. + // It was dropped, left predicted, or, the data shorter than a row of 14 columns, reported as rows longer than the + // data (#224). + int[] plain = predictor == 2 + ? [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 13] + : [0, 1, 2, 3, 4, 2, 1, 1, 1, 1, 2, 3, 5, 7, 13]; + var raw = FilterDamageTests.NineBitCodes([256, .. plain]); + raw[^1].Should().Be(0x0D); + var file = HiddenLengthFile( + guard, Invariant($"/Filter /LZWDecode /DecodeParms << /Predictor {predictor} /Columns {columns} >>"), raw, "\n"); + byte[] whole; + + using (var document = PdfDocument.Open(file)) + { + whole = document.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray(); + } + + using var guarded = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Decode().ToArray().Should().Equal(whole[..^1]); + stream.Data.EndHidden.Should().BeTrue(); + guarded.Diagnostics.Select(entry => entry.Code).Should().Equal(GuardCode(guard)); + } + + [Theory] + [InlineData("cut", "/LZWDecode", "\n\n\n\n")] + [InlineData("left out", "/LZWDecode", "\t\t\t\t\t\n")] + [InlineData("cut", "/ASCIIHexDecode", "\n%c\n")] + [InlineData("left out", "/ASCIIHexDecode", " %c\n")] + public void White_space_and_comments_a_hidden_length_leaves_are_not_decoded_as_data_that_lacks_its_end_marker( + string guard, string filter, string tail) + { + // Content stream 4's data, LZW or ASCIIHex, lacks its end marker, then white space, or a comment, before the + // endstream the next object proves its own: what it decodes to, read whole, is "q", and a missing end marker. The + // LZW data read the white space as more codes, one of them undefined, and the ASCIIHex data a comment's digit, its % + // a byte passed over: faults the file's length leaves out, and bytes it does not decode to (#224). + var data = filter == "/LZWDecode" ? [0x80, 0x1C, 0x40] : "71"u8.ToArray(); + var file = HiddenLengthFile(guard, "/Filter " + filter, data, tail); + + using (var whole = PdfDocument.Open(file)) + { + whole.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray().Should().Equal("q"u8.ToArray()); + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.FilterFailed); + } + + using var document = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Decode().ToArray().Should().Equal("q"u8.ToArray()); + stream.Data.EndHidden.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(GuardCode(guard)); + } + + [Theory] + [InlineData("cut")] + [InlineData("left out")] + public void An_object_stream_whose_last_code_a_hidden_end_leaves_a_byte_short_ends_where_the_guard_hides(string guard) + { + // Object stream 6's LZW data lacks its end-of-data code, and its last byte, 0x0A, which ends the last code, the + // catalog's closing " >>", stands right before the endstream the next object proves its own; its /Length, object 20, + // is cut or left out of the index. The search took it for + // the end-of-line before the keyword, which only the hidden length says is the data's or not: the codes run out a + // byte short, which says nothing of the file. The catalog's dictionary, left unclosed, was reported as the file's + // syntax fault (#224). + var letters = "ggjlxbmorogsxwbojsdnaqehqoilkdhznwbupwnsouprkjamamuxnotnanksexotmnfxewtjihojygfiwdqvarkhjijdmbaavguqtbvkp"; + var first = $"<< /A ({letters}) /B << /C 1 >> /D << /E 2 >> >>"; + var plain = Invariant($"9 0 1 {first.Length + 1} {first} << /Type /Catalog /Pages 2 0 R >>"); + var lzw = FilterDamageTests.LzwEncode(Encoding.ASCII.GetBytes(plain), endCode: false); + lzw[^1].Should().Be(0x0A); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = (2, 6, 1), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [20] = (1, file.Object(20, (guard == "cut" ? new string(' ', 1100) : string.Empty) + Invariant($"{lzw.Length}")), 0), + [6] = (1, file.Position, 0), + [9] = (2, 6, 0), + }; + file.Raw(Invariant($"6 0 obj\n<< /Type /ObjStm /N 2 /First {plain.IndexOf('<', StringComparison.Ordinal)} /Filter /LZWDecode /Length 20 0 R >>\nstream\n")); + file.Raw(Encoding.Latin1.GetString(lzw) + "endstream\nendobj\n"); + file.CrossReferenceStream(7, 21, rows, "/Root 1 0 R"); + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.GetObject(new PdfObjectId(1)).AsDictionary().Required().IsOfType(PdfName.Catalog).Should().BeTrue(); + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.FilterFailed); + } + + using var document = PdfDocument.Open(file.Bytes, HiddenLengthOptions(guard)); + + document.GetObject(new PdfObjectId(1)).AsDictionary().Required().IsOfType(PdfName.Catalog).Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(GuardCode(guard)); + } + + [Theory] + [InlineData("cut", "/ASCII85Decode", "%74#:-qIocCi:aF<,*OE;u", "\n", false)] + [InlineData("left out", "/ASCII85Decode", "%74#:\n%-qIocCi:aF<,*OE;u", "\n", false)] + [InlineData("cut", "/ASCII85Decode", "6<#'\\7PQ#?1*BP.+?)\n%u2_d'1<+I+\"87cURD]g__+B3(u78s", "\n", false)] + [InlineData("left out", "/ASCII85Decode", "<~6<#'\\7PQ#?1*BP.+?)\n%u2_d'1<+I+\"87cURD]g__+B3(u78s~>", "\n", false)] + [InlineData("cut", "/RunLengthDecode", "%xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "\n", true)] + [InlineData("cut", "/LZWDecode", "\u0080\u001C\u0040", " \n", true)] + [InlineData("left out", "/FlateDecode", "hello world \t", "\n", false)] + [InlineData("cut", "[/ASCIIHexDecode /FlateDecode] /DecodeParms << /Predictor 12 /Columns 4 >>", "{predicted}>", "\n", false)] + [InlineData("cut", "[/LZWDecode /ASCIIHexDecode]", "{lzw-hex}", "\n", true)] + [InlineData("left out", "[/RunLengthDecode /ASCIIHexDecode]", "\u0006717171\n", "", true)] + [InlineData("cut", "/ASCIIHexDecode", " ", "\n", false)] + public void Data_a_hidden_length_ends_decodes_as_read_whole_but_for_what_the_run_after_it_may_hold( + string guard, string filter, string data, string tail, bool cut) + { + // Content stream 4's data, then white space before an endstream the next object proves its own, under a /Length a + // guard cut or left out of the index (#224). Only the run of white space, and of comments past white space where no + // data holds a %, is what the hidden length may leave out. A % the data starts with was taken for a comment and the + // whole data dropped; a % at a line's start, a digit in ASCII85, the rest of the line; spaces after LZW codes read + // as their end code; the chain's Flate predictor run on an ASCIIHex step; data that is only white space under + // ASCIIHex taken for cut. What the first filter decodes is the whole read's, and only where it may lack what the run + // holds — LZW or RunLength data without its end marker — is its end the guard's, for the filters after it too. + var bytes = data switch + { + "{predicted}>" => Encoding.ASCII.GetBytes(Convert.ToHexString(Zlib([2, 1, 2, 3, 4, 2, 0, 0, 0, 0])) + ">"), + "{lzw-hex}" => FilterDamageTests.NineBitCodes([256, '7', '1', '7', '1', '7', '1', '7', '1', '7', '1', '7', '1', '7', '1', 13]), + _ => Encoding.Latin1.GetBytes(data), + }; + var file = HiddenLengthFile(guard, "/Filter " + filter, bytes, tail); + byte[] whole; + + using (var document = PdfDocument.Open(file)) + { + whole = document.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray(); + } + + using var guarded = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + var decoded = PdfFilterPipeline.Decode(stream, null, out _, out var cutByGuard).ToArray(); + + stream.Data.EndHidden.Should().BeTrue(); + decoded.Should().Equal(whole); + cutByGuard.Should().Be(cut); + guarded.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.FilterFailed && entry.Message.Contains("end", StringComparison.Ordinal)) + .Should().BeEmpty(); + } + + [Theory] + [InlineData("cut", 12)] + [InlineData("left out", 2)] + public void Flate_data_a_hidden_length_ends_where_its_deflate_body_does_decodes_as_read_whole(string guard, int predictor) + { + // Content stream 4's Flate data, under a /Length a guard cut or left out of the index, its endstream proven its own: + // raw deflate whose last byte is a NUL, zlib whose checksum is wrong past bytes that look like a comment after white + // space, zlib whose predicted rows end it whole. The deflate body ends in the data as the search took it: what it + // decodes to is whole, the repair raw deflate asks for reported as read whole. It was decoded again up to the NUL, or + // to the % the checksum's bytes hold, its last bytes lost and its end taken for the guard's (#224). + var raw = new MemoryStream(); + + using (var deflate = new DeflateStream(raw, CompressionLevel.Optimal, leaveOpen: true)) + { + deflate.Write("BT /F1 12 Tf 72 712 Td (Hello, world) Tj ET"u8); + } + + byte[] stored = [0x78, 0x01, 0x01, 15, 0, unchecked((byte)~15), 0xFF, .. "BT (a %b) Tj ET"u8, 0, 0, 0, 0]; + byte[] rows = predictor == 2 ? [1, 2, 3, 4, 1, 1, 1, 1] : [2, 1, 2, 3, 4, 2, 0, 0, 0, 0]; + (string Entries, byte[] Data)[] cases = + [ + ("/Filter /FlateDecode", raw.ToArray()), + ("/Filter /FlateDecode", stored), + (Invariant($"/Filter /FlateDecode /DecodeParms << /Predictor {predictor} /Columns 4 >>"), Zlib(rows)), + ]; + + raw.ToArray()[^1].Should().Be(0); + + foreach (var (entries, data) in cases) + { + var file = HiddenLengthFile(guard, entries, data, "\n"); + byte[] whole; + string[] faults; + + using (var document = PdfDocument.Open(file)) + { + whole = document.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().ToArray(); + faults = [.. document.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.FilterFailed).Select(entry => entry.Message)]; + } + + using var guarded = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + + PdfFilterPipeline.Decode(stream, null, out _, out var cutByGuard).ToArray().Should().Equal(whole); + cutByGuard.Should().BeFalse(); + guarded.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.FilterFailed).Select(entry => entry.Message) + .Should().Equal(faults); + } + } + + [Theory] + [InlineData("cut", false)] + [InlineData("left out", true)] + public void A_decoding_guard_flate_data_a_hidden_length_ends_reaches_is_reached_as_read_whole(string guard, bool checksumWrong) + { + // Content stream 4's zlib data, its checksum right or wrong, under a /Length a guard cut or left out of the index, its + // endstream proven its own, decodes to 4 MB, past a MaxDecodedStreamLength of 1 MB, which stops it read whole. Taken + // for a deflate body that ends in the data, its output would be whole: the guard unreported, and what it cut the + // file's. + var zlib = Zlib(new byte[4 << 20]); + zlib[^1] ^= checksumWrong ? (byte)0x55 : (byte)0; + var file = HiddenLengthFile(guard, "/Filter /FlateDecode", zlib, "\n"); + var hidden = HiddenLengthOptions(guard); + + using (var whole = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 1 << 20 } })) + { + var read = whole.GetObject(new PdfObjectId(4)).AsStream().Required(); + PdfFilterPipeline.Decode(read, null, out _, out var cut).Length.Should().Be(1 << 20); + cut.Should().BeTrue(); + } + + using var guarded = PdfDocument.Open(file, hidden with { Limits = hidden.Limits with { MaxDecodedStreamLength = 1 << 20 } }); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + var decoded = PdfFilterPipeline.Decode(stream, null, out _, out var cutByGuard); + + stream.Data.EndHidden.Should().BeTrue(); + decoded.Length.Should().Be(1 << 20); + cutByGuard.Should().BeTrue(); + guarded.Diagnostics.Select(entry => entry.Code).Should().Equal(GuardCode(guard), PdfDiagnosticCodes.LimitDecodedStream); + } + + [Theory] + [InlineData("cut", "/LZWDecode", "\n\n")] + [InlineData("left out", "/FlateDecode", " \t\n")] + [InlineData("cut", "/RunLengthDecode", " % c\n\n")] + public void Data_that_is_all_the_run_a_hidden_length_may_leave_out_is_the_guard_s_to_lack_under_a_filter_with_an_end( + string guard, string filter, string tail) + { + // Content stream 4's data is nothing but white space, and a comment past it, before the endstream the next object + // proves its own, under a /Length a guard cut or left out of the index: the hidden length may leave it all out, and + // the LZW, Flate or RunLength data it gives, which a whole read ends where its format says, the guard's to lack. Under + // ASCIIHex, data of white space alone is whole (above). + var file = HiddenLengthFile(guard, "/Filter " + filter, [], tail); + + using var guarded = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + var decoded = PdfFilterPipeline.Decode(stream, null, out _, out var cutByGuard); + + stream.Data.EndHidden.Should().BeTrue(); + decoded.Length.Should().Be(0); + cutByGuard.Should().BeTrue(); + guarded.Diagnostics.Select(entry => entry.Code).Should().Equal(GuardCode(guard)); + } + + [Theory] + [InlineData("cut", "/ASCII85Decode", " \t\n")] + [InlineData("left out", "/ASCII85Decode", "\r\n\f\n")] + [InlineData("left out", "[/ASCII85Decode /FlateDecode]", " \n")] + [InlineData("cut", "/ASCIIHexDecode", " %c\n")] + public void Data_that_is_all_the_run_a_hidden_length_may_leave_out_decodes_whole_to_nothing_under_ascii85_or_asciihex( + string guard, string filter, string tail) + { + // Content stream 4's /Length is 0, then nothing but white space, or under ASCIIHex a comment past it, before the + // endstream the next object proves its own, under a /Length a guard cut or left out of the index: the search took + // the run the hidden length may leave out for the data (#224). White space decodes to nothing in ASCII85 or ASCIIHex + // data, and so does a comment in ASCIIHex, whatever the hidden length holds of them: what the stream decodes to, + // nothing, is whole, as read whole, for a Flate filter after ASCII85 too, and no end marker the run lacks is + // reported. Under a filter with an end, the run is the guard's to lack (above). + var file = HiddenLengthFile(guard, "/Filter " + filter, [], tail); + + using (var document = PdfDocument.Open(file)) + { + document.GetObject(new PdfObjectId(4)).AsStream().Required().Decode().Length.Should().Be(0); + document.Diagnostics.Should().BeEmpty(); + } + + using var guarded = PdfDocument.Open(file, HiddenLengthOptions(guard)); + var stream = guarded.GetObject(new PdfObjectId(4)).AsStream().Required(); + var decoded = PdfFilterPipeline.Decode(stream, null, out var leftEncoded, out var cutByGuard); + + stream.Data.EndHidden.Should().BeTrue(); + stream.GetRawBytes().ToArray().Should().Equal(Encoding.Latin1.GetBytes(tail[..^1])); + decoded.Length.Should().Be(0); + leftEncoded.Should().BeFalse(); + cutByGuard.Should().BeFalse(); + guarded.Diagnostics.Select(entry => entry.Code).Should().Equal(GuardCode(guard)); + } + + /// + /// A file whose content stream 4 holds and before its endstream, which + /// object 5 after it proves its own, its /Length object 20, which a MaxObjectLength of 1,024 cuts, or, left + /// out, a MaxIndexEntries of 20 leaves out of the index. + /// + private static byte[] HiddenLengthFile(string guard, string entries, byte[] data, string tail) + { + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + data.Length.ToString(CultureInfo.InvariantCulture)), + [4] = file.Object(4, $"<< /Length 20 0 R {entries} >>\nstream\n{Encoding.Latin1.GetString(data)}{tail}endstream"), + [5] = file.Object(5, "(five)"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + return file.Bytes; + } + + private static PdfReaderOptions HiddenLengthOptions(string guard) => new() + { + Limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }, + }; + + private static string GuardCode(string guard) => + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries; + + private static byte[] Zlib(ReadOnlySpan data) + { + using var output = new MemoryStream(); + + using (var stream = new ZLibStream(output, CompressionLevel.Optimal, leaveOpen: true)) + { + stream.Write(data); + } + + return output.ToArray(); + } + + private static byte[] ZLibDecompress(byte[] zlib) + { + using var input = new MemoryStream(zlib); + using var stream = new ZLibStream(input, CompressionMode.Decompress); + using var output = new MemoryStream(); + stream.CopyTo(output); + return output.ToArray(); + } + + [Theory] + [InlineData("next object", true)] + [InlineData("next object", false)] + [InlineData("next object past white space", true)] + [InlineData("next object past white space", false)] + [InlineData("xref", true)] + [InlineData("xref", false)] + [InlineData("startxref", true)] + [InlineData("startxref", false)] + [InlineData("end of file", true)] + [InlineData("end of file", false)] + [InlineData("object no row places", true)] + [InlineData("object no row places", false)] + public void An_endstream_where_a_guard_hid_the_length_ends_data_that_is_the_file_s_where_what_follows_proves_it_its_own( + string follower, bool endObj) + { + // Object 4's /Length, object 20, is one MaxObjectLength cuts or MaxIndexEntries leaves out of the index, and its + // endstream, which its endobj follows or not, is followed by the next object the index places, a cross-reference + // section's keyword, or the end of the file: no endstream the data spells is, and the data is the file's, the + // object's endobj judged. Followed by an object no row places, the endstream may be one the data spells, which only + // the length tells from the stream's own: the data up to it is the guard's, the object's endobj unjudged (#224). + var proven = follower != "object no row places"; + + foreach (var guard in (string[])["cut", "left out"]) + { + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + using var document = PdfDocument.Open(StreamFollowedBy(guard, follower, endObj), new PdfReaderOptions { Limits = limits }); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.GetRawBytes().ToArray().Should().Equal("BT ET"u8.ToArray()); + stream.Data.CutByGuard.Should().Be(!proven, guard); + document.Reader.IsEndObjMissing(4, out _).Should().Be(proven && !endObj, guard); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + } + } + + [Theory] + [InlineData("cut")] + [InlineData("left out")] + public void An_endstream_the_file_ends_with_ends_data_that_is_the_file_s_where_a_guard_hid_the_length(string guard) + { + // Object 4's /Length, object 20, is one MaxObjectLength cuts or MaxIndexEntries leaves out of the index, and the file + // ends on its endstream, with no end-of-line after it: the keyword, which no byte goes on, is whole, and the end of + // the file that follows proves it the stream's own. The data is the file's, and its object lacks its endobj, as where + // an end-of-line ends the file (#224). + var file = StreamFollowedBy(guard, "end of file", endObj: false)[..^1]; + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = limits }); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + Encoding.Latin1.GetString(file).Should().EndWith("\nendstream"); + stream.GetRawBytes().ToArray().Should().Equal("BT ET"u8.ToArray()); + stream.Data.CutByGuard.Should().BeFalse(); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_cross_reference_stream_s_length_a_guard_cut_leaves_rows_its_proven_endstream_ends_to_the_file(bool compressed) + { + // Stream 5's /Length, object 4, which only its own row places, is one MaxObjectLength cuts before its value; its data + // lost its last bytes — its Flate data, its checksum and the end of its last row —, and its endstream, endobj and + // the startxref follow. The chain took the rows the data lacks for the guard's, and the Flate data run out said + // nothing, where a load of the stream, its endstream proven its own, reads the file's end. They are the file's, as + // read whole: the section is incomplete, and the index rebuilt (#224). + var file = ChainFiles.AloneWithIndirectLength(compressed: compressed, dropped: 6, lengthPadding: 1100); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + whole.Reader.Structure.Sections.Should().ContainSingle().Which.Incomplete.Should().BeTrue(); + document.Reader.Structure.Sections.Should().ContainSingle().Which.Incomplete.Should().BeTrue(); + document.Reader.Structure.Sections[0].CutByLimit.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Where(code => code != PdfDiagnosticCodes.LimitObject) + .Should().Equal(whole.Diagnostics.Select(entry => entry.Code)); + document.Diagnostics.Select(entry => entry.Code).Should().Contain(PdfDiagnosticCodes.LimitObject); + + if (compressed) + { + document.Diagnostics.Should().Contain(entry => entry.Message.StartsWith("A Flate stream ends before its data does", StringComparison.Ordinal)); + } + } + + [Theory] + [InlineData("cut", false)] + [InlineData("left out", false)] + [InlineData("cut", true)] + [InlineData("left out", true)] + public void An_endstream_and_an_endobj_the_data_spells_where_a_guard_hid_the_length_end_data_that_is_the_guard_s(string guard, bool hex) + { + // Object 4's content spells "endstream" and "endobj" in a string, which more of the content follows: read whole, the + // length passes them. Where a guard hid it, the search stopped there, the endobj taken for the object's own and the + // data, cut short, for the file's — its ASCIIHex data said to lack its ">". Nothing after them proves the endstream + // the stream's own: the data up to it is the guard's (#224). + const string Spelled = "\nendstream\nendobj\n"; + var data = hex ? "4254202861" + Spelled + "29205466>" : "BT (a" + Spelled + ") Tf ET"; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [4] = file.Object(4, $"<< /Length 20 0 R{(hex ? " /Filter /ASCIIHexDecode" : string.Empty)} >>\nstream\n{data}\nendstream"), + [6] = file.Object(6, "(six)"), + [20] = file.Object(20, (guard == "cut" ? new string(' ', 2000) : string.Empty) + data.Length.ToString(CultureInfo.InvariantCulture)), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = 20 }; + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal(Encoding.Latin1.GetBytes(data)); + } + + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = limits }); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + _ = stream.Decode(); + + stream.GetRawBytes().ToArray().Should().Equal(Encoding.Latin1.GetBytes(data[..data.IndexOf('\n', StringComparison.Ordinal)])); + stream.Data.CutByGuard.Should().BeTrue(); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + guard == "cut" ? PdfDiagnosticCodes.LimitObject : PdfDiagnosticCodes.LimitIndexEntries); + } + + [Theory] + [InlineData(279, false)] + [InlineData(279, true)] + [InlineData(200, false)] + public void A_decoding_guard_deferred_data_reached_stands_when_the_data_is_decoded_again_knowing_its_end_hidden(int bound, bool throwOnLimit) + { + // Stream 5's rows decode to 280 bytes, through ASCIIHex then Flate, under a /Length only it places. Under 279, Flate + // stops at the bound, and the data, which lacks the page's row, is decoded again knowing its end hidden: ASCIIHex + // drops the digit its end parts, and Flate, given less, stayed within the bound. The guard went unreported, and the + // page's row was charged to the file — the section incomplete, the index rebuilt (#220, #224). + var file = ChainFiles.HexFlateRowsLackingTheirLastDigit(); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.FilterFailed, PdfDiagnosticCodes.FilterFailed); + whole.GetObject(new PdfObjectId(39)).AsDictionary().IsOfType(PdfName.Page).Should().BeTrue(); + } + + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = bound }, ThrowOnLimit = throwOnLimit }; + + if (throwOnLimit) + { + var opening = () => PdfDocument.Open(file, options); + opening.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxDecodedStreamLength)); + return; + } + + using var document = PdfDocument.Open(file, options); + var section = document.Reader.Structure.Sections.Should().ContainSingle().Which; + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + section.CutByLimit.Should().BeTrue(); + section.Incomplete.Should().BeFalse(); + document.GetObject(new PdfObjectId(39)).Should().BeOfType(); + document.WasRepaired.Should().BeFalse(); + var findings = new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId); + findings.Should().Contain(PdfValidationRuleIds.XRefCheckedInPart).And.NotContain(PdfValidationRuleIds.XRefSectionMalformed); + } + + [Theory] + [InlineData(1000, 1, 0)] + [InlineData(1, 0, 1)] + [InlineData(0, 0, 1)] + public void A_decoding_guard_deferred_data_reached_is_reported_once_whatever_room_the_diagnostics_have( + int capacity, int reports, int leftOut) + { + // As above, under 279: the guard the first decoding reached stands, its report held as the stream's decoding + // reports are, so that decoding the stream again reports it no more; where the diagnostics had no room for it, it + // is counted as left out, while the ASCIIHex end fault the first decoding met, dropped with that decoding, is not + // (#144, #220, #224). + var options = new PdfReaderOptions + { + Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 279 }, + DiagnosticCapacity = capacity, + }; + using var document = PdfDocument.Open(ChainFiles.HexFlateRowsLackingTheirLastDigit(), options); + + document.Diagnostics.Count.Should().Be(reports); + document.Diagnostics.SuppressedCount.Should().Be(leftOut); + + var stream = document.GetObject(new PdfObjectId(5)).AsStream().Required(); + _ = stream.Decode(); + _ = stream.Decode(); + + document.Diagnostics.Count(entry => entry.Code == PdfDiagnosticCodes.LimitDecodedStream).Should().Be(reports); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void A_cross_reference_stream_with_no_row_of_its_own_proves_the_endstream_before_it(bool ownRow) + { + // Content stream 4's /Length, object 5, is cut by MaxObjectLength, and the cross-reference stream follows it, with a + // row of its own or none, as PDFBox writes it: the chain read it there, and it places it all the same. With no row, + // nothing proved the endstream the stream's own, and its data, whole, was the guard's (#224). + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [5] = (1, file.Object(5, new string(' ', 2000) + "5"), 0), + [4] = (1, file.Object(4, "<< /Length 5 0 R >>\nstream\nBT ET\nendstream"), 0), + }; + file.CrossReferenceStream(6, ownRow ? 7 : 6, rows, "/Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Data.CutByGuard.Should().BeFalse(); + stream.GetRawBytes().ToArray().Should().Equal("BT ET"u8.ToArray()); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + } + + [Fact] + public void A_cross_reference_stream_the_chain_read_proves_the_endstream_of_the_older_one_before_it() + { + // Older cross-reference stream 6 takes its /Length from object 9, which MaxObjectLength cuts, and declares 11 rows + // where its data holds 10; newer stream 7, which no row of its own places, follows its endobj, the only startxref + // after it. The chain read stream 7 first, which proves the endstream before it the older stream's own: the row it + // lacks is the file's to lack, as read whole, not the guard's (#224). + var file = new HandWrittenFile(version: "1.5"); + var placed = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [9] = (1, file.Object(9, new string(' ', 2000) + "70"), 0), + }; + var older = file.Position; + placed[6] = (1, older, 0); + file.Object(6, "<< /Type /XRef /Size 11 /W [1 4 2] /Root 1 0 R /Length 9 0 R >>\nstream\n" + Rows(10, placed) + "\nendstream"); + var newer = file.Position; + var rows = Rows(10, placed); + file.Object(7, Invariant($"<< /Type /XRef /Size 10 /W [1 4 2] /Root 1 0 R /Prev {older} /Length {rows.Length} >>\nstream\n{rows}\nendstream")); + file.Raw(Invariant($"startxref\n{newer}\n%%EOF\n")); + + using var whole = PdfDocument.Open(file.Bytes); + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + var section = document.Reader.Structure.Sections[^1]; + + section.Incomplete.Should().BeTrue(); + section.CutByLimit.Should().BeFalse(); + section.Fault.Should().Be(whole.Reader.Structure.Sections[^1].Fault).And.Be("it holds 10 rows where its /Index and /Size declare 11"); + + static string Rows(int size, Dictionary placed) + { + var data = new StringBuilder(); + + for (var row = 0; row < size; row++) + { + var (type, field2, field3) = row == 0 ? (0, 0L, 65535) : placed.TryGetValue(row, out var given) ? given : (0, 0L, 0); + data.Append((char)type) + .Append((char)((field2 >> 24) & 0xFF)).Append((char)((field2 >> 16) & 0xFF)) + .Append((char)((field2 >> 8) & 0xFF)).Append((char)(field2 & 0xFF)) + .Append((char)((field3 >> 8) & 0xFF)).Append((char)(field3 & 0xFF)); + } + + return data.ToString(); + } + } + + [Theory] + [InlineData(20)] + [InlineData(40)] + public void A_cross_reference_stream_named_before_white_space_proves_the_endstream_before_it_however_much(int padding) + { + // As above, with no row, startxref naming the white space before the cross-reference stream's header: the section + // said where the header lies only within the first 32 bytes its probe read, and past them nothing proved the + // endstream the stream's own. The validator said the offset lies 32 bytes before the header (#224). + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [5] = (1, file.Object(5, new string(' ', 2000) + "5"), 0), + [4] = (1, file.Object(4, "<< /Length 5 0 R >>\nstream\nBT ET\nendstream"), 0), + }; + file.Raw(new string(' ', padding)); + var header = file.CrossReferenceStream(6, 6, rows, "/Root 1 0 R"); + var bytes = Encoding.Latin1.GetBytes(Encoding.Latin1.GetString(file.Bytes) + .Replace(Invariant($"startxref\n{header}\n"), Invariant($"startxref\n{header - padding}\n"), StringComparison.Ordinal)); + using var document = PdfDocument.Open(bytes, ObjectGuard); + + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.Data.CutByGuard.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + new PdfValidator().Validate(document).Findings.Should().ContainSingle(finding => finding.RuleId == PdfValidationRuleIds.XRefOffsetImprecise) + .Which.Message.Should().Contain(Invariant($"{padding} bytes before the header")); + } + + [Fact] + public void The_place_a_row_names_before_the_next_header_is_read_once_however_often_the_stream_is_parsed() + { + // Stream 10's /Length, object 9, is cut by MaxObjectLength, and object 11, which follows it, is placed by a row + // naming the line feed before its header: the proof reads past that offset to the header. It was read again on + // every parse of the stream, the cache having let it go, counted with the searches each time, until they could read + // no more and the stream, proven its own at first, came back cut by the guard (#224). + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [9] = (1, file.Object(9, new string(' ', 2000) + "1"), 0), + [10] = (1, file.Object(10, "<< /Length 9 0 R >>\nstream\nq\nendstream"), 0), + }; + rows[11] = (1, file.Object(11, "(eleven)") - 1, 0); + + for (var number = 12; number < 90; number++) + { + rows[number] = (1, file.Object(number, Invariant($"({number})")), 0); + } + + file.CrossReferenceStream(90, 91, rows, "/Root 1 0 R"); + var options = new PdfReaderOptions { ObjectCacheCapacity = 64, Limits = PdfReaderLimits.Default with { MaxObjectLength = 1024 } }; + using var document = PdfDocument.Open(file.Bytes, options); + + document.GetObject(new PdfObjectId(10)).AsStream().Required().Data.CutByGuard.Should().BeFalse(); + var read = document.Reader.EndStreamSearchBytes; + read.Should().Be(64, "four first reads of 16 bytes: past the data, the endstream, the endobj, and the row's offset"); + + for (var round = 0; round < 20; round++) + { + for (var number = 12; number < 90; number++) + { + _ = document.GetObject(new PdfObjectId(number)); + } + + document.GetObject(new PdfObjectId(10)).AsStream().Required().Data.CutByGuard.Should().BeFalse(); + } + + document.Reader.EndStreamSearchBytes.Should().Be(read); + } + + [Fact] + public void The_proofs_of_small_streams_whose_length_a_guard_hid_read_less_than_the_file() + { + // 400 streams of a byte share a /Length, object 9, which MaxObjectLength cuts before its value: each one's endstream is + // found among the bytes the parser holds, and proven its own by the next object the index places, a few bytes past + // its endobj. The proofs read those few bytes, which count toward what the searches may read, four times the file: a + // file of such streams has them read less than itself (#224). + const int Streams = 400; + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), 0), + [3] = (1, file.Object(3, Page), 0), + [9] = (1, file.Object(9, new string(' ', 2000) + "1"), 0), + }; + + for (var number = 10; number < 10 + Streams; number++) + { + rows[number] = (1, file.Object(number, "<< /Length 9 0 R >>\nstream\nq\nendstream"), 0); + } + + file.CrossReferenceStream(10 + Streams, 11 + Streams, rows, "/Root 1 0 R"); + var bytes = file.Bytes; + + using var document = PdfDocument.Open(bytes, ObjectGuard); + + for (var number = 10; number < 10 + Streams; number++) + { + var stream = document.GetObject(new PdfObjectId(number)).AsStream().Required(); + stream.Data.CutByGuard.Should().BeFalse(Invariant($"stream {number}'s endstream is proven its own")); + stream.Data.Length.Should().Be(1); + } + + document.Reader.EndStreamSearchBytes.Should().Be(Streams * 48, "each proof reads three times 16 bytes").And.BeLessThan(bytes.Length); + } + + [Theory] + [InlineData(true, false)] + [InlineData(false, true)] + [InlineData(true, true)] + public void The_proofs_of_streams_white_space_follows_read_less_than_three_times_the_file( + bool beforeEndObj, bool beforeNextHeader) + { + // Thirty streams whose /Length, object 20, MaxObjectLength cuts end in 65,520 bytes of white space, before their endobj + // or before the next object's header, which its row names the start of. The proof read again the white space before + // the endobj the reader had read on through, and doubled through that before the next header before reading it once + // more from the row: close to four times the file, the most the searches may read, where three were said (#224). + const int Streams = 30; + var run = new string(' ', 65_520); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, new string(' ', 1100) + "1"), + }; + + for (var number = 30; number < 30 + Streams; number++) + { + rows[number] = file.Position; + file.Raw(Invariant( + $"{(beforeNextHeader ? run : string.Empty)}{number} 0 obj\n<< /Length 20 0 R >>\nstream\nq\nendstream{(beforeEndObj ? run : string.Empty)}\nendobj\n")); + } + + rows[30 + Streams] = file.Object(30 + Streams, "(end)"); + file.Table(31 + Streams, rows, Invariant($"/Size {31 + Streams} /Root 1 0 R")); + var bytes = file.Bytes; + + using var document = PdfDocument.Open(bytes, ObjectGuard); + + for (var number = 30; number < 30 + Streams; number++) + { + document.GetObject(new PdfObjectId(number)).AsStream().Required().Data.CutByGuard.Should().BeFalse(); + } + + document.Reader.EndStreamSearchBytes.Should().BeLessThan(3L * bytes.Length); + } + + [Fact] + public void The_proofs_of_endstreams_data_spells_before_white_space_read_less_than_three_times_the_file() + { + // Thirty streams whose /Length, object 20, MaxObjectLength cuts spell an endstream in their data, 65,520 bytes of white + // space and a regular byte after it, which no endobj follows. The reader read on through the white space to tell + // whether an endobj follows, and the proof read it again from the keyword to what follows: close to four times the + // file, the most the searches may read, where three were said (#224). + const int Streams = 30; + var run = new string(' ', 65_520); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, Invariant($"{new string(' ', 1100)}{12 + run.Length}")), + }; + + for (var number = 30; number < 30 + Streams; number++) + { + rows[number] = file.Position; + file.Raw(Invariant($"{number} 0 obj\n<< /Length 20 0 R >>\nstream\nq\nendstream{run}x\nendstream\nendobj\n")); + } + + rows[30 + Streams] = file.Object(30 + Streams, "(end)"); + file.Table(31 + Streams, rows, Invariant($"/Size {31 + Streams} /Root 1 0 R")); + var bytes = file.Bytes; + + using var document = PdfDocument.Open(bytes, ObjectGuard); + + for (var number = 30; number < 30 + Streams; number++) + { + document.GetObject(new PdfObjectId(number)).AsStream().Required().Data.CutByGuard.Should().BeTrue(); + } + + document.Reader.EndStreamSearchBytes.Should().BeLessThan(3L * bytes.Length); + } + + [Theory] + [InlineData("Rx", false)] + [InlineData("RG", false)] + [InlineData("R\n", true)] + public void An_R_a_window_s_edge_reaches_makes_a_reference_only_where_the_file_ends_it_there(string tail, bool reference) + { + // Object 4, the stream's /Filter, holds "6 0 " and the tail after 100 spaces, and MaxObjectLength ends its window + // right after the R; object 6 holds /ASCIIHexDecode. An R a regular byte goes on was taken whole: the cut object + // held a reference the file never wrote, which the /Filter followed to object 6, past the guard's cut, and the data + // was decoded. The edge cuts what may be a reference, as under a bound one byte smaller, and the data is left as it + // is, the guard's (#224, #263). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, new string(' ', 100) + "6 0 " + tail), + [6] = file.Object(6, "/ASCIIHexDecode"), + [7] = file.Object(7, "<< /Length 7 /Filter 4 0 R >>\nstream\n414243>\nendstream"), + }; + file.Table(8, rows, "/Size 8 /Root 1 0 R"); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = "4 0 obj\n".Length + 105 } }; + using var document = PdfDocument.Open(file.Bytes, options); + + var data = document.GetObject(new PdfObjectId(7)).AsStream().Required().Decode().ToArray(); + + Encoding.ASCII.GetString(data).Should().Be(reference ? "ABC" : "414243>"); + document.Reader.IsCutAtLimit(4).Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + } + + [Fact] + public void The_proof_of_an_endstream_weighs_each_caller_s_next_object_against_what_follows_its_object() + { + // What follows object 4's end is read once, and each caller's next object weighed against it: the answer kept for a + // first caller that knew of no next object was served to every later one, whose index places object 6 there (#224). + var file = StreamFollowedBy("cut", "next object", endObj: true); + var six = (long)Encoding.Latin1.GetString(file).IndexOf("6 0 obj", StringComparison.Ordinal); + using var document = PdfDocument.Open(file, ObjectGuard); + var data = document.GetObject(new PdfObjectId(4)).AsStream().Required().Data; + var provider = (IPdfStreamDataProvider)document.Reader; + + provider.IsOwnEndStream(data.Position + data.Length, EndObjState.Present, null).Should().BeFalse(); + provider.IsOwnEndStream(data.Position + data.Length, EndObjState.Present, six).Should().BeTrue(); + provider.IsOwnEndStream(data.Position + data.Length, EndObjState.Present, six - 1).Should().BeTrue(); + provider.IsOwnEndStream(data.Position + data.Length, EndObjState.Present, six + 1).Should().BeFalse(); + } + + [Fact] + public void The_proof_of_an_endstream_said_to_be_followed_by_an_endobj_proves_nothing_where_none_can_be_read() + { + // Object 4's endstream ends the file but for an end-of-line, or is followed by its endobj, which ends the file. Told + // that an endobj follows the keyword, the proof reads on past it to that endobj: none follows where the file ends, + // nor where its source, cut short after the document opened, gives no byte past that end-of-line, and nothing is + // proven — where the file holds one, the end of the file after it proves the endstream the stream's own (#224). + var bare = StreamFollowedBy("cut", "end of file", endObj: false); + var ended = StreamFollowedBy("cut", "end of file", endObj: true); + var dataEnd = (long)Encoding.Latin1.GetString(ended).LastIndexOf("\nendstream", StringComparison.Ordinal); + using var withoutEndObj = PdfDocument.Open(bare, ObjectGuard); + using var whole = PdfDocument.Open(ended, ObjectGuard); + using var source = new CutShortSource(ended); + using var cut = PdfDocument.Open(source, ObjectGuard, ownsSource: false); + source.CutAt((int)dataEnd + "\nendstream\n".Length); + + ((IPdfStreamDataProvider)withoutEndObj.Reader).IsOwnEndStream(dataEnd, EndObjState.Present, null).Should().BeFalse(); + ((IPdfStreamDataProvider)withoutEndObj.Reader).IsOwnEndStream(dataEnd, EndObjState.Absent, null).Should().BeTrue(); + ((IPdfStreamDataProvider)cut.Reader).IsOwnEndStream(dataEnd, EndObjState.Present, null).Should().BeFalse(); + ((IPdfStreamDataProvider)whole.Reader).IsOwnEndStream(dataEnd, EndObjState.Present, null).Should().BeTrue(); + } + + [Fact] + public void No_place_outside_the_file_is_said_to_end_data_beside_white_space() + { + // Content stream 4, whose /Length MaxObjectLength cut, ends its data in a space right before an endstream object 5 + // proves its own: only the hidden length says whether the space is the data's, so the end may lie beside it. No search + // ends data before the file's first byte or past its last, and there no byte says white space lies beside it (#224). + var file = HexDataEndingInASpaceBeforeAProvenEndStream(); + using var document = PdfDocument.Open(file, ObjectGuard); + var data = document.GetObject(new PdfObjectId(4)).AsStream().Required().Data; + var provider = (IPdfStreamDataProvider)document.Reader; + + data.EndHidden.Should().BeTrue(); + provider.EndsBesideWhiteSpace(data.Position + data.Length).Should().BeTrue(); + provider.EndsBesideWhiteSpace(-1).Should().BeFalse(); + provider.EndsBesideWhiteSpace(0).Should().BeFalse(); + provider.EndsBesideWhiteSpace(file.Length + 1L).Should().BeFalse(); + } + + [Fact] + public void Data_whose_last_byte_the_file_lost_under_the_document_is_no_longer_said_to_end_where_the_guard_hides() + { + // Content stream 4, whose /Length MaxObjectLength cut, ends its data in a space right before an endstream object 5 + // proves its own: its end is the guard's to hide. The file then loses that space under the document, and the stream, + // which the cache let go for 64 other objects, is parsed again: the search and its proof are not read again, but no + // byte says white space ends the data any longer, and its end is not hidden; its data is what the file still holds + // (#224). + var file = HexDataEndingInASpaceBeforeAProvenEndStream(others: 64); + var source = new CutShortSource(file); + using var document = PdfDocument.Open(source, ObjectGuard with { ObjectCacheCapacity = 64 }, ownsSource: true); + var first = document.GetObject(new PdfObjectId(4)).AsStream().Required().Data; + first.EndHidden.Should().BeTrue(); + + for (var number = 30; number < 30 + 64; number++) + { + _ = document.GetObject(new PdfObjectId(number)); + } + + source.CutAt((int)(first.Position + first.Length - 1)); + var again = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + again.Data.Position.Should().Be(first.Position); + again.Data.EndHidden.Should().BeFalse(); + Encoding.ASCII.GetString(again.GetRawBytes().Span).Should().Be("48656C6C6F"); + } + + /// + /// A page whose content stream, object 4, holds 48656C6C6F under /ASCIIHexDecode, the space right before + /// its endstream, which object 5, after it, proves its own; its /Length, object 20, holds 11 after 2,000 + /// spaces, which cuts; then strings, objects 30 and up. + /// + private static byte[] HexDataEndingInASpaceBeforeAProvenEndStream(int others = 0) + { + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, Page), + [20] = file.Object(20, new string(' ', 2000) + "11"), + [4] = file.Object(4, "<< /Length 20 0 R /Filter /ASCIIHexDecode >>\nstream\n48656C6C6F endstream"), + [5] = file.Object(5, "(five)"), + }; + + for (var number = 30; number < 30 + others; number++) + { + rows[number] = file.Object(number, Invariant($"({number})")); + } + + var size = 30 + others; + file.Table(size, rows, Invariant($"/Size {size} /Root 1 0 R")); + return file.Bytes; + } + + [Theory] + [InlineData("xref")] + [InlineData("startxref")] + public void An_endstream_the_data_glues_to_a_section_keyword_where_a_guard_hid_the_length_ends_data_that_is_the_guard_s(string keyword) + { + // Object 4's content spells "endstreamxref" in a string, one token and no endstream keyword: the proof stepped over + // nine bytes and found the section's keyword past them, and the data, cut short there, was taken for the file's, its + // object said to lack its endobj. A token that is no endstream proves nothing: the data is the guard's (#224). + var data = "BT (aendstream" + keyword + " ) Tj ET"; + var file = ContentLengthIn(new string(' ', 2000) + data.Length.ToString(CultureInfo.InvariantCulture), data); + + using var document = PdfDocument.Open(file, ObjectGuard); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.GetRawBytes().ToArray().Should().Equal("BT (a"u8.ToArray()); + stream.Data.CutByGuard.Should().BeTrue(); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void Rows_a_cross_reference_stream_whose_length_a_guard_cut_lacks_are_the_file_s_where_what_follows_its_endstream_proves_it(bool objectAfter) + { + // Stream 5's /Length, object 4, which only its own row places, is one MaxObjectLength cuts before its value, and its + // data lacks the page's row. Its endobj is followed by object 8, which only its own rows place, or by the startxref: + // either proves its endstream its own. The chain asked the sections read before it alone, and took the row the data + // lacks for the guard's, the page unread; a load of the stream then kept the chain's answer. The row is the file's + // to lack, as read whole: the index is rebuilt, the page read (#224). + var file = ChainFiles.AloneWithACutLengthLackingItsLastRow(objectAfter); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + whole.Reader.Structure.Sections[0].Incomplete.Should().BeTrue(); + document.Reader.Structure.Sections[0].Incomplete.Should().BeTrue(); + document.Reader.Structure.Sections[0].CutByLimit.Should().BeFalse(); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + document.WasRepaired.Should().BeTrue(); + document.GetObject(new PdfObjectId(5)).AsStream().Required().Data.CutByGuard.Should().BeFalse(); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void The_nearest_object_a_cross_reference_stream_s_rows_place_past_its_data_proves_its_endstream_whatever_their_order(bool typeField) + { + // As above, but for objects 8, 9 and 10, which follow stream 5 in that order and which only its rows place, 9 first, + // then 8, then 10 — with a type field, or without, every row then of type 1. The object that follows the stream's + // endobj is the nearest of them, 8, which proves the endstream its own whichever of the rows places it: the row the + // data lacks is the file's to lack, as read whole, the index rebuilt, the page read (#224). + var file = ChainFiles.AloneWithACutLengthBeforeObjectsItsRowsPlace(typeField); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + whole.Reader.Structure.Sections[0].Incomplete.Should().BeTrue(); + document.Reader.Structure.Sections[0].Incomplete.Should().BeTrue(); + document.Reader.Structure.Sections[0].CutByLimit.Should().BeFalse(); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + document.WasRepaired.Should().BeTrue(); + document.GetObject(new PdfObjectId(8)).Should().BeOfType().Which.ToText().Should().Be("eight"); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject, PdfDiagnosticCodes.XRefRebuilt); + } + + [Theory] + [InlineData(1, false)] + [InlineData(5, false)] + [InlineData(1, true)] + [InlineData(5, true)] + public void Rows_a_cross_reference_stream_whose_length_a_guard_cut_lacks_are_the_guard_s_where_nothing_read_proves_its_endstream(int updates, bool glued) + { + // Stream 5's /Length, object 4, which only its own row places, is one MaxObjectLength cuts before its value, and its + // data lacks the page's row; 100 spaces part its endstream from its endobj, which the update's first object follows. + // Read on to, that proves the endstream its own: the row is the file's to lack, as read whole, the index rebuilt, the + // page read. Five updates whose data runs over 500,000 spaces to an endstream they share read the file about once + // each, as much as the document's searches may (EndStreamSearchPasses): nothing is read past stream 5's endstream, + // which nothing then proves its own, and where its data ends is the guard's — the page's row with it, which reads as + // null until the guard is lifted. So it is where 60 characters glued to the endstream make it no keyword, which + // proves nothing, whether the searches can still read past it or not (#224). + var exhausted = updates == 5; + var guards = exhausted || glued; + var file = ChainFiles.UpdatesSpendingTheSearches(updates, padding: 500_000, cutTarget: true, glued); + using var document = PdfDocument.Open(file, ObjectGuard); + var section = document.Reader.Structure.Sections[^1]; + + (document.Reader.EndStreamSearchBytes >= PdfFileReader.EndStreamSearchPasses * file.Length).Should().Be(exhausted); + document.Reader.Structure.Sections.Should().HaveCount(updates + 1).And.OnlyContain(entry => entry.State == XRefSectionState.Read); + section.CutByLimit.Should().Be(guards); + section.Incomplete.Should().Be(!guards); + + if (guards) + { + document.GetObject(new PdfObjectId(3)).Should().BeOfType(); + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + return; + } + + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + document.WasRepaired.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject, PdfDiagnosticCodes.XRefRebuilt); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_group_an_unproven_endstream_parts_where_a_guard_cut_the_length_leaves_its_rows_to_the_guard(bool carry) + { + // Stream 7's /Length, which only its own row places, is one MaxObjectLength parts after its first digit, and its + // ASCII85 text spells an endstream four digits into the group that holds the page's row's last bytes: nothing after + // it proves it the stream's own, and where the data ends is the guard's. Decoded as data whose end the file gives, + // the group was completed with u's, which guessed its bytes — and carried into the page's row's generation, read + // then as 1 —: the row the parted group holds is the guard's, the page read as null until the guard is lifted (#224). + var file = ChainFiles.Ascii85RowsPartedBySpelledEndStream(carry); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + whole.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.GetObject(new PdfObjectId(3)).Should().BeOfType(); + document.Reader.Structure.Sections.Should().ContainSingle().Which.CutByLimit.Should().BeTrue(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId).Should().NotContain(PdfValidationRuleIds.XRefGenerationMismatch); + } + + [Theory] + [InlineData(20)] + [InlineData(49)] + public void A_predicted_row_an_unproven_endstream_parts_where_a_guard_cut_the_length_is_undone_as_far_as_it_goes(int columns) + { + // Stream 7's /Length, object 5, which the newer table places, is one MaxObjectLength parts after its first digit, and + // its rows, Flate behind a PNG predictor, spell an endstream after the catalog's: the data up to it decodes to the + // catalog's row behind its tag, shorter than one predicted row. Decoded as data whose end the file gives, it was left + // as decoded, the tag read as the catalog's row's type, the catalog freed and the index rebuilt. Where the end is + // the guard's, the row is undone as far as the data goes, and the catalog read (#224). + var file = ChainFiles.PredictedRowsPartedBySpelledEndStream(columns); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.GetObject(new PdfObjectId(1)).AsDictionary().Required().IsOfType(PdfName.Catalog).Should().BeTrue(); + } + + [Fact] + public void Rows_a_predicted_row_undone_as_far_as_it_goes_completes_are_read_though_no_guard_stopped_the_decoding_before() + { + // As above, the rows of objects 0 to 3 before those that spell an endstream, 21 columns: the data up to it decodes to + // a predicted row that holds three, and a second, parted, that holds the page's. Decoded as data whose end the file + // gives, the parted row was dropped; decoded again knowing its end hidden, it is undone as far as it goes, and holds + // the page's row whole. That the data decoded again holds no more rows than the decoding before is for a decoding a + // guard stopped, whose bound a filter after a dropped group may be given more room under (#220). + var file = ChainFiles.PredictedRowsPartedBySpelledEndStream(21, pageBefore: true); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + whole.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Theory] + [InlineData(29)] + [InlineData(20)] + public void A_filter_run_out_where_a_guard_s_cut_of_the_length_leaves_the_data_says_nothing_of_the_file(int free) + { + // As above, the rows compressed in a stored block. With 29 free rows the data is 284 bytes, which MaxObjectLength + // parts after its 28, where the rows' second spelled endstream confirms it: the data is taken again to it; with 20, + // 221, parted after its 22, which nothing confirms: the data stays at the first spelled endstream. Either way it ends + // where the guard hides the length, and the Flate run out there was reported as the file's fault (#224). + var file = ChainFiles.UpdatePlacingAPartedLength(free, compressed: true); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[1].CutByLimit.Should().BeTrue(); + } + + [Fact] + public void Rows_data_taken_again_to_an_integer_a_guard_parted_in_the_own_row_s_object_still_lacks_are_the_guard_s() + { + // As above, but the stream's own row places object 5, whose 420 MaxObjectLength parts after its 42, which the rows' + // second spelled endstream confirms: the 54 rows the data taken again lacks are the guard's, not the file's (#224). + var file = ChainFiles.OwnRowPlacingAPaddedLength(throughReference: false); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + whole.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[0].CutByLimit.Should().BeTrue(); + document.Reader.Structure.Sections[0].Incomplete.Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().BeFalse(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_whole_reference_a_guard_cut_after_leads_the_own_row_s_length_on_as_a_whole_read_follows_it(bool lackingRow) + { + // The stream's own row places object 5, which MaxObjectLength cuts after "6 0 R", whole; object 6, which a newer + // section places, holds the length. The data is taken again to it, as read whole: the guard hid nothing the length + // needs, and the rows the data lacks — one, where the /Index declares one more — are the file's, as read whole. They + // went to the guard (#224). + var file = ChainFiles.OwnRowPlacingAPaddedLength(throughReference: true, lackingRow); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + whole.Diagnostics.Should().BeEmpty(); + + foreach (var read in (ReadOnlySpan)[whole, document]) + { + var stream = read.Reader.Structure.Sections[1]; + stream.CutByLimit.Should().BeFalse(); + stream.Incomplete.Should().Be(lackingRow); + read.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + new PdfValidator().Validate(read).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().Be(lackingRow); + } + } + + [Theory] + [InlineData(false, false)] + [InlineData(true, false)] + [InlineData(false, true)] + [InlineData(true, true)] + public void A_whole_reference_a_guard_cut_after_leads_a_chain_on_whichever_section_reads_it_first(bool bothNameIt, bool lackingRow) + { + // Object 5 holds "6 0 R", which MaxObjectLength cuts after; object 6, which the newer stream places, holds the length + // the older stream's /Length gives through 5. With the newer stream's /Length naming 5 too, the chain reads 5 before + // any section places 6, and what the cut left of it was kept as that read judged it: the older stream's data could + // not be taken again, its rows past the spelled endstream went to the guard and the file was said to give a length + // no endstream confirms; with a row lacking, the section was cut where it is the file's to lack (#224). The guard hides + // nothing either length needs: each reads as read whole. + var file = ChainFiles.UpdatesReachingALengthThroughACutReference(bothNameIt, lackingRow); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + whole.Diagnostics.Should().BeEmpty(); + + foreach (var read in (ReadOnlySpan)[whole, document]) + { + var sections = read.Reader.Structure.Sections; + sections.Should().HaveCount(3); + sections.Should().AllSatisfy(section => section.CutByLimit.Should().BeFalse()); + sections[2].Incomplete.Should().Be(lackingRow); + read.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + new PdfValidator().Validate(read).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().Be(lackingRow); + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_value_the_chain_reads_through_a_whole_reference_a_guard_cut_after_is_read_and_reaches_the_guard(bool throwOnLimit) + { + // Stream 7's /W names object 5, which MaxObjectLength cuts after "9 0 R", whole; object 9, which the newer table + // places, holds [1 4 2]. The chain ended at object 5 for the guard, the section cut before its first row, though the + // guard hid nothing the value needs. It reads through, as a load reads through it, and reaches the guard once the + // chain is read: reported, or thrown at opening when the caller asks for it (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W 5 0 R /Length {length}", + "9 0 R" + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + placedToo: new Dictionary { [9] = "[1 4 2]" }); + + if (throwOnLimit) + { + var opening = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + opening.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + return; + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections.Should().HaveCount(2).And.OnlyContain(section => !section.CutByLimit); + document.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void An_object_read_through_for_a_place_that_holds_no_section_reaches_the_guard_only_where_a_kept_value_reads_it(bool kept) + { + // The newer table's /Prev lands 2 bytes into object 8, a stream whose /Length names object 5, which MaxObjectLength + // cuts after "6 0 R", whole: the chain reads object 8 as a section, refuses it, and finds stream 7 after it. What it + // read for object 8 is no section's, and nothing of it was to be kept, but object 5, read through, was loaded at + // opening, the guard reported, or thrown. Where stream 7's /Size names object 5 too, the chain keeps its value, and + // the guard is reached as before (#224, #263). + var file = ChainFiles.UnderAnUpdate( + kept ? "/Type /XRef /Size 5 0 R /W [1 4 2] /Length {length}" : "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + "6 0 R" + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + placedToo: new Dictionary { [6] = "8", [8] = "<< /Length 5 0 R >>\nstream\nabc\nendstream" }); + var text = Encoding.Latin1.GetString(file); + var stream = text.IndexOf("7 0 obj", StringComparison.Ordinal); + var eight = text.IndexOf("8 0 obj", StringComparison.Ordinal); + file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {eight + 2} "), StringComparison.Ordinal)); + + using var document = PdfDocument.Open(file, ObjectGuard); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + kept ? [PdfDiagnosticCodes.XRefOffsetAdjusted, PdfDiagnosticCodes.LimitObject] : [PdfDiagnosticCodes.XRefOffsetAdjusted]); + + var opening = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }).Dispose(); + + if (kept) + { + opening.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + else + { + opening.Should().NotThrow(); + } + } + + [Fact] + public void What_the_offset_a_section_is_named_at_holds_keeps_what_it_read_through_as_a_load_would() + { + // As above, the /Prev naming object 8's header exactly: the place the chain names is read as a load reads it, and + // keeps all of its reading though it holds no section — its stream's length reported, and object 5, read through, + // loaded once the chain is read, the guard reported, or thrown. Only a place guessed near it leaves nothing (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + "6 0 R" + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + placedToo: new Dictionary { [6] = "8", [8] = "<< /Length 5 0 R >>\nstream\nabc\nendstream" }); + var text = Encoding.Latin1.GetString(file); + var stream = text.IndexOf("7 0 obj", StringComparison.Ordinal); + var eight = text.IndexOf("8 0 obj", StringComparison.Ordinal); + file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {eight} "), StringComparison.Ordinal)); + + using var document = PdfDocument.Open(file, ObjectGuard); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + PdfDiagnosticCodes.StreamLengthInvalid, PdfDiagnosticCodes.XRefOffsetAdjusted, PdfDiagnosticCodes.LimitObject); + + var opening = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }).Dispose(); + opening.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Fact] + public void A_size_beside_an_index_that_names_an_object_a_guard_cut_after_a_whole_reference_is_not_read() + { + // Stream 7's /Size names object 5, which MaxObjectLength cuts after "9 0 R", whole, beside an /Index the rows are + // numbered by: the chain read it through, though it numbers nothing, and loaded object 5 at opening, reporting the + // guard, or throwing it. It is left as written, as the rows are read without it (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 5 0 R /Index [0 8] /W [1 4 2] /Length {length}", + "9 0 R" + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + placedToo: new Dictionary { [9] = "8" }); + + using var document = PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.Structure.Sections.Should().HaveCount(2).And.OnlyContain(section => section.State == IO.XRef.XRefSectionState.Read); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_size_beside_an_index_in_the_newest_section_is_not_read_to_make_room_for_an_older_section_s_rows(bool olderStream) + { + // As above, stream 7 the newest section, over an older table or stream: its dictionary, the index's trailer, gave the + // /Size the index made room by for the older section's rows, read through to object 5 and loaded at opening. Room is + // made by a /Size as written, which ISO 32000-1 makes direct (Table 15, 7.5.8.2): a reference gives none. + var file = ChainFiles.StreamSizedThroughAReferenceOverAnOlderSection(olderStream); + + using var document = PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.Structure.Sections.Should().HaveCount(2).And.OnlyContain(section => section.State == IO.XRef.XRefSectionState.Read); + document.GetObject(new PdfObjectId(3)).AsDictionary().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Fact] + public void Rows_past_an_endstream_they_spell_are_the_guard_s_when_it_cut_the_deferred_length_before_its_value() + { + // As above, object 4's value lying past the cut: how far the data runs is unknown, and the rows past the endstream are + // the guard's. Object 8 read as the first revision wrote it, its section said to lack them (#224). + var file = ChainFiles.UpdateWithIndirectLength(spellingEndStream: true, four: new string(' ', 3000) + "{length}"); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.GetObject(new PdfObjectId(8)).Should().BeOfType(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections[0].CutByLimit.Should().BeTrue(); + document.Reader.Structure.Sections[0].Incomplete.Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.XRefSectionMalformed).Should().BeFalse(); + } + + [Fact] + public void Rows_past_a_spelled_endstream_are_the_file_s_when_the_length_s_chain_gives_a_wrong_length() + { + // As above, object 9 read whole, with no guard, and 1,000 more than the data's length: nothing confirms it, and the + // rows past the endstream the rows spell are the file's to lack — the section incomplete, the index rebuilt (#220). + var file = ChainFiles.UpdatePlacingAPaddedLength(chained: true, padding: 0, lengthOff: 1000); + using var document = PdfDocument.Open(file); + + document.GetObject(new PdfObjectId(3)); + + var section = document.Reader.Structure.Sections[1]; + section.Incomplete.Should().BeTrue(); + section.CutByLimit.Should().BeFalse(); + document.Diagnostics.Should().NotContain(entry => entry.Code.StartsWith("limit.", StringComparison.Ordinal)); + } + + [Theory] + [InlineData(31)] + [InlineData(32)] + public void A_length_whose_chain_ends_at_its_last_object_a_guard_cut_is_the_guard_s(int objects) + { + // Content stream 4's /Length names object 100, which names the next, the last, after 3,000 spaces, holding 4: the + // parse resolves a /Length through 32 objects, one more than a reference, and the last it reads, cut, is the guard's + // — the 31st no longer taken for the target unread, nor the 32nd left past it (#224). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, "<< /Length 100 0 R >>\nstream\n0 0 m\nendstream"), + }; + var last = 100 + objects - 1; + + for (var number = 100; number <= last; number++) + { + rows[number] = file.Object(number, number < last ? Invariant($"{number + 1} 0 R") : new string(' ', 3000) + "4"); + } + + file.Table(last + 1, rows, Invariant($"/Size {last + 1} /Root 1 0 R")); + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().Length.Should().Be(5); + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.StreamLengthInvalid); + } + + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + document.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().Length.Should().Be(5); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.TryGetStreamLengthFault(4, out _).Should().BeFalse(); + } + + [Theory] + [InlineData(31, "cut")] + [InlineData(32, "cut")] + [InlineData(31, "left out of the index")] + [InlineData(32, "left out of the index")] + public void A_missing_endstream_names_the_last_object_of_a_length_s_chain_a_guard_cut_or_left_out_of_the_index(int objects, string guard) + { + // As above, content stream 4 having lost its endstream, and the last object of its /Length's chain cut after 3,000 + // spaces — or placed only by an older table, which MaxXRefSectionCount leaves unread: the fault and the report of the + // missing endstream name that object, the 32nd as the 31st, and not the one before it, which no guard cut and the + // index holds (#224). + var file = new HandWrittenFile(); + var last = 100 + objects - 1; + var unindexed = guard == "left out of the index"; + var rows = new Dictionary(); + var trailer = Invariant($"/Size {last + 1} /Root 1 0 R"); + + if (unindexed) + { + var older = file.Table(last + 1, new Dictionary { [last] = file.Object(last, "5") }, Invariant($"/Size {last + 1}")); + trailer += Invariant($" /Prev {older}"); + } + else + { + rows[last] = file.Object(last, new string(' ', 3000) + "5"); + } + + rows[1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + rows[2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"); + rows[4] = file.Object(4, "<< /Length 100 0 R >>\nstream\n0 0 m"); + + for (var number = 100; number < last; number++) + { + rows[number] = file.Object(number, Invariant($"{number + 1} 0 R")); + } + + // The newer table numbers no row for the last object when only the older one places it. + file.Table(unindexed ? last : last + 1, rows, trailer); + var options = unindexed ? new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxXRefSectionCount = 1 } } : ObjectGuard; + using var document = PdfDocument.Open(file.Bytes, options); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal("0 0 m"u8.ToArray()); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated).Which.Message.Should().StartWith( + Invariant($"The stream's /Length names object {last} 0, which one of the reader's limits {guard}, and no endstream follows its data")); + document.Reader.TryGetStreamLengthFault(4, out var fault).Should().BeTrue(); + fault.Form.Should().Be(unindexed ? StreamLengthForm.ObjectUnindexed : StreamLengthForm.ObjectCut); + fault.Reference.Should().Be(new PdfObjectId(last)); + } + + [Theory] + [InlineData(31)] + [InlineData(32)] + public void A_cross_reference_stream_s_length_whose_chain_ends_at_its_last_object_a_guard_cut_waits_for_the_chain(int objects) + { + // Stream 7's /Length names object 5, which the newer table places, and which names object 100, the next, and so on; + // the last, after 3,000 spaces, holds the length. The chain read follows as many objects as the parse resolves a + // /Length through, so the cut last one makes it wait for the chain, which then leaves it to the guard (#220, #224). + var placed = new Dictionary(); + var last = 100 + objects - 2; + + for (var number = 100; number <= last; number++) + { + placed[number] = number < last ? Invariant($"{number + 1} 0 R") : new string(' ', 3000) + "56"; + } + + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", "100 0 R", compressed: false, predicted: false, placed: true, placedToo: placed); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData(31)] + [InlineData(32)] + public void A_cross_reference_stream_with_no_endstream_whose_length_s_chain_ends_at_its_last_object_a_guard_cut_is_the_guard_s(int objects) + { + // As above, stream 7 having lost its endstream: the fault that says its data ends nowhere follows the /Length as far + // as the parse resolves one, so the 32nd object, cut, leaves the section to the guard — not a chain of references + // that reaches no value, which rebuilt the index (#220, #224). + var placed = new Dictionary(); + var last = 100 + objects - 2; + + for (var number = 100; number <= last; number++) + { + placed[number] = number < last ? Invariant($"{number + 1} 0 R") : new string(' ', 3000) + "56"; + } + + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length 5 0 R", "100 0 R", compressed: false, predicted: false, placed: true, placedToo: placed); + var at = Encoding.Latin1.GetString(file).IndexOf("endstream", StringComparison.Ordinal); + Encoding.ASCII.GetBytes("endstrXam").CopyTo(file, at); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[1].CutByLimit.Should().BeTrue(); + document.Reader.Structure.Sections[1].Fault.Should().EndWith(Invariant($"the reference {last} 0 R, which one of the reader's limits cut")); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData(31, false)] + [InlineData(32, false)] + [InlineData(31, true)] + [InlineData(32, true)] + public void Rows_past_a_spelled_endstream_are_the_guard_s_when_the_length_s_chain_ends_at_its_last_object_a_guard_cut(int objects, bool ownRow) + { + // Stream 7's /Length names object 5, which the newer table places — or object 4, which only stream 7's own row + // places —, which names object 100, the next, and so on; the last, after 3,000 spaces, holds the length, and the rows + // spell an endstream. Whether the length leads to a cut follows it as far as the parse resolves one, from the object + // the newer table places or from what the own row's object holds: the 32nd, cut, leaves the rows past the endstream + // to the guard — the page, whose row is one, reads as null —, where a shorter walk gave them to the file, the section + // incomplete and the index rebuilt (#220, #224). + var file = ChainFiles.UpdatePlacingAPaddedLengthChain(objects, ownRow); + + using (var whole = PdfDocument.Open(file)) + { + whole.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.GetObject(new PdfObjectId(3)).Should().BeOfType(); + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + var section = document.Reader.Structure.Sections[1]; + section.CutByLimit.Should().BeTrue(); + section.Incomplete.Should().BeFalse(); + + // Loaded, the stream ends at the endstream its rows spell, which nothing past its object's end proves its own — an + // object the index places, the end of the file, xref or startxref —: the data is the guard's, and the object is not + // said to lack its endobj, which is the guard's to hide. + document.GetObject(new PdfObjectId(7)).AsStream().Required().Data.CutByGuard.Should().BeTrue(); + document.Reader.IsEndObjMissing(7, out _).Should().BeFalse(); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData(31)] + [InlineData(32)] + public void A_prev_whose_chain_ends_at_its_last_object_a_guard_cut_ends_the_chain_as_the_guard_s(int objects) + { + // The newer table's /Prev names object 21, which names the next, the last, after 3,000 spaces, holding the older + // table's offset: a /Prev reads through 32 objects, as a whole read does, and the last, cut, ends the chain for the + // guard, which the 32nd did not: it was no offset, and the index was rebuilt (#263). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 1 1] /Resources << >> >>"), + }; + var older = file.Table(4, new Dictionary(rows), "/Size 4 /Root 1 0 R"); + var newer = new Dictionary(); + var last = 21 + objects - 1; + + for (var number = 21; number <= last; number++) + { + newer[number] = file.Object( + number, number < last ? Invariant($"{number + 1} 0 R") : new string(' ', 3000) + older.ToString(CultureInfo.InvariantCulture)); + } + + var table = file.Position; + var text = new StringBuilder(Invariant($"xref\n0 1\n0000000000 65535 f \n21 {objects}\n")); + + foreach (var number in newer.Keys) + { + text.Append(CultureInfo.InvariantCulture, $"{newer[number]:D10} 00000 n \n"); + } + + text.Append(CultureInfo.InvariantCulture, $"trailer\n<< /Size {last + 1} /Root 1 0 R /Prev 21 0 R >>\nstartxref\n{table}\n%%EOF\n"); + file.Raw(text.ToString()); + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.WasRepaired.Should().BeFalse(); + whole.Reader.Structure.Sections.Should().HaveCount(2); + } + + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections.Should().ContainSingle().Which.ChainHiddenByGuard.Should().NotBeNull(); + } + + [Theory] + [InlineData(false, false)] + [InlineData(true, false)] + [InlineData(false, true)] + public void Rows_past_a_spelled_endstream_are_the_guard_s_when_the_length_leads_to_an_object_a_newer_section_places_and_it_cut( + bool chained, bool spelledFirst) + { + // Stream 7's /Length leads to an object the newer table places — object 5 itself, or object 9 through object 4, + // which only stream 7's own row places — and MaxObjectLength cuts before its value: the rows past the endstream they + // spell are the guard's, as when the cut object is the one the section's own row places (#224) — the data taken to it + // holding no row whole too, the spelled rows first. Read whole, the file is clean. + var file = ChainFiles.UpdatePlacingAPaddedLength(chained, spelledFirst: spelledFirst); + + using (var whole = PdfDocument.Open(file)) + { + whole.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + var section = document.Reader.Structure.Sections[1]; + section.CutByLimit.Should().BeTrue(); + section.Incomplete.Should().BeFalse(); + new PdfValidator().Validate(document).Findings.Should().NotContain(finding => finding.Severity == PdfValidationSeverity.Error); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData("Prev")] + [InlineData("XRefStm")] + public void A_section_offset_a_guard_cut_ends_the_chain_as_the_guard_s(string key) + { + // The newer section's /Prev, object 9, or its /XRefStm, object 6, is one that section places and MaxObjectLength cuts + // before its value: the section it names was said to be missing, and the index rebuilt. The chain ends there for the + // guard, which is reached once the chain is read — thrown at opening when the caller asks for it (#263). The + // validation says what it names was not checked — the older sections, or the stream of the hybrid's revision —, and + // judges nothing the chain they end would hold. + var reference = key == "Prev" ? "9 0 R" : "6 0 R"; + var file = key == "Prev" + ? ChainFiles.UpdateNamingItsPrevious(reference, padding: 3000) + : ChainFiles.HybridNamingItsStream(reference, padding: 3000); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections.Should().ContainSingle(); + var report = new PdfValidator().Validate(document); + report.Contains(PdfValidationRuleIds.XRefSectionNotFound).Should().BeFalse(); + report.Contains(PdfValidationRuleIds.FileSizeWrong).Should().BeFalse(); + var section = document.Reader.Structure.Sections[0].Offset; + report.Findings.Where(finding => finding.RuleId == PdfValidationRuleIds.XRefCheckedInPart).Select(finding => finding.Message).Should().Equal( + key == "Prev" + ? $"The cross-reference section at offset {section} names older sections, but its /Prev is the reference {reference}, which one of the reader's limits cut: they were not checked." + : $"The cross-reference section at offset {section} names the cross-reference stream of its revision, but its /XRefStm is the reference {reference}, which one of the reader's limits cut: that stream, and any section older than it, were not checked."); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Fact] + public void A_prev_naming_an_object_whose_header_a_guard_cut_ends_the_chain_as_the_guard_s() + { + // The newer table's /Prev is object 9, which holds the older table's offset; 3,000 blanks part its generation from + // its obj, white space a header may hold, and MaxObjectLength cuts the header itself before its keyword: the chain + // reads nothing of the object, its value no more than its header. The chain ends there for the guard, as where the + // guard cuts the value, and the guard is reached once the chain is read, where loading object 9 reaches it — thrown at + // opening when the caller asks for it (#263). The validation says the older sections, and the header of object 9, + // were not checked. That load then rebuilds the index, as a read of a header past the guard does, which this test + // leaves aside (#292). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + [3] = file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 1 1] /Resources << >> >>"), + }; + var older = file.Table(4, rows, "/Size 4 /Root 1 0 R"); + var four = file.Object(4, "(an update)"); + var nine = file.Position; + file.Raw(Invariant($"9 0{new string(' ', 3000)}obj\n{older}\nendobj\n")); + var newer = file.Position; + file.Raw(Invariant( + $"xref\n4 1\n{four:D10} 00000 n \n9 1\n{nine:D10} 00000 n \ntrailer\n<< /Size 10 /Root 1 0 R /Prev 9 0 R >>\nstartxref\n{newer}\n%%EOF\n")); + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.WasRepaired.Should().BeFalse(); + whole.Reader.Structure.Sections.Should().HaveCount(2); + } + + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + + var section = document.Reader.Structure.Sections.Should().ContainSingle().Which; + section.Offset.Should().Be(newer); + section.ChainHiddenByGuard.Should().Be("its /Prev is the reference 9 0 R, which one of the reader's limits cut"); + document.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.LimitObject).Select(entry => entry.ToString()).Should().Equal( + Invariant($"Warning limit.object at {nine}: Object 9 runs past 1 KB, its stream data aside; only what lies within it was read. Raise PdfReaderLimits.MaxObjectLength to read past it.")); + var report = new PdfValidator().Validate(document); + report.Contains(PdfValidationRuleIds.XRefSectionNotFound).Should().BeFalse(); + report.Findings.Where(finding => finding.RuleId == PdfValidationRuleIds.XRefCheckedInPart).Select(finding => finding.Message).Should().Equal( + Invariant($"The cross-reference section at offset {newer} names older sections, but its /Prev is the reference 9 0 R, which one of the reader's limits cut: they were not checked."), + Invariant($"Reading the header of object 9 where its entry places it, at offset {nine}, or near it, the reader reached PdfReaderLimits.MaxObjectLength: the entry was not checked. Raising the limit lets it be.")); + + var throwing = () => PdfDocument.Open(file.Bytes, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Fact] + public void A_cross_reference_stream_s_prev_a_guard_cut_ends_the_chain_as_the_guard_s() + { + // Cross-reference stream 10's /Prev, object 9, is one its own rows place, and MaxObjectLength cuts before its value: + // the chain ends there for the guard, as at a table's /Prev, thrown at opening when the caller asks for it (#263). + // Read whole, the stream's /Size 13 is one more than object 12, which only the older table indexes, and object 4 is + // a page the tree does not list: the validation says the table was not checked, and judges neither on the stream. + var file = ChainFiles.StreamUpdateHidingItsHighestNumber(padding: 3000); + + using (var whole = PdfDocument.Open(file)) + { + whole.Reader.Structure.Sections.Should().HaveCount(2); + var report = new PdfValidator().Validate(whole); + report.Contains(PdfValidationRuleIds.FileSizeWrong).Should().BeFalse(); + report.Contains(PdfValidationRuleIds.PageTreePageOrphaned).Should().BeTrue(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + var section = document.Reader.Structure.Sections.Should().ContainSingle().Which; + section.Kind.Should().Be(XRefSectionKind.Stream); + section.ChainHiddenByGuard.Should().NotBeNull(); + var guarded = new PdfValidator().Validate(document); + guarded.Findings.Where(finding => finding.RuleId == PdfValidationRuleIds.XRefCheckedInPart).Select(finding => finding.Message).Should().Equal( + $"The cross-reference section at offset {section.Offset} names older sections, but its /Prev is the reference 9 0 R, which one of the reader's limits cut: they were not checked."); + guarded.Contains(PdfValidationRuleIds.FileSizeWrong).Should().BeFalse(); + guarded.Contains(PdfValidationRuleIds.PageTreePageOrphaned).Should().BeFalse(); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData("9 0 R", true)] + [InlineData("9 0 R", false)] + [InlineData("12 0 R", false)] + [InlineData("4.0", false)] + public void A_hybrid_stream_s_own_prev_is_not_followed_the_chain_going_on_through_the_table_s(string prev, bool guarded) + { + // A hybrid file's stream gives a /Prev of its own, which the chain does not follow: it goes on through the table's, + // as ISO 32000-1 (7.5.8.4) describes. Read anyway, one that leads to an object MaxObjectLength cuts ended the chain + // for the guard, the older table lost, and one no section places was said to name a missing section (#263). The + // rule on trailer values judges it as the file wrote it, a reference or a real where Table 17 asks for a direct + // integer, without following it: following it reached the guard the reading never met. + var file = ChainFiles.HybridWithStreamPrev(prev); + using var document = PdfDocument.Open(file, guarded ? ObjectGuard : null); + + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections.Should().HaveCount(3); + document.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + document.Diagnostics.Should().BeEmpty(); + var report = new PdfValidator().Validate(document); + report.Contains(PdfValidationRuleIds.XRefSectionNotFound).Should().BeFalse(); + report.Contains(PdfValidationRuleIds.XRefCheckedInPart).Should().BeFalse(); + + using var judged = PdfDocument.Open(file, guarded ? ObjectGuard : null); + var trailers = new ValidationProfile("trailers", 1, [new TrailerValueWrongRule()]); + new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(judged).Findings.Select(finding => finding.Message) + .Should().ContainSingle().Which.Should().Contain(prev == "4.0" ? "gives its /Prev as a real number" : $"writes its /Prev as the reference {prev}"); + judged.Diagnostics.Should().BeEmpty(); + } + + [Theory] + [InlineData("stream", "writes its /Prev as the reference 9 0 R, to a real number: ISO 32000-1 makes it direct (7.5.8.2), and Table 17 of ISO 32000-1 an integer.")] + [InlineData("table", "writes its /Prev as the reference 9 0 R, to a real number: ISO 32000-1 makes it direct (Table 15), and Table 15 of ISO 32000-1 an integer.")] + [InlineData("hybrid stream", "writes its /Prev as the reference 9 0 R, where ISO 32000-1 makes it direct (7.5.8.2).")] + public void A_prev_written_as_a_reference_to_a_real_is_judged_through_it_only_where_the_reader_follows_it(string section, string expected) + { + // Object 9, which a /Prev names, holds a real with no fractional part. The reader follows a cross-reference stream's + // /Prev and a table's: the rule says both faults of each. A hybrid file's stream's own /Prev it does not follow, and + // the rule judges it as written, a reference (#263). + var file = section switch + { + "stream" => ChainFiles.StreamUpdateHidingItsHighestNumber(padding: 0, real: true), + "table" => ChainFiles.UpdateNamingItsPrevious("9 0 R", real: true), + _ => ChainFiles.HybridWithStreamPrev("9 0 R", padding: 0, nine: "4.0"), + }; + using var document = PdfDocument.Open(file); + var trailers = new ValidationProfile("trailers", 1, [new TrailerValueWrongRule()]); + + new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(document).Findings.Select(finding => finding.Message) + .Should().ContainSingle().Which.Should().EndWith(expected); + } + + [Theory] + [InlineData("/Filter /FlateDecode /DecodeParms << /Columns 5 0 R >>", true, false)] + [InlineData("/Filter /FlateDecode /DecodeParms << /EarlyChange 5 0 R >>", true, false)] + [InlineData("/DecodeParms 5 0 R", false, false)] + [InlineData("/Filter /FlateDecode /DecodeParms << /Predictor 12 /Columns 5 0 R >>", true, true)] + public void The_rule_on_trailer_values_follows_a_parameter_only_where_decoding_reads_it(string entries, bool compressed, bool read) + { + // Stream 7's /DecodeParms, or a parameter in it, is a reference to object 5, which holds 7.0. Decoding reads /Columns + // only past a predictor, /EarlyChange only for LZW, and /DecodeParms only for a Flate or LZW step: the rule followed + // each all the same, judged a real decoding never read, and reached the guard the reading never met where + // MaxObjectLength cuts object 5 (#263). Where decoding reads it, it is followed as before. + var trailers = new ValidationProfile("trailers", 1, [new TrailerValueWrongRule()]); + var file = ChainFiles.UnderAnUpdate($"/Type /XRef /Size 8 /W [1 4 2] {entries} /Length {{length}}", "7.0", compressed, predicted: read, placed: true); + using var document = PdfDocument.Open(file); + + new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(document).Findings + .Any(finding => finding.Message.Contains("as a real number", StringComparison.Ordinal)).Should().Be(read); + + if (read) + { + return; + } + + var padded = ChainFiles.UnderAnUpdate( + $"/Type /XRef /Size 8 /W [1 4 2] {entries} /Length {{length}}", new string(' ', 3000) + "7.0", compressed, predicted: false, placed: true); + using var guarded = PdfDocument.Open(padded, ObjectGuard); + + guarded.Diagnostics.Should().BeEmpty(); + new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(guarded); + guarded.Diagnostics.Should().BeEmpty(); + } + + [Fact] + public void The_rule_on_trailer_values_follows_no_parameter_past_the_step_MaxFilterChainLength_ends_decoding_at() + { + // Stream 7's second Flate step reads /Columns past its predictor, in object 6, which holds 7.0; MaxFilterChainLength + // lets the first step alone run. Decoding never read the parameter, and the rule, held to the bound the reader read + // under, judges nothing it leads to (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Filter [/FlateDecode /FlateDecode] /DecodeParms [null 5 0 R] /Length {length}", + "<< /Predictor 12 /Columns 6 0 R >>", + compressed: true, + predicted: false, + placed: true, + placedToo: new Dictionary { [6] = "7.0" }); + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxFilterChainLength = 1 } }); + var trailers = new ValidationProfile("trailers", 1, [new TrailerValueWrongRule()]); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitFilterChain); + new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(document).Findings.Should().BeEmpty(); + } + + [Fact] + public void The_rule_on_trailer_values_judges_a_parameter_decoding_read_whatever_the_cache_let_go_before_validating() + { + // Stream 7's /DecodeParms, object 5, gives /Columns past its predictor as a reference to object 100, which holds 7.0. + // Read before validating, with 63 objects after it, object 5 left a cache of 64: the rule read it again, a reference + // to /Columns decoding had not followed, and lost the finding decoding's own read gives (#263). + var placed = new Dictionary { [100] = "7.0" }; + + for (var number = 200; number < 270; number++) + { + placed[number] = "(x)"; + } + + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Filter /FlateDecode /DecodeParms 5 0 R /Length {length}", + "<< /Predictor 12 /Columns 100 0 R >>", + compressed: true, + predicted: true, + placed: true, + placedToo: placed); + using var document = PdfDocument.Open(file, new PdfReaderOptions { ObjectCacheCapacity = 64 }); + _ = document.GetObject(new PdfObjectId(5)); + + for (var number = 200; number < 263; number++) + { + _ = document.GetObject(new PdfObjectId(number)); + } + + var trailers = new ValidationProfile("trailers", 1, [new TrailerValueWrongRule()]); + + new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(document).Findings.Select(finding => finding.Message) + .Should().ContainSingle(message => message.EndsWith( + "gives /Columns of its /DecodeParms as a real number, where Table 8 of ISO 32000-1 asks for an integer.", StringComparison.Ordinal)); + } + + [Theory] + [InlineData("7.0")] + [InlineData("padded")] + public void The_rule_on_trailer_values_judges_a_w_element_past_the_third_as_written(string five) + { + // Stream 7's /W gives a fourth element, a reference to object 5, which holds 7.0, or 7 past 3,000 spaces, which + // MaxObjectLength cuts. The reader reads three: the rule followed the fourth all the same, judged a real the reading + // never read, and reached the guard the reading never met. It is judged as written, a reference (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2 5 0 R] /Length {length}", + five == "7.0" ? five : new string(' ', 3000) + "7", + compressed: false, + predicted: false, + placed: true); + using var document = PdfDocument.Open(file, ObjectGuard); + var trailers = new ValidationProfile("trailers", 1, [new TrailerValueWrongRule()]); + + new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(document).Findings.Select(finding => finding.Message) + .Should().ContainSingle().Which.Should().EndWith("writes element 3 of its /W as the reference 5 0 R, where ISO 32000-1 makes it direct (7.5.8.2)."); + document.Diagnostics.Should().BeEmpty(); + } + + [Theory] + [InlineData("Prev", "9 0 R", true)] + [InlineData("XRefStm", "9 0 R", true)] + [InlineData("Prev", "12 0 R", false)] + [InlineData("XRefStm", "12 0 R", false)] + public void A_table_an_xrefstm_names_in_a_stream_s_place_has_its_own_links_left_unread(string key, string value, bool guarded) + { + // The update's /XRefStm names a classic table, where ISO 32000-1 names a cross-reference stream (Table 19): its rows + // are read, and the chain goes on through the update's /Prev, as past a stream. The table's own /Prev or /XRefStm, + // which a stream's is not since #263, was read all the same: one that leads to an object MaxObjectLength cuts ended + // the chain for the guard, the original table and the catalog lost, and one no section places was said to name a + // missing section. The rule on trailer values judges it as the file wrote it. + var file = ChainFiles.HybridNamingATable(key, value); + using var document = PdfDocument.Open(file, guarded ? ObjectGuard : null); + + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections.Should().HaveCount(3); + document.Diagnostics.Should().BeEmpty(); + document.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + var report = new PdfValidator().Validate(document); + report.Contains(PdfValidationRuleIds.XRefSectionNotFound).Should().BeFalse(); + report.Contains(PdfValidationRuleIds.XRefCheckedInPart).Should().BeFalse(); + + using var judged = PdfDocument.Open(file, guarded ? ObjectGuard : null); + var trailers = new ValidationProfile("trailers", 1, [new TrailerValueWrongRule()]); + var findings = new PdfValidator(new PdfValidatorOptions { Profile = trailers }).Validate(judged).Findings.Select(finding => finding.Message); + + // A table's /XRefStm may be a reference (Table 19); its /Prev is direct (Table 15). + findings.Should().HaveCount(key == "Prev" ? 1 : 0).And.OnlyContain(message => message.Contains($"writes its /Prev as the reference {value}", StringComparison.Ordinal)); + judged.Diagnostics.Should().BeEmpty(); + } + + [Theory] + [InlineData("Prev")] + [InlineData("XRefStm")] + public void A_section_whose_prev_a_guard_cut_keeps_the_chain_it_hides_when_it_is_found_where_it_was_moved(string key) + { + // A third table's /Prev names the table whose /Prev, object 9, or whose /XRefStm, object 6, MaxObjectLength cuts, 3 + // bytes before it: the section is found where it lies, and what that value hides is said of it there, as where it is + // named (#263). + var two = key == "Prev" + ? ChainFiles.UpdateNamingItsPrevious("9 0 R", padding: 3000) + : ChainFiles.HybridNamingItsStream("6 0 R", padding: 3000); + var text = Encoding.Latin1.GetString(two); + var named = long.Parse(text[(text.LastIndexOf("startxref\n", StringComparison.Ordinal) + "startxref\n".Length)..].Split('\n')[0], CultureInfo.InvariantCulture); + var five = two.Length; + var appended = Invariant($"5 0 obj\n(five)\nendobj\n"); + var third = five + appended.Length; + appended += Invariant( + $"xref\n0 1\n0000000000 65535 f\r\n5 1\n{five:D10} 00000 n\r\ntrailer\n<< /Size 10 /Root 1 0 R /Prev {named - 3} >>\nstartxref\n{third}\n%%EOF\n"); + var file = two.Concat(Encoding.Latin1.GetBytes(appended)).ToArray(); + + using var document = PdfDocument.Open(file, ObjectGuard); + + var moved = document.Reader.Structure.Sections.Should().ContainSingle(section => section.State == XRefSectionState.Relocated).Which; + moved.ChainHiddenByGuard.Should().NotBeNull(); + new PdfValidator().Validate(document).Findings + .Where(finding => finding.RuleId == PdfValidationRuleIds.XRefCheckedInPart) + .Should().ContainSingle(finding => finding.Message.Contains( + key == "Prev" ? "names older sections" : "names the cross-reference stream of its revision", StringComparison.Ordinal)); + } + + [Theory] + [InlineData("/W 5 0 R", "[1 4 2]")] + [InlineData("/W [1 4 2] /Index 5 0 R", "[0 8]")] + public void A_cross_reference_stream_a_guard_cut_a_value_of_is_found_where_it_was_moved(string entries, string value) + { + // The newer table's /Prev lands 3 bytes into stream 7, whose /W or /Index names object 5, which MaxObjectLength cuts + // before its value. The stream, found 3 bytes away, was dropped as no section, since the guard kept it from being + // read: the section was said to be missing, and the index rebuilt, for the guard's cut. It is the section moved, + // which the guard ends the chain at, as where the /Prev names it exactly (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 " + entries + " /Length {length}", new string(' ', 1524) + value, compressed: false, predicted: false, placed: true); + var text = Encoding.Latin1.GetString(file); + var stream = text.IndexOf("7 0 obj", StringComparison.Ordinal); + file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {stream + 3} "), StringComparison.Ordinal)); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.XRefOffsetAdjusted); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.XRefOffsetAdjusted, PdfDiagnosticCodes.LimitObject); + document.WasRepaired.Should().BeFalse(); + var moved = document.Reader.Structure.Sections[^1]; + moved.Offset.Should().Be(stream); + moved.CutByLimit.Should().BeTrue(); + var findings = new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId); + findings.Should().Contain(PdfValidationRuleIds.XRefCheckedInPart).And.NotContain(PdfValidationRuleIds.XRefSectionNotFound); + + // The /Prev names the wrong offset whatever the guard did: the section moved is reported, as read whole. + findings.Should().Contain(PdfValidationRuleIds.XRefSectionShifted); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void A_cross_reference_stream_whose_deferred_length_a_guard_cut_is_found_where_it_was_moved(bool typed) + { + // The newer table's /Prev lands 3 bytes into stream 7, whose /Length names object 5, which MaxObjectLength cuts before + // its value, and whose endstream is misspelled: no endstream ends its data, and where it would end is the guard's. + // Found 3 bytes away, it is the section moved, which the guard ends the chain at, as where the /Prev names it, whether + // or not it gives its /Type (#263). + var built = ChainFiles.UnderAnUpdate( + (typed ? "/Type /XRef " : string.Empty) + "/Size 8 /W [1 4 2] /Length 5 0 R", new string(' ', 1100) + "56", compressed: false, predicted: false, placed: true); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("7 0 obj", StringComparison.Ordinal); + var endStream = text.IndexOf("endstream", stream, StringComparison.Ordinal); + text = text[..endStream] + "endstreaX" + text[(endStream + "endstream".Length)..]; + var file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {stream + 3} "), StringComparison.Ordinal)); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.StreamLengthInvalid, PdfDiagnosticCodes.XRefOffsetAdjusted); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.XRefOffsetAdjusted, PdfDiagnosticCodes.LimitObject); + document.WasRepaired.Should().BeFalse(); + var moved = document.Reader.Structure.Sections[^1]; + moved.Offset.Should().Be(stream); + moved.CutByLimit.Should().BeTrue(); + var findings = new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId); + findings.Should().Contain([PdfValidationRuleIds.XRefCheckedInPart, PdfValidationRuleIds.XRefSectionShifted]) + .And.NotContain(PdfValidationRuleIds.XRefSectionNotFound); + + var opening = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }).Dispose(); + opening.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData(0)] + [InlineData(3)] + public void A_cross_reference_stream_a_guard_cut_the_dictionary_of_after_its_type_is_the_section_where_it_was_moved(int shift) + { + // Stream 7 writes /Type /XRef, then a string MaxTrailerLength cuts: the guard keeps it from being read. Named 3 bytes + // off, it was dropped as a place that says nothing of what it is — the section said missing, the index rebuilt, an + // error found —, though it declared itself a section before the cut, as named exactly, where it is the guard's (#263). + var built = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Pad (" + new string('x', 300) + ") /Length {length}", "0", compressed: false, predicted: false, placed: true); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("7 0 obj", StringComparison.Ordinal); + var file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {stream + shift} "), StringComparison.Ordinal)); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxTrailerLength = 128 } }; + + using var document = PdfDocument.Open(file, options); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + shift == 0 ? [PdfDiagnosticCodes.LimitTrailer] : [PdfDiagnosticCodes.LimitTrailer, PdfDiagnosticCodes.XRefOffsetAdjusted]); + document.WasRepaired.Should().BeFalse(); + document.Reader.Structure.Sections[^1].CutByLimit.Should().BeTrue(); + var findings = new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId); + findings.Should().Contain(PdfValidationRuleIds.XRefCheckedInPart).And.NotContain(PdfValidationRuleIds.XRefSectionNotFound); + } + + [Theory] + [InlineData(0)] + [InlineData(2)] + public void A_classic_table_a_guard_cut_the_rows_of_is_the_section_where_it_was_moved(int shift) + { + // The older table's first row lies past MaxXRefSectionLength, beyond a long comment: the guard keeps it from being + // read. Named 2 bytes off, it was dropped — the section said missing, an error found — though its xref keyword + // declares a section, as named exactly, where it is the guard's (#263). + var file = new HandWrittenFile(); + var one = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + var two = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + var three = file.Object(3, Page); + var older = file.Position; + file.Raw(Invariant( + $"xref\n%{new string('x', 2000)}\n0 4\n0000000000 65535 f\r\n{one:D10} 00000 n\r\n{two:D10} 00000 n\r\n{three:D10} 00000 n\r\ntrailer\n<< /Size 4 /Root 1 0 R >>\n")); + var newer = file.Position; + file.Raw(Invariant($"xref\ntrailer\n<< /Size 4 /Root 1 0 R /Prev {older + shift} >>\nstartxref\n{newer}\n%%EOF\n")); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxXRefSectionLength = 1024 } }; + + using var document = PdfDocument.Open(file.Bytes, options); + + document.Diagnostics.Select(entry => entry.Code).Should().Contain(PdfDiagnosticCodes.LimitXRefSectionLength) + .And.NotContain(PdfDiagnosticCodes.XRefSectionMissing); + var findings = new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId); + findings.Should().Contain(PdfValidationRuleIds.XRefCheckedInPart).And.NotContain(PdfValidationRuleIds.XRefSectionNotFound); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void A_stream_with_no_numbering_whose_width_a_guard_cut_is_no_section_the_guard_cut(bool named) + { + // Object 6, a stream 61 bytes before stream 7, writes its /W as 5 0 R, which MaxObjectLength cuts, and neither /Size + // nor /Index: no section, whatever the guard hid, as read whole. Where the /Prev names it, or a place inside it, it + // was taken for the section the guard cut, and stream 7, the section, was lost (#263). + var built = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + "[1 4 2]" + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + six: "<< /W 5 0 R /Length 3 >>\nstream\nabc\nendstream"); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("\n7 0 obj", StringComparison.Ordinal) + 1; + var six = text.IndexOf("\n6 0 obj", StringComparison.Ordinal) + 1; + var prev = named ? six : six + 21; + var file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {prev} "), StringComparison.Ordinal)); + + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + + document.Diagnostics.Select(entry => entry.Message).Should().Equal(whole.Diagnostics.Select(entry => entry.Message)); + document.Reader.Structure.Sections[^1].Offset.Should().Be(stream); + document.GetObject(new PdfObjectId(3)).AsDictionary().IsOfType(PdfName.Page).Should().BeTrue(); + } + + [Fact] + public void A_font_whose_widths_a_guard_cut_where_the_chain_names_a_section_is_no_section_the_guard_cut() + { + // The /Prev names object 6 exactly, a CIDFont whose /W precedes what MaxTrailerLength cuts: the guard reached, which + // the place the chain names keeps, cut no section, the font saying what it is. It was taken for the section the + // guard cut, the chain ended there, and the catalog was lost; read whole, the section is found farther on (#263). + var built = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + "0", + compressed: false, + predicted: false, + placed: true, + six: "<< /Type /Font /Subtype /CIDFontType2 /W [0 [500]] /BaseFont /" + new string('F', 200) + " >>"); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("\n7 0 obj", StringComparison.Ordinal) + 1; + var six = text.IndexOf("\n6 0 obj", StringComparison.Ordinal) + 1; + var file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {six} "), StringComparison.Ordinal)); + + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxTrailerLength = 128 } }); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitTrailer, PdfDiagnosticCodes.XRefOffsetAdjusted); + document.Diagnostics[0].Message.Should().StartWith("The dictionary of the object read as a cross-reference stream runs past 128 bytes"); + document.Reader.Structure.Sections[^1].Offset.Should().Be(stream); + document.GetObject(new PdfObjectId(1)).AsDictionary().IsOfType(PdfName.Catalog).Should().BeTrue(); + document.WasRepaired.Should().BeFalse(); + } + + [Theory] + [InlineData("/Type /XRef /Size 8 /W [1 4] /Filter 5 0 R", "Filter", null)] + [InlineData("/Type /XRef /Size 8 /W [1 -4 2] /Filter 5 0 R", "Filter", null)] + [InlineData("/Type /XRef /Size 8 /W [0 0 0] /Filter 5 0 R", "Filter", null)] + [InlineData("/Type /XRef /Size 8 /W [1 /x 2] /Filter 5 0 R", "Filter", null)] + [InlineData("/Type /XRef /W [1 4] /Filter 5 0 R", "Filter", null)] + [InlineData("/Type /XRef /Size 9 0 R /W [1 4 2] /Filter 5 0 R", "Filter", "-1")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [0 9 0 R] /Filter 5 0 R", "Filter", "-1")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index 9 0 R /Filter 5 0 R", "Filter", "[0 8 3]")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [0 5 0 R 3]", "Index element", null)] + [InlineData("/Type /XRef /Size 8 /W 5 0 R /Filter 9 0 R", "W", null)] + [InlineData("/Type /XRef /Size 8 /W 5 0 R /Index 9 0 R", "W", null)] + [InlineData("/Type /XRef /Size 8 /W [1 4] /Length 5 0 R", "Length", null)] + [InlineData("/Type /XRef /W [1 4 2] /Index [0] /Length 5 0 R", "Length", null)] + [InlineData("/Type /XRef /Size 8 /W 9 0 R /Length 5 0 R", "Length", null)] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [5 0 R -1]", "Index element", null)] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [5 0 R /x]", "Index element", null)] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [/x 5 0 R]", "Index element", null)] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [5 0 R 3000000000]", "Index element", null, false, "its /Index gives a subsection of 3,000,000,000 rows, more than object numbers 0 to 2147483647 hold")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [5 0 R 2147483649]", "Index element", null, false, "its /Index gives a subsection of 2,147,483,649 rows, more than object numbers 0 to 2147483647 hold")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [-1 5 0 R]", "Index element", null, false, "its /Index gives a subsection from object -1, outside object numbers 0 to 2147483647")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [2147483648 5 0 R]", "Index element", null, false, "its /Index gives a subsection from object 2147483648, outside object numbers 0 to 2147483647")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [9 0 R -1] /Length 5 0 R", "Length", "cut:0")] + public void What_makes_a_cross_reference_stream_unreadable_whatever_a_value_a_guard_cut_holds_is_the_file_s( + string entries, string cut, string? nine, bool asReadWhole = true, string? message = null) + { + // Stream 7's /Filter, an /Index element, its /W or its /Length names object 5, which MaxObjectLength cuts — the length + // where no endstream ends the data. Its /W gives no three widths, or none past 0; its /Index holds no pairs, or a + // count of rows below 0, and its /Size no count, written direct or through object 9; or a value past the cut names + // object 9, which nothing places: no value the guard hid makes the section readable, and read whole it is the + // file's fault. Only a numbering written direct was judged: the section was the guard's, the chain ended, the index + // was not rebuilt and the catalog was lost (#263). Nor was a subsection one of whose elements the guard cut, though + // the other is wrong whatever the cut one holds: no integer, a count below 0 or past what object numbers hold, a + // first number outside them — said as read whole where the cut element is in no part of what is said. + var (five, compressed) = cut switch + { + "Filter" => ("/FlateDecode" + new string(' ', 1100), true), + "Index element" => ("8" + new string(' ', 1100), false), + "W" => ("[1 4 2]" + new string(' ', 1100), false), + _ => (new string(' ', 1100) + "56", false), + }; + var built = ChainFiles.UnderAnUpdate( + cut == "Length" ? entries : entries + " /Length {length}", + five, + compressed, + predicted: false, + placed: true, + placedToo: nine is null + ? null + : new Dictionary { [9] = nine.StartsWith("cut:", StringComparison.Ordinal) ? nine[4..] + new string(' ', 1100) : nine }); + var text = Encoding.Latin1.GetString(built); + + if (cut == "Length") + { + var endStream = text.IndexOf("endstream", text.IndexOf("7 0 obj", StringComparison.Ordinal), StringComparison.Ordinal); + text = text[..endStream] + "endstreaX" + text[(endStream + "endstream".Length)..]; + } + + var file = Encoding.Latin1.GetBytes(text); + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + var unreadable = whole.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.XRefSectionUnreadable).Which; + var guarded = document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.XRefSectionUnreadable).Which; + + if (asReadWhole) + { + guarded.Message.Should().Be(unreadable.Message); + } + else + { + guarded.Message.Should().Contain(message); + } + + document.WasRepaired.Should().BeTrue(); + document.Reader.Structure.Sections.Should().NotContain(section => section.CutByLimit); + document.GetObject(new PdfObjectId(1)).AsDictionary().IsOfType(PdfName.Catalog).Should().BeTrue(); + } + + [Theory] + [InlineData("/Type /XRef /Size 9 0 R /W [1 4 2] /Index 5 0 R", "[0 8]", "/Index is the reference 5 0 R")] + [InlineData("/Type /XRef /W [1 4 2] /Index 5 0 R", "[0 8]", "/Index is the reference 5 0 R")] + [InlineData("/Type /XRef /Size 8 /W [0 0 5 0 R]", "7", "/W holds the reference 5 0 R")] + [InlineData("/Type /XRef /Size 8 /W [1 5 0 R 2] /Index [0 9 0 R]", "4", "/W holds the reference 5 0 R")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [5 0 R 8]", "0", "/Index holds the reference 5 0 R")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [5 0 R 0]", "0", "/Index holds the reference 5 0 R")] + [InlineData("/Type /XRef /Size 8 /W [1 4 2] /Index [2147483647 5 0 R]", "1", "/Index holds the reference 5 0 R")] + public void A_section_a_value_s_cut_keeps_from_being_read_is_the_guard_s_where_nothing_read_says_it_is_wrong( + string entries, string five, string named) + { + // Stream 7's /W, a /W element, its /Index or an /Index element names object 5, which MaxObjectLength cuts, and what + // the rest says is right for some value of it: a /Size no section places beside an /Index, which the rows are read + // without; no /Size beside an /Index; widths of 0 beside the one cut; a count that numbers objects; a second value + // cut, object 9, after the first. The section is the guard's, named by the first value it cut (#263). + var file = ChainFiles.UnderAnUpdate( + entries + " /Length {length}", + five + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + placedToo: entries.Contains("[0 9 0 R]", StringComparison.Ordinal) + ? new Dictionary { [9] = "8" + new string(' ', 1100) } + : null); + + using var document = PdfDocument.Open(file, ObjectGuard); + var section = document.Reader.Structure.Sections[^1]; + + section.CutByLimit.Should().BeTrue(); + section.Fault.Should().Contain(named); + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + } + + [Theory] + [InlineData("/W [1 4] /Size 8 /Filter 5 0 R")] + [InlineData("/W 5 0 R /Size 8 /Filter 9 0 R")] + public void A_place_a_section_may_have_moved_to_that_a_value_past_a_guard_s_cut_makes_unreadable_is_dropped(string entries) + { + // The /Prev lands 21 bytes into object 6, a stream whose /Filter, or /W, names object 5, which MaxObjectLength cuts, + // beside a /W that gives no three widths, or a /Filter that names object 9, which nothing places: no section, + // whatever the guard hid, as read whole. It was taken for the section the guard cut, and stream 7, the section, was + // lost (#263). + var built = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + (entries.StartsWith("/W 5", StringComparison.Ordinal) ? "[1 4 2]" : "/FlateDecode") + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + six: "<< " + entries + " /Length 3 >>\nstream\nabc\nendstream"); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("\n7 0 obj", StringComparison.Ordinal) + 1; + var six = text.IndexOf("\n6 0 obj", StringComparison.Ordinal) + 1; + var file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {six + 21} "), StringComparison.Ordinal)); + + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Message).Should().Equal(whole.Diagnostics.Select(entry => entry.Message)); + document.Reader.Structure.Sections[^1].Offset.Should().Be(stream); + document.GetObject(new PdfObjectId(1)).AsDictionary().IsOfType(PdfName.Catalog).Should().BeTrue(); + } + + [Theory] + [InlineData("<< /Type /Font /W 5 0 R /Size 8 /Length 56 >>\nstream\n{data}\nendstream", "[1 4 2]")] + [InlineData("<< /Type /Font /W [1 4 2] /Size 8 /Length 5 0 R >>\nstream\nabc\nendstreaX", "3")] + [InlineData("<< /Type /Xref /W [1 5 0 R 2] /Size 8 /Length 56 >>\nstream\n{data}\nendstream", "4")] + public void A_stream_whose_value_a_guard_cut_where_the_chain_names_a_section_is_the_section_the_guard_cut_whatever_its_type( + string six, string five) + { + // The /Prev names object 6 exactly, a stream that declares /Type /Font, or misspells XRef, whose /W, a /W element, or + // whose /Length where no endstream ends its data before object 8, names object 5, which MaxObjectLength cuts. Where + // the chain names it, a stream with a /W is read as the section whole, whatever its type: the value hides whether its + // rows are, and the section is the guard's. It was taken for no section, the section looked for nearby, and a stream + // found there — an older revision's — read in its place (#263). + six = six.Replace("{data}", new string('0', 56), StringComparison.Ordinal); + var built = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + five == "3" ? new string(' ', 1100) + five : five + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + six: six, + placedToo: new Dictionary { [8] = "(eight)" }); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("\n7 0 obj", StringComparison.Ordinal) + 1; + var sixAt = text.IndexOf("\n6 0 obj", StringComparison.Ordinal) + 1; + var file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {sixAt} "), StringComparison.Ordinal)); + + using var document = PdfDocument.Open(file, ObjectGuard); + + var section = document.Reader.Structure.Sections[^1]; + section.Offset.Should().Be(sixAt); + section.CutByLimit.Should().BeTrue(); + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + } + + [Theory] + [InlineData("<< /Type /Font /W 5 0 R /Size 8 /Length 3 >>\nstream\nabc\nendstream", "[1 4 2]", "/Prev", "holds at most 3 of the 8 rows", true)] + [InlineData("<< /Type /XRef /W 5 0 R /Size 8 /Length 3 /Info 8 0 R /ID [<01> <02>] >>\nstream\nabc\nendstream", "[1 4 2]", "/Prev", "holds at most 3 of the 8 rows", true)] + [InlineData("<< /Type /XRef /W 5 0 R /Size 8 /Length 3 >>\nstream\nabc\nendstream", "[1 4 2]", "/Prev+2", "holds at most 3 of the 8 rows", true)] + [InlineData("<< /Type /XRef /W 5 0 R /Size 8 /Length 3 >>\nstream\nabc\nendstream", "[1 4 2]", "/XRefStm", "holds at most 3 of the 8 rows", true)] + [InlineData("<< /Type /Xref /W [1 5 0 R 2] /Size 8 /Length 3 >>\nstream\nabc\nendstream", "4", "/Prev", "holds at most 1 of the 8 rows", true)] + [InlineData("<< /Type /XRef /W [1 4 2] /Index [5 0 R 8] /Length 3 >>\nstream\nabc\nendstream", "0", "/Prev", "its data, 3 bytes decoded, holds at most 0 of the 8 rows", false)] + [InlineData("<< /Type /XRef /W [1 4 2] /Index [5 0 R 1] /Length 1 >>\nstream\na\nendstream", "0", "/Prev", "its data, 1 byte decoded, holds at most 0 of the 1 row its", false)] + [InlineData("<< /Type /XRef /W [1 4 2] /Index [5 0 R 1 9 5 0 R] /Length 3 >>\nstream\nabc\nendstream", "0", "/Prev", "holds at most 0 of the 1 or more rows", false)] + [InlineData("<< /Type /XRef /W [1 4 2] /Index [5 0 R 2147483648] /Length 56 >>\nstream\n{56}\nendstream", "0", "/Prev", "holds at most 8 of the 2,147,483,648 rows", true)] + [InlineData("<< /Type /XRef /W [1 9223372036854775807 5 0 R] /Size 8 /Length 56 >>\nstream\n{56}\nendstream", "0", "/Prev", "holds at most 0 of the 8 rows", false)] + [InlineData("<< /Type /XRef /W [4611686018427387904 4611686018427387904 5 0 R] /Size 8 /Length 56 >>\nstream\n{56}\nendstream", "0", "/Prev", "holds at most 0 of the 8 rows", false)] + [InlineData("<< /Type /XRef /W 5 0 R /Size 8 /Filter /FlateDecode /Length {length} >>\nstream\n{flate}\nendstream", "[1 4 2]", "/Prev", "its data, 5 bytes decoded, holds at most 5 of the 8 rows", true)] + [InlineData("<< /Type /XRef /W 5 0 R /Size 8 /Filter [/FlateDecode] /DecodeParms [<< /Predictor 1 >>] /Length {length} >>\nstream\n{flate}\nendstream", "[1 4 2]", "/Prev", "its data, 5 bytes decoded, holds at most 5 of the 8 rows", true)] + [InlineData("<< /Type /XRef /W 5 0 R /Size 8 /Filter /FlateDecode /DecodeParms << /Predictor 1 >> /Length {length} >>\nstream\n{flate}\nendstream", "[1 4 2]", "/Prev", "its data, 5 bytes decoded, holds at most 5 of the 8 rows", true)] + public void A_section_whose_data_holds_fewer_rows_than_it_declares_whatever_a_value_a_guard_cut_holds_is_the_file_s_to_lack( + string six, string five, string naming, string fault, bool hidesRows) + { + // As above, object 6's data, whose length it gives, holds fewer rows than its /Size or the /Index counts read declare, + // however few bytes the widths read and the one cut give a row — a width past what data holds counted as such —: + // read whole, it lacks rows, the file's, its trailer merged, the chain going on, and the index rebuilt. It was taken + // for the guard's: the chain ended, nothing was rebuilt and the catalog was lost (#263); then for a section that + // cannot be read, whose trailer was lost, and which, where the chain names a place a few bytes off, was passed over + // for an older revision's stream nearby. Named by /XRefStm it is read as named. Filters written direct are a name or + // an array of names, beside parameters written as a dictionary, or an array of them, that holds no reference. + using var output = new MemoryStream(); + + using (var zlib = new ZLibStream(output, CompressionLevel.NoCompression, leaveOpen: true)) + { + zlib.Write(new byte[5]); + } + + var flate = output.ToArray(); + six = six + .Replace("{flate}", Encoding.Latin1.GetString(flate), StringComparison.Ordinal) + .Replace("{length}", Invariant($"{flate.Length}"), StringComparison.Ordinal) + .Replace("{56}", new string('0', 56), StringComparison.Ordinal); + var built = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + five + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + six: six, + placedToo: new Dictionary { [8] = "(eight)" }); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("\n7 0 obj", StringComparison.Ordinal) + 1; + var sixAt = text.IndexOf("\n6 0 obj", StringComparison.Ordinal) + 1; + var named = naming switch + { + "/Prev+2" => Invariant($"/Prev {sixAt + 2} "), + "/XRefStm" => Invariant($"/XRefStm {sixAt} "), + _ => Invariant($"/Prev {sixAt} "), + }; + var file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), named, StringComparison.Ordinal)); + string[] wholeCodes; + string[] wholeRules; + string[] wholeKeys; + + using (var whole = PdfDocument.Open(file)) + { + whole.WasRepaired.Should().BeTrue(); + whole.Reader.Structure.Sections[^1].Incomplete.Should().BeTrue(); + wholeCodes = [.. whole.Diagnostics.Select(entry => entry.Code)]; + wholeRules = [.. new PdfValidator().Validate(whole).Findings.Select(finding => finding.RuleId).Distinct().Order()]; + wholeKeys = [.. whole.Trailer.Keys.Select(key => key.Value).Order()]; + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + var section = document.Reader.Structure.Sections[^1]; + section.Offset.Should().Be(sixAt); + section.State.Should().Be(naming == "/Prev+2" ? XRefSectionState.Relocated : XRefSectionState.Read); + section.CutByLimit.Should().Be(hidesRows); + section.Incomplete.Should().BeTrue(); + section.Fault.Should().Contain(fault); + document.WasRepaired.Should().BeTrue(); + document.GetObject(new PdfObjectId(1)).AsDictionary().Required().IsOfType(PdfName.Catalog).Should().BeTrue(); + document.Trailer.Keys.Select(key => key.Value).Order().Should().Equal(wholeKeys); + + // The guard is reached once the chain is read. Rows the data may hold, which the guard hid, are said unchecked, and + // the highest number the chain uses, which they may raise, is not judged. + document.Diagnostics.Select(entry => entry.Code).Should().Contain(PdfDiagnosticCodes.LimitObject); + document.Diagnostics.Select(entry => entry.Code).Where(code => code != PdfDiagnosticCodes.LimitObject).Should().Equal(wholeCodes); + new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId).Distinct().Order().Should().Equal( + hidesRows + ? [.. wholeRules.Where(rule => rule != PdfValidationRuleIds.FileSizeWrong).Append(PdfValidationRuleIds.XRefCheckedInPart).Order()] + : wholeRules); + } + + [Theory] + [InlineData("/Filter [8 0 R]", "/FlateDecode")] + [InlineData("/Filter [/FlateDecode] /DecodeParms [<< /Predictor 8 0 R >>]", "1")] + public void Data_whose_filters_a_reference_gives_says_nothing_of_the_rows_a_section_beside_a_value_a_guard_cut_lacks( + string filters, string eight) + { + // As above, object 6's /W names object 5, which MaxObjectLength cuts, and its data, whose length it gives, decodes to + // 5 bytes: fewer than the 8 rows its /Size declares, at a byte a row. Read whole, it lacks rows. But an element of its + // /Filter array, or a value of the parameters its /DecodeParms array gives its one step, is a reference to object 8, + // which the newer table places — as a PDF 1.x file may write it, only ISO 32000-2 making it direct (7.5.8.2, + // pdf-issues #246) —, and only data whose filters are written direct, no reference in them, is decoded to judge the + // rows beside a cut (ADR 34). The section is the guard's, the chain ending there and nothing rebuilt. + var file = NamedBesideCut( + "<< /Type /XRef /W 5 0 R /Size 8 " + filters + " /Length {length} >>\nstream\n{data}\nendstream", eight, flates: 1, out var sixAt); + + using (var whole = PdfDocument.Open(file)) + { + whole.WasRepaired.Should().BeTrue(); + whole.Reader.Structure.Sections[^1].Incomplete.Should().BeTrue(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + var section = document.Reader.Structure.Sections[^1]; + section.Offset.Should().Be(sixAt); + section.State.Should().Be(XRefSectionState.Malformed); + section.CutByLimit.Should().BeTrue(); + section.Incomplete.Should().BeFalse(); + section.Fault.Should().Be("its /W is the reference 5 0 R, which one of the reader's limits cut"); + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + } + + [Theory] + [InlineData(nameof(PdfReaderLimits.MaxFilterChainLength), PdfDiagnosticCodes.LimitFilterChain)] + [InlineData(nameof(PdfReaderLimits.MaxDecodedStreamLength), PdfDiagnosticCodes.LimitDecodedStream)] + public void Data_a_guard_on_decoding_stops_says_nothing_of_the_rows_a_section_beside_a_value_a_guard_cut_lacks( + string limit, string code) + { + // As above, but object 6's /Size declares 40 rows, and its filters, written direct, are two Flate steps, the second of + // which a MaxFilterChainLength of 1 leaves undone, or one, whose 5 bytes a MaxDecodedStreamLength of 4 cuts: what the + // guard on decoding leaves is fewer bytes than the 40 rows, at the one byte a row the cut /W leaves at the fewest. Read + // whole, the section lacks rows; under the guard on decoding alone, the rows past what it decoded are the guard's. + // Beside the value MaxObjectLength cuts too, how many rows the data holds is the guard's to hide: the section is the + // guard's, the chain ending there, nothing rebuilt, and each guard reported. + var twoSteps = limit == nameof(PdfReaderLimits.MaxFilterChainLength); + var file = NamedBesideCut( + "<< /Type /XRef /W 5 0 R /Size 40 /Filter " + (twoSteps ? "[/FlateDecode /FlateDecode]" : "/FlateDecode") + + " /Length {length} >>\nstream\n{data}\nendstream", + "(eight)", + twoSteps ? 2 : 1, + out var sixAt); + var decoding = twoSteps ? PdfReaderLimits.Default with { MaxFilterChainLength = 1 } : PdfReaderLimits.Default with { MaxDecodedStreamLength = 4 }; + + using (var whole = PdfDocument.Open(file)) + { + whole.WasRepaired.Should().BeTrue(); + whole.Reader.Structure.Sections[^1].Incomplete.Should().BeTrue(); + } + + using (var decoded = PdfDocument.Open(file, new PdfReaderOptions { Limits = decoding })) + { + decoded.WasRepaired.Should().BeFalse(); + decoded.Reader.Structure.Sections[^1].CutByLimit.Should().BeTrue(); + decoded.Reader.Structure.Sections[^1].Incomplete.Should().BeFalse(); + } + + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = decoding with { MaxObjectLength = 1024 } }); + + var section = document.Reader.Structure.Sections[^1]; + section.Offset.Should().Be(sixAt); + section.State.Should().Be(XRefSectionState.Malformed); + section.CutByLimit.Should().BeTrue(); + section.Incomplete.Should().BeFalse(); + section.Fault.Should().Be("its /W is the reference 5 0 R, which one of the reader's limits cut"); + document.WasRepaired.Should().BeFalse(); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(code, PdfDiagnosticCodes.LimitObject); + } + + /// + /// Builds the file of the tests above: object 6, a stream whose dictionary and data gives — + /// {length} standing for the data's length, and {data} for 5 bytes of 0 under + /// steps of Flate kept uncompressed —, which stream 7 indexes; object 5, [1 4 2] past MaxObjectLength; and a newer + /// table that places objects 5 and 8, which holds , and names object 6 through its /Prev. + /// + private static byte[] NamedBesideCut(string six, string eight, int flates, out long sixAt) + { + var data = new byte[5]; + + for (var i = 0; i < flates; i++) + { + using var output = new MemoryStream(); + + using (var zlib = new ZLibStream(output, CompressionLevel.NoCompression, leaveOpen: true)) + { + zlib.Write(data); + } + + data = output.ToArray(); + } + + var built = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W [1 4 2] /Length {length}", + "[1 4 2]" + new string(' ', 1100), + compressed: false, + predicted: false, + placed: true, + six: six + .Replace("{data}", Encoding.Latin1.GetString(data), StringComparison.Ordinal) + .Replace("{length}", Invariant($"{data.Length}"), StringComparison.Ordinal), + placedToo: new Dictionary { [8] = eight }); + var text = Encoding.Latin1.GetString(built); + var stream = text.IndexOf("\n7 0 obj", StringComparison.Ordinal) + 1; + sixAt = text.IndexOf("\n6 0 obj", StringComparison.Ordinal) + 1; + return Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {sixAt} "), StringComparison.Ordinal)); + } + + [Theory] + [InlineData("/Prev")] + [InlineData("/XRefStm")] + public void Rows_a_section_lacking_rows_holds_beside_a_value_a_guard_cut_are_the_guard_s_and_no_older_section_stands_for_them( + string naming) + { + // Stream 6, of the second revision, places the catalog's newer definition in the seven rows its data holds of the + // eight its /Index declares, whose first number names object 5, which MaxObjectLength cuts; the first revision's + // table places the older one. Read whole, its rows win. The rows the data holds are read with the value the guard + // cut: the guard's, as rows a guard cut are, the chain ending there, where an older section's rows would stand for + // them — the older catalog was served, and the guard neither reported nor thrown under ThrowOnLimit. + var file = LackingBesideCut(naming, "/Index [5 0 R 8] /Size 8", rows => string.Concat(rows[..7]), out var sixAt); + + using (var whole = PdfDocument.Open(file)) + { + whole.WasRepaired.Should().BeFalse(); + whole.Reader.Structure.Sections.Single(section => section.Offset == sixAt).Incomplete.Should().BeTrue(); + Revision(whole).Should().Be("two"); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + PdfDiagnosticCodes.LimitObject, PdfDiagnosticCodes.XRefRebuilt, PdfDiagnosticCodes.ObjectRedefined); + document.Reader.Structure.Sections.Should().HaveCount(2); + var section = document.Reader.Structure.Sections[^1]; + section.Offset.Should().Be(sixAt); + section.State.Should().Be(XRefSectionState.Read); + section.CutByLimit.Should().BeTrue(); + section.Incomplete.Should().BeTrue(); + section.Fault.Should().Contain("holds at most 7 of the 8 rows"); + Revision(document).Should().Be("two"); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Theory] + [InlineData("/Prev")] + [InlineData("/XRefStm")] + public void A_section_lacking_rows_beside_a_value_a_guard_cut_reaches_the_guard_once_the_chain_is_read(string naming) + { + // As above, but stream 6's data, three bytes, holds no row of seven bytes: the section lacks rows whatever the value + // holds, its trailer merged and the chain going on, the first revision's table indexing every object, so that + // nothing is rebuilt. The value the guard cut was never loaded: the guard was neither reported nor thrown. + var file = LackingBesideCut(naming, "/Index [5 0 R 8] /Size 8", _ => "abc", out var sixAt); + string[] wholeCodes; + + using (var whole = PdfDocument.Open(file)) + { + whole.Reader.Structure.Sections.Single(section => section.Offset == sixAt).Incomplete.Should().BeTrue(); + wholeCodes = [.. whole.Diagnostics.Select(entry => entry.Code)]; + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal([.. wholeCodes, PdfDiagnosticCodes.LimitObject]); + document.Reader.Structure.Sections.Should().HaveCount(3); + var section = document.Reader.Structure.Sections.Single(section => section.Offset == sixAt); + section.CutByLimit.Should().BeFalse(); + section.Incomplete.Should().BeTrue(); + Revision(document).Should().Be("one"); + + var throwing = () => PdfDocument.Open(file, ObjectGuard with { ThrowOnLimit = true }); + throwing.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + + [Fact] + public void The_validator_charges_the_file_with_no_size_rows_a_section_lacking_rows_holds_beside_a_value_a_guard_cut_hide() + { + // Stream 6 places object 9 in the three rows its data holds of the five its /Index declares, whose first number a guard + // cut: read whole, the highest number the chain uses is 9, and the /Size 10 the trailers give is right. The rows the + // guard hid were not counted: the file was charged with a /Size two too large, and nothing said they went unchecked. + var file = LackingBesideCut( + "/Prev", "/Index [5 0 R 2 9 3] /Size 10", rows => rows[0] + rows[1] + rows[9], out _, tableSize: 10); + string[] wholeRules; + + using (var whole = PdfDocument.Open(file)) + { + wholeRules = [.. new PdfValidator().Validate(whole).Findings.Select(finding => finding.RuleId).Distinct().Order()]; + } + + wholeRules.Should().NotContain(PdfValidationRuleIds.FileSizeWrong); + + using var document = PdfDocument.Open(file, ObjectGuard); + + new PdfValidator().Validate(document).Findings.Select(finding => finding.RuleId).Distinct().Order().Should().Equal( + [.. wholeRules.Append(PdfValidationRuleIds.XRefCheckedInPart).Order()]); + } + + /// Gives the /Rev of the catalog reads, which says which revision it is. + private static string? Revision(PdfDocument document) => + (document.GetObject(new PdfObjectId(1)).AsDictionary().Required().GetRaw(PdfName.Get("Rev")) as PdfString)?.ToText(); + + /// + /// Builds three revisions: the first's catalog, /Rev (one), pages and table; the second's catalog, + /// /Rev (two), object 5 — 0, past MaxObjectLength —, object 9 and cross-reference stream 6, whose dictionary holds + /// and whose data gives from the rows that would place objects 0 to + /// 9; the third's table, placing object 5, which names stream 6 by . + /// + private static byte[] LackingBesideCut( + string naming, string numbering, Func data, out long sixAt, int tableSize = 8) + { + static string Row(int type, long offset, int generation) => string.Concat( + (char)type, (char)((offset >> 24) & 0xFF), (char)((offset >> 16) & 0xFF), (char)((offset >> 8) & 0xFF), + (char)(offset & 0xFF), (char)(generation >> 8), (char)(generation & 0xFF)); + + var file = new HandWrittenFile("", "1.5"); + var older = file.Object(1, "<< /Type /Catalog /Pages 2 0 R /Rev (one) >>"); + var two = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + var three = file.Object(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595 842] /Resources << >> >>"); + var first = file.Table(4, new Dictionary { [1] = older, [2] = two, [3] = three }, "/Size 4 /Root 1 0 R"); + var one = file.Object(1, "<< /Type /Catalog /Pages 2 0 R /Rev (two) >>"); + var five = file.Object(5, "0" + new string(' ', 1100)); + var nine = file.Object(9, "(nine)"); + sixAt = file.Position; + var rows = data( + [ + Row(0, 0, 65535), Row(1, one, 0), Row(1, two, 0), Row(1, three, 0), Row(0, 0, 0), Row(1, five, 0), Row(1, sixAt, 0), + Row(0, 0, 0), Row(0, 0, 0), Row(1, nine, 0), + ]); + var prev = naming == "/Prev" ? Invariant($" /Prev {first}") : string.Empty; + file.Raw(Invariant($"6 0 obj\n<< /Type /XRef /W [1 4 2] {numbering} /Root 1 0 R{prev} /Length {rows.Length} >>\nstream\n{rows}\nendstream\nendobj\n")); + file.Raw("%" + new string('-', 1200) + "\n"); + var table = file.Position; + var link = naming == "/Prev" ? Invariant($"/Prev {sixAt}") : Invariant($"/Prev {first} /XRefStm {sixAt}"); + file.Raw(Invariant($"xref\n0 1\n0000000000 65535 f \n5 1\n{five:D10} 00000 n \ntrailer\n<< /Size {tableSize} /Root 1 0 R {link} >>\nstartxref\n{table}\n%%EOF\n")); + return file.Bytes; + } + + [Fact] + public void Data_a_length_the_chain_cannot_read_yet_leaves_to_a_spelled_endstream_says_nothing_of_the_rows_it_holds() + { + // Stream 7's /W names object 5, which a newer table places and MaxObjectLength cuts, and its /Length object 8, which + // only its own rows place: its data runs to the first endstream, which its rows spell after six of eight. That data + // says nothing of how many rows the section holds, which the length, once read, may take further — read whole, it + // is taken again to its 56 bytes and every row is read —: the section is the guard's, not the file's to lack rows + // the data taken so far cannot hold at six bytes a row (#220, #263). + var file = new HandWrittenFile("", "1.5"); + var one = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + var two = file.Object(2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"); + var three = file.Object(3, Page); + var five = file.Object(5, "1" + new string(' ', 1100)); + var eight = file.Object(8, "56"); + var at = file.Position; + + static string Row(int type, long offset, int generation) => string.Concat( + (char)type, (char)((offset >> 24) & 0xFF), (char)((offset >> 16) & 0xFF), (char)((offset >> 8) & 0xFF), + (char)(offset & 0xFF), (char)(generation >> 8), (char)(generation & 0xFF)); + + var rows = Row(0, 0, 65535) + Row(1, one, 0) + Row(1, two, 0) + Row(1, three, 0) + Row(1, eight, 0) + Row(1, at, 0) + + "\nendstream\0\0\0\0"; + file.Raw(Invariant($"7 0 obj\n<< /Type /XRef /W [5 0 R 4 2] /Index [0 4 8 1 7 1 20 2] /Root 1 0 R /Length 8 0 R >>\nstream\n{rows}\nendstream\nendobj\n")); + file.Raw("%" + new string('-', 1200) + "\n"); + var table = file.Position; + file.Raw(Invariant($"xref\n0 1\n0000000000 65535 f \n5 1\n{five:D10} 00000 n \ntrailer\n<< /Size 22 /Root 1 0 R /Prev {at} >>\nstartxref\n{table}\n%%EOF\n")); + + using (var whole = PdfDocument.Open(file.Bytes)) + { + whole.WasRepaired.Should().BeFalse(); + whole.Reader.Structure.Sections[^1].Incomplete.Should().BeFalse(); + whole.GetObject(new PdfObjectId(3)).AsDictionary().Required().IsOfType(PdfName.Page).Should().BeTrue(); + } + + using var document = PdfDocument.Open(file.Bytes, ObjectGuard); + var section = document.Reader.Structure.Sections[^1]; + + section.CutByLimit.Should().BeTrue(); + section.Incomplete.Should().BeFalse(); + document.WasRepaired.Should().BeFalse(); + } + + [Fact] + public void A_cross_reference_stream_that_misspells_its_type_and_whose_value_a_guard_cut_is_the_section_the_guard_cut() + { + // Stream 7 writes /Type /Xref, which read whole is no fault of the reader's to report, and a /W element that names + // object 5, which MaxObjectLength cuts. It was refused for the type it gives, the file charged with a section it could + // not read, and the index rebuilt (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /Xref /Size 8 /W [1 5 0 R 2] /Length {length}", "4" + new string(' ', 1100), compressed: false, predicted: false, placed: true); + + using (var whole = PdfDocument.Open(file)) + { + whole.Diagnostics.Should().BeEmpty(); + } + + using var document = PdfDocument.Open(file, ObjectGuard); + + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + document.Reader.Structure.Sections[^1].CutByLimit.Should().BeTrue(); + document.WasRepaired.Should().BeFalse(); + } + + [Fact] + public void A_place_a_section_may_have_moved_to_that_cannot_be_read_leaves_what_it_read_through_to_the_rebuild() + { + // The newer table's /Prev lands 3 bytes into stream 7, whose /W names object 5, which MaxObjectLength cuts after + // "6 0 R", whole, and whose /Index names object 9, which nothing places: the stream, read through object 5, cannot be + // read, the file's fault, and is no section. Object 5 was loaded once the chain was read all the same: nothing of a + // place dropped is kept. The index is rebuilt, as read whole, and the rebuild, which reads object 5, reaches the + // guard (#263). + var file = ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W 5 0 R /Index 9 0 R /Length {length}", + "6 0 R" + new string(' ', 1524), + compressed: false, + predicted: false, + placed: true, + placedToo: new Dictionary { [6] = "[1 4 2]" }); + var text = Encoding.Latin1.GetString(file); + var stream = text.IndexOf("7 0 obj", StringComparison.Ordinal); + file = Encoding.Latin1.GetBytes(text.Replace(Invariant($"/Prev {stream} "), Invariant($"/Prev {stream + 3} "), StringComparison.Ordinal)); + + using var whole = PdfDocument.Open(file); + using var document = PdfDocument.Open(file, ObjectGuard); + + whole.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.XRefSectionMissing, PdfDiagnosticCodes.XRefRebuilt); + document.Diagnostics.Select(entry => entry.Code).Should().Equal( + PdfDiagnosticCodes.XRefSectionMissing, PdfDiagnosticCodes.XRefRebuilt, PdfDiagnosticCodes.LimitObject); + } + + [Fact] + public void A_chain_a_guard_ends_at_a_hidden_prev_leaves_unjudged_what_needs_the_whole_chain() + { + // Read whole, the update's /Size 13 is one more than object 12, which only the older table indexes, and object 4 is + // a page the tree does not list. With the older table hidden by the guard, object 12 is unknown: /Size is not judged + // against the 9 the update alone gives, nor is a page judged left out of the tree (#263). + var file = ChainFiles.UpdateHidingItsHighestNumber(padding: 3000); + + using (var whole = PdfDocument.Open(file)) + { + var report = new PdfValidator().Validate(whole); + report.Contains(PdfValidationRuleIds.FileSizeWrong).Should().BeFalse(); + report.Contains(PdfValidationRuleIds.PageTreePageOrphaned).Should().BeTrue(); + } + + using var guarded = PdfDocument.Open(file, ObjectGuard); + var findings = new PdfValidator().Validate(guarded).Findings.Select(finding => finding.RuleId).ToArray(); + + findings.Should().Contain(PdfValidationRuleIds.XRefCheckedInPart); + findings.Should().NotContain(PdfValidationRuleIds.FileSizeWrong); + findings.Should().NotContain(PdfValidationRuleIds.PageTreePageOrphaned); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_parameter_a_guard_cut_before_an_image_filter_leaves_no_encoded_image(bool guarded) + { + // Object 4's Flate step, before its DCT, reads its parameters in object 20, which MaxObjectLength cuts: decoding ends + // there and gives the Flate data back, which HasImageFilter took for the JPEG (#263). + var jpeg = new byte[] { 0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0xFF, 0xD9 }; + using var output = new MemoryStream(); + + using (var zlib = new ZLibStream(output, CompressionLevel.NoCompression, leaveOpen: true)) + { + zlib.Write(jpeg); + } + + var flate = output.ToArray(); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object( + 4, + string.Create(CultureInfo.InvariantCulture, $"<< /Filter [/FlateDecode /DCTDecode] /DecodeParms [20 0 R null] /Length {flate.Length} >>\nstream\n") + + Encoding.Latin1.GetString(flate) + "\nendstream"), + [20] = file.Object(20, new string(' ', 3000) + "<< /Predictor 1 >>"), + }; + file.Table(21, rows, "/Size 21 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes, guarded ? ObjectGuard : null); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + + stream.HasImageFilter().Should().Be(!guarded); + stream.Decode().ToArray().Should().Equal(guarded ? flate : jpeg); + + // Loading the parameter reaches the guard, as decoding would: a caller who asked for it is told so. + if (guarded) + { + using var throwing = PdfDocument.Open(file.Bytes, ObjectGuard with { ThrowOnLimit = true }); + var image = throwing.GetObject(new PdfObjectId(4)).AsStream().Required(); + image.Invoking(value => value.HasImageFilter()).Should().Throw() + .Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxObjectLength)); + } + } + + private static string Invariant(FormattableString text) => text.ToString(CultureInfo.InvariantCulture); +} diff --git a/tests/AdCodicem.Pdf.Tests/HostileInputMemoryTests.cs b/tests/AdCodicem.Pdf.Tests/HostileInputMemoryTests.cs index 5d8ddf9d..1de290e4 100644 --- a/tests/AdCodicem.Pdf.Tests/HostileInputMemoryTests.cs +++ b/tests/AdCodicem.Pdf.Tests/HostileInputMemoryTests.cs @@ -5,6 +5,7 @@ using AdCodicem.Pdf.Documents; using AdCodicem.Pdf.IO; using AdCodicem.Pdf.Objects; +using AdCodicem.Pdf.Validation; namespace AdCodicem.Pdf.Tests; @@ -70,6 +71,76 @@ public void Disposing_a_document_lets_go_of_the_object_streams_it_decoded_though GC.KeepAlive(held); } + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Streams_whose_data_share_a_stretch_of_the_file_hold_none_of_it_once_read(bool fromPath) + { + // Two hundred streams, each header inside the data of the one before, every /Length confirmed by the one endstream + // a megabyte on: each stream's data is the whole stretch, and each kept a copy of it for as long as the stream was + // held, 200 MB from a file of 1 MB (#170). The streams are held here; their data is not. + const int Streams = 200; + var file = HostileInputTests.Overlapping(Streams, filler: 1024 * 1024); + var path = fromPath ? Path.GetTempFileName() : null; + + try + { + if (path is not null) + { + File.WriteAllBytes(path, file); + } + + var before = Retained(); + using var document = path is null ? PdfDocument.Open(file) : PdfDocument.Open(path); + var held = new List(Streams); + long read = 0; + + for (var k = 0; k < Streams; k++) + { + var stream = document.GetObject(new PdfObjectId(3 + k)).AsStream().Required(); + read += stream.GetRawBytes().Length; + held.Add(stream); + } + + var after = Retained(); + + read.Should().BeGreaterThan(Streams * 1024L * 1024, "each stream's data is the whole stretch"); + after.Should().BeLessThan(before + (8L * 1024 * 1024), "the streams, and no copy of their data"); + document.Diagnostics.Should().BeEmpty(); + GC.KeepAlive(held); + } + finally + { + if (path is not null) + { + File.Delete(path); + } + } + } + + [Fact] + public void Validating_object_streams_whose_data_share_a_stretch_of_the_file_holds_no_copy_of_their_data() + { + // Three hundred object streams of one member each, each nested in the one before, every /Length confirmed by the one + // endstream a megabyte on: validating them, and reading their members, kept each one's data with its stream in the + // object cache, whatever the object stream budget let go — 300 MB from a file of 1 MB (#170). + const int Streams = 300; + var file = HostileInputTests.Overlapping(Streams, filler: 1024 * 1024, objectStreams: true); + var before = Retained(); + using var document = PdfDocument.Open(file); + + new PdfValidator().Validate(document); + + for (var k = 0; k < Streams; k++) + { + document.GetObject(new PdfObjectId(3000 + k)); + } + + var after = Retained(); + + after.Should().BeLessThan(before + (8L * 1024 * 1024), "a file in memory lends each object stream its data, and no copy is kept"); + } + [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.NoInlining)] private static (PdfObject Held, long Kept) OpenDecodeAndDispose(byte[] file) { diff --git a/tests/AdCodicem.Pdf.Tests/HostileInputTests.cs b/tests/AdCodicem.Pdf.Tests/HostileInputTests.cs index b56daab9..0838e9b2 100644 --- a/tests/AdCodicem.Pdf.Tests/HostileInputTests.cs +++ b/tests/AdCodicem.Pdf.Tests/HostileInputTests.cs @@ -19,6 +19,7 @@ public class HostileInputTests { private static readonly TimeSpan Budget = TimeSpan.FromSeconds(10); + [Theory] [InlineData("/A 1 ")] [InlineData("/A null ")] @@ -169,6 +170,204 @@ public void An_object_stream_header_s_tables_start_no_larger_than_its_bytes_can_ allocated.Should().BeLessThanOrEqualTo(2 * (sizeof(int) + 32), "two tables of one pair each"); } + [Theory] + [InlineData(0)] + [InlineData(100_000)] + [InlineData(1_000_000)] + public void A_length_the_file_cannot_hold_is_settled_by_the_endstream_found_without_reaching_a_guard(int dataLength) + { + // "/Length 2147483647", the data, its endstream, then an object of 17 MB: the window grew to MaxObjectLength to + // settle a length the file cannot hold, read 21.5 MB, reported limit.object and marked the object cut, though its + // data was found whole, and threw under ThrowOnLimit (#174). The endstream found settles it, the data read once. + var data = dataLength == 0 ? "short" : new string('d', dataLength); + var file = new TestPdfBuilder() + .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") + .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") + .WithObject(5, "<< /Length 2147483647 >>\nstream\n" + data + "\nendstream") + .WithObject(6, "(" + new string('p', 17 * 1024 * 1024) + ")") + .BuildClassic(rootNumber: 1); + + foreach (var throwOnLimit in new[] { false, true }) + { + using var source = new StrictCountingSource(file); + using var document = PdfDocument.Open(source, new PdfReaderOptions { ThrowOnLimit = throwOnLimit }, ownsSource: false); + var opened = source.BytesRead; + var stream = document.GetObject(new PdfObjectId(5)).AsStream().Required(); + + stream.Data.Length.Should().Be(data.Length); + document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( + string.Create(CultureInfo.InvariantCulture, $"The stream declared 2147483647 bytes but ended after {data.Length}.")); + document.Reader.IsCutAtLimit(5).Should().BeFalse(); + (source.BytesRead - opened).Should().BeLessThan(data.Length + (4L * PdfFileReader.InitialObjectWindow), "the data, read once"); + } + } + + [Fact] + public void Many_lengths_the_file_cannot_hold_read_the_file_once_not_once_each() + { + // Fifty such streams before an object of 17 MB: each grew its window to MaxObjectLength, and the reader read the + // file sixty times over (#174). + const int Streams = 50; + var builder = new TestPdfBuilder() + .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") + .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>"); + + for (var k = 0; k < Streams; k++) + { + builder.WithObject(10 + k, "<< /Length 2147483647 >>\nstream\nshort\nendstream"); + } + + var file = builder.WithObject(100, "(" + new string('p', 17 * 1024 * 1024) + ")").BuildClassic(rootNumber: 1); + using var source = new StrictCountingSource(file); + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + var opened = source.BytesRead; + + for (var k = 0; k < Streams; k++) + { + document.GetObject(new PdfObjectId(10 + k)).AsStream().Required().Data.Length.Should().Be(5); + } + + (source.BytesRead - opened).Should().BeLessThan(Streams * 2L * PdfFileReader.InitialObjectWindow, "a window and a few bytes for each"); + document.Diagnostics.Should().HaveCount(Streams).And.OnlyContain(diagnostic => diagnostic.Code == PdfDiagnosticCodes.StreamLengthInvalid); + } + + [Fact] + public void A_stream_whose_endobj_follows_its_length_without_endstream_reads_no_further_than_its_window_whatever_follows() + { + // iPRES's t02-05-01-014 before an object of 20 MB: no endstream after the data, which the window held, so it grew to + // MaxObjectLength looking for one, took 16 MB as the data of a 68-byte stream, reported limit.object and marked the + // object cut (#174). Its endobj plays the endstream's part. + var data = new string('d', 68); + var file = new TestPdfBuilder() + .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") + .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") + .WithObject(5, "<< /Length 68 >>\nstream\n" + data + "\r\n\n") + .WithObject(6, "(" + new string('p', 20 * 1024 * 1024) + ")") + .BuildClassic(rootNumber: 1); + + using var source = new StrictCountingSource(file); + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + var opened = source.BytesRead; + + document.GetObject(new PdfObjectId(5)).AsStream().Required().Data.Length.Should().Be(68); + + (source.BytesRead - opened).Should().BeLessThanOrEqualTo(PdfFileReader.InitialObjectWindow, "the window holds the object and what follows it"); + document.Diagnostics.Should().ContainSingle().Which.Code.Should().Be(PdfDiagnosticCodes.StreamTruncated); + document.Reader.IsCutAtLimit(5).Should().BeFalse(); + } + + [Fact] + public void A_length_the_file_cannot_hold_with_no_endstream_takes_the_rest_of_the_file_without_reaching_a_guard() + { + // "/Length 2147483647" and 20 MB of data to the end of the file, with no endstream and no endobj: the window grew to + // MaxObjectLength and the guard was reported (#174). The data runs to the end of the file, the object without its + // endobj, as for any stream the end of the file cuts. + var text = "%PDF-1.7\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n2 0 obj\n<< /Type /Pages /Kids [] /Count 0 >>\nendobj\n" + + "5 0 obj\n<< /Length 2147483647 >>\nstream\n"; + var file = Encoding.ASCII.GetBytes(text + new string('d', 20 * 1024 * 1024)); + + using var document = PdfDocument.Open(file); + var stream = document.GetObject(new PdfObjectId(5)).AsStream().Required(); + + stream.Data.Length.Should().Be(20 * 1024 * 1024); + document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamTruncated).Which.Message.Should().Be( + "The stream declared 2147483647 bytes, and no endstream follows its data before the end of the file; the 20971520 bytes to the end of the file are taken as its data."); + document.Diagnostics.Should().NotContain(d => d.Code.StartsWith("limit.", StringComparison.Ordinal)); + document.Reader.IsCutAtLimit(5).Should().BeFalse(); + document.Reader.IsEndObjMissing(5, out _).Should().BeTrue(); + } + + [Fact] + public void Sections_whose_lengths_name_one_object_a_guard_cut_read_it_once() + { + // A hundred cross-reference streams, chained by /Prev, each with /Length 5 0 R and rows that spell an endstream + // first; the newest table places object 5, which holds 28 then 2,000,000 spaces, past MaxObjectLength. Each section + // takes its data again to the 28 the cut left, which an endstream confirms, and the chain goes on: the cut object + // was read again for each section, and opening read the file several times over (#224). + const int Sections = 100; + var file = new HandWrittenFile(version: "1.5"); + var one = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"); + var two = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"); + var five = file.Object(5, "28" + new string(' ', 2_000_000)); + var rows = Row(1, one) + Row(1, two); + long previous = -1; + + for (var k = 0; k < Sections; k++) + { + var at = file.Position; + var prev = previous < 0 ? string.Empty : string.Create(CultureInfo.InvariantCulture, $" /Prev {previous}"); + file.Raw(string.Create(CultureInfo.InvariantCulture, $"{100 + k} 0 obj\n<< /Type /XRef /Size 200 /W [1 4 2] /Index [8 2 1 2] /Root 1 0 R /Length 5 0 R{prev} >>\nstream\n")); + file.Raw("\nendstream\0\0\0\0" + rows + "\nendstream\nendobj\n"); + previous = at; + } + + var table = file.Position; + file.Raw(string.Create( + CultureInfo.InvariantCulture, + $"xref\n0 1\n0000000000 65535 f\r\n5 1\n{five:D10} 00000 n\r\ntrailer\n<< /Size 200 /Root 1 0 R /Prev {previous} >>\nstartxref\n{table}\n%%EOF\n")); + var bytes = file.Bytes; + using var source = new StrictCountingSource(bytes); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = 64 * 1024 } }; + + using var document = PdfDocument.Open(source, options, ownsSource: false); + + document.Reader.Structure.Sections.Should().HaveCount(Sections + 1).And.OnlyContain(section => !section.CutByLimit); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + source.BytesRead.Should().BeLessThan(bytes.Length); + + static string Row(int type, long offset) => string.Concat( + (char)type, (char)((offset >> 24) & 0xFF), (char)((offset >> 16) & 0xFF), (char)((offset >> 8) & 0xFF), (char)(offset & 0xFF), '\0', '\0'); + } + + [Fact] + public void A_deferred_length_s_data_with_no_endstream_makes_its_section_malformed_reading_the_file_a_few_times() + { + // A cross-reference stream whose /Length only it indexes, 3 MB of rows, and its endstream misspelled: its search + // reads the data to the end of the file, and its section is malformed — the file's fault, no guard's (#220). The + // rebuild scans the file, and its stream, its length read then, is searched again: the file is read about four + // times, as its sound twin is twice — the search and the decoding. + var file = ChainFiles.AloneWithIndirectLength(padding: 450_000); + file[(int)PdfTemplate.OffsetOf(file, "endstream")] = (byte)'x'; + using var source = new StrictCountingSource(file); + + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + + document.Reader.Structure.Sections.Should().ContainSingle().Which.State.Should().Be(XRefSectionState.Malformed); + document.Diagnostics.Should().NotContain(d => d.Code.StartsWith("limit.", StringComparison.Ordinal)); + source.BytesRead.Should().BeLessThan(5L * file.Length); + } + + [Fact] + public void Reading_streams_whose_data_share_a_stretch_of_a_file_held_in_memory_allocates_none_of_their_data() + { + // Two hundred streams, each header inside the data of the one before, every /Length confirmed by the one endstream + // a megabyte on: each stream's data is the whole stretch, and reading each one's copied it, 200 MB from a file of + // 1 MB (#170). A source in memory lends each stream's data where it lies. + const int Streams = 200; + var file = Overlapping(Streams, filler: 1024 * 1024); + + var allocated = Allocations.Least(() => PdfDocument.Open(file), ReadEvery, out var document, out var read); + + using (document) + { + read.Should().BeGreaterThan(Streams * 1024L * 1024, "each stream's data is the whole stretch"); + allocated.Should().BeLessThan(2L * 1024 * 1024, "the streams' objects, and none of their data"); + document.Diagnostics.Should().BeEmpty(); + } + + static long ReadEvery(PdfDocument document) + { + long read = 0; + + for (var k = 0; k < Streams; k++) + { + read += document.GetObject(new PdfObjectId(3 + k)).AsStream().Required().GetRawBytes().Length; + } + + return read; + } + } + [Fact] public void An_object_stream_header_s_tables_grow_no_larger_than_its_bytes_can_hold() { @@ -1563,8 +1762,10 @@ public void Reports_chains_that_run_too_deep_once() [Fact] public void Does_not_take_a_keyword_the_probe_saw_cut_for_a_cross_reference_table() { - // The section is probed through a few bytes, which spaces and "xref" fill: the probe sees "xref", but the - // token is "xrefs". The table's own window reads it whole and refuses it, and the index is rebuilt. + // The section is probed through a few bytes, which spaces and "xref" fill, but the token is "xrefs". The probe reads + // on past a token its edge reaches, finds no keyword there, nor an object, and the index is rebuilt (#174). Where + // MaxTrailerLength stops the probe right after "xref", the table's own window reads the token whole and refuses it + // under another rule (#295). var written = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") @@ -1575,6 +1776,8 @@ public void Does_not_take_a_keyword_the_probe_saw_cut_for_a_cross_reference_tabl using var document = PdfDocument.Open(bytes); document.WasRepaired.Should().BeTrue(); + document.Reader.Structure.Sections[0].State.Should().Be(XRefSectionState.NotFound); + document.Reader.Structure.Sections[0].Fault.Should().Be("holds neither the xref keyword nor an object"); document.Catalog.Required().IsOfType(PdfName.Catalog).Should().BeTrue(); } @@ -3464,6 +3667,70 @@ private static byte[] Claiming(int count) return file.Bytes; } + /// + /// Writes a file of streams, objects 3 and on, each one's header inside the data of the one + /// before, every /Length ending at the one endstream after bytes: each length + /// confirmed, each stream's data the whole stretch from its start to that endstream (#170). With + /// , objects 10 and on, each an object stream of one member, objects 3000 and on, that a + /// cross-reference stream places. + /// + internal static byte[] Overlapping(int streams, int filler, bool objectStreams = false) + { + var file = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), 0), + [2] = (1, file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), 0), + }; + var start = file.Position; + var body = new StringBuilder(); + var lengthAt = new int[streams]; + var dataStart = new int[streams]; + + for (var k = 0; k < streams; k++) + { + var number = (objectStreams ? 10 : 3) + k; + var member = string.Create(CultureInfo.InvariantCulture, $"{3000 + k} 0 "); + rows[number] = (1, start + body.Length, 0); + body.Append(CultureInfo.InvariantCulture, $"{number} 0 obj\n<< "); + + if (objectStreams) + { + body.Append(CultureInfo.InvariantCulture, $"/Type /ObjStm /N 1 /First {member.Length} "); + } + + body.Append("/Length "); + lengthAt[k] = body.Length; + body.Append("0000000000 >>\nstream\n"); + dataStart[k] = body.Length; + + if (objectStreams) + { + body.Append(member); + rows[3000 + k] = (2, number, 0); + } + } + + body.Append('x', filler); + var end = body.Length; + body.Append("\nendstream\nendobj\n"); + + for (var k = 0; k < streams; k++) + { + var digits = (end - dataStart[k]).ToString("D10", CultureInfo.InvariantCulture); + + for (var d = 0; d < digits.Length; d++) + { + body[lengthAt[k] + d] = digits[d]; + } + } + + file.Raw(body.ToString()); + var index = objectStreams ? 6000 : 1000; + file.CrossReferenceStream(index, index + 1, rows, "/Root 1 0 R"); + return file.Bytes; + } + /// /// Writes a file whose object stream holds members one byte apart, objects 1000 and on, over /// as many open brackets: read to the end of the data, each would take every bracket after it (#200). diff --git a/tests/AdCodicem.Pdf.Tests/ParserTests.cs b/tests/AdCodicem.Pdf.Tests/ParserTests.cs index 6aa17f1e..7ed5019f 100644 --- a/tests/AdCodicem.Pdf.Tests/ParserTests.cs +++ b/tests/AdCodicem.Pdf.Tests/ParserTests.cs @@ -163,6 +163,150 @@ public void Asks_for_more_data_only_for_a_header_its_end_cuts_that_no_whole_toke parser.IsTruncated.Should().Be(more); } + [Fact] + public void Ends_the_containers_an_object_left_open_at_the_next_object_s_header_and_leaves_it_to_be_read() + { + // The object lost its ] and its endobj: the next object's header ends the array, and is where the parser stands to + // read that object (#230). + var diagnostics = new PdfDiagnostics(); + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes("[1 2\n6 0 obj\n(six)\nendobj\n"), diagnostics: diagnostics); + + parser.ParseObject().AsArray().Required().Should().HaveCount(2); + parser.TryReadIndirectObject(out var id, out var value).Should().BeTrue(); + + id.Should().Be(new PdfObjectId(6)); + value.Should().BeOfType(); + diagnostics.Should().ContainSingle().Which.Message.Should().Be("An object header came inside an array, which was never closed."); + } + + [Theory] + [InlineData("hello\nendobj\n% a note\n6 0 obj\n", 5)] + [InlineData("hello\nendobj % note endobj\n6 0 obj\n", 5)] + [InlineData("hello\nendobj\n% a note\nx\n6 0 obj\n", -1)] + [InlineData("hello endobjx\n6 0 obj\n", -1)] + [InlineData("helloendobj\n6 0 obj\n", -1)] + [InlineData("endobj\n6 0 obj\n", 0)] + [InlineData("hello\n/endobj\n6 0 obj\n", -1)] + [InlineData("hello\n%endobj\n6 0 obj\n", -1)] + [InlineData("hello\n)endobj\n6 0 obj\n", 7)] + [InlineData("hello\n[endobj\n6 0 obj\n", 7)] + public void Ends_a_stream_s_data_at_its_object_s_endobj_that_only_white_space_and_comments_follow(string data, int length) + { + // Parsed alone, a stream with no endstream ends where its object's endobj ends it, a comment after which is white + // space; a token after it leaves the data running to the next object's header (#174). A solidus or a percent sign + // right before the keyword makes it part of a name or of a comment that starts there: "/endobj" is a name. + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes("4 0 obj\n<< /Length 200 >>\nstream\n" + data)); + + parser.TryReadIndirectObject(out _, out var value).Should().BeTrue(); + + value.AsStream().Required().RawLength.Should().Be(length < 0 ? data.IndexOf("6 0 obj", StringComparison.Ordinal) : length); + } + + [Theory] + [InlineData("4 0 obj\n<< /Length 3 >>\nstream")] + [InlineData("4 0 obj\n<< /Length 3 >>\nstream\r")] + public void Leaves_a_stream_whose_window_ends_on_its_keyword_to_a_larger_one(string text) + { + // Parsed alone, which no file backs, a buffer that ends on the stream keyword, or on its carriage return, cannot say + // where the data starts: a larger one says. + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes(text), endsData: false); + + parser.TryReadIndirectObject(out _, out _); + + parser.IsTruncated.Should().BeTrue(); + } + + [Fact] + public void Leaves_a_stream_with_no_endstream_in_a_window_to_a_larger_one() + { + // Parsed alone in a window that is not the end of the data, a stream with no length to confirm, whose endstream and + // next header lie past the window, asks for a larger one, which may show them. + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes("4 0 obj\n<< >>\nstream\nabc"), endsData: false); + + parser.TryReadIndirectObject(out _, out _); + + parser.IsTruncated.Should().BeTrue(); + } + + [Theory] + [InlineData("[", "an array")] + [InlineData("<< /A ", "a dictionary")] + public void Ends_a_container_skipped_for_its_depth_at_the_next_object_s_header(string opening, string what) + { + // Past the depth the parser follows, the container is skipped by counting brackets: the next object's header ends it, + // as it ends one the parser follows (#230). + var text = string.Concat(Enumerable.Repeat(opening, 130)) + "1\n6 0 obj\n"; + var diagnostics = new PdfDiagnostics(); + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes(text), diagnostics: diagnostics); + + _ = parser.ParseObject(); + + diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.SyntaxDepthExceeded, PdfDiagnosticCodes.SyntaxTruncatedObject); + diagnostics[1].Message.Should().Be( + $"An object header came inside {what}, which was never closed; the 128 arrays or dictionaries around it were never closed either."); + } + + [Theory] + [InlineData("<< /A 1\n6")] + [InlineData("<< /A 1\n6 0")] + [InlineData("<< /A 1\n6 0 ")] + [InlineData("<< /A 1\n6 0 o")] + [InlineData("<< /A 1\n6 0 ob")] + [InlineData("<< /A 1\n6 0 obj")] + [InlineData("<< /A 1 /B\n6 0 obj")] + [InlineData("<< /A [1\n6 0 obj")] + public void Tokens_a_window_s_edge_leaves_undecided_as_a_header_are_no_stray_keys(string text) + { + // Inside a dictionary, the next object's header the window's edge cuts short: its number and generation were reported + // as keys that are not names, and kept so wherever no larger window follows, where a larger one reads a header (#230). + var diagnostics = new PdfDiagnostics(); + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes(text), diagnostics: diagnostics, endsData: false); + + parser.ParseObject().Should().BeOfType(); + + parser.IsTruncated.Should().BeTrue(); + diagnostics.Should().BeEmpty(); + } + + [Theory] + [InlineData("[1 2\n6 0 obj", true, 2)] + [InlineData("[1 2\n6 0 obj ", false, 2)] + [InlineData("[1 2\n6 0 obj\n", false, 2)] + public void Takes_an_object_header_a_window_s_edge_reaches_for_none_until_a_larger_window_shows_obj_whole( + string text, bool more, int count) + { + // "obj" the edge reaches may be the start of "objx", no keyword at all: what the window reads is dropped for a larger + // window, which says (#230). The window ends the array where the header may start, its number no element. + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes(text), endsData: false); + + parser.ParseObject().AsArray().Required().Should().HaveCount(count); + parser.IsTruncated.Should().Be(more); + } + + [Theory] + [InlineData("<< /A [1 << /B 2\n6 0 obj\n", 2)] + [InlineData("<< /A 1\n6 0 obj\n", 1)] + [InlineData("<< /A\n6 0 obj\n", 1)] + public void A_header_where_a_key_or_a_value_is_expected_ends_the_dictionary(string text, int depthOfCut) + { + var diagnostics = new PdfDiagnostics(); + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes(text), diagnostics: diagnostics); + + parser.ParseObject().Should().BeOfType(); + + diagnostics.Should().ContainSingle().Which.Position.Should().Be(depthOfCut == 1 ? 0 : text.IndexOf("<< /B", StringComparison.Ordinal)); + parser.IsTruncated.Should().BeFalse(); + } + + [Fact] + public void A_value_alone_that_is_a_header_s_number_is_an_integer() + { + // Outside a container a header ends nothing: "4 0 obj 5 0 obj" holds the integer 5, and its endobj is missing (#230). + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes("5 0 obj\n")); + + parser.ParseObject().Should().BeOfType().Which.Value.Should().Be(5); + } + [Fact] public void Reads_a_stream_whose_declared_length_is_correct() { @@ -182,12 +326,26 @@ public void Recovers_a_stream_whose_declared_length_is_wrong() } [Theory] - [InlineData(0, true)] - [InlineData(-1, false)] - [InlineData(11, false)] - public void Looks_for_endstream_only_inside_the_bytes_it_is_given(long position, bool expected) + [InlineData("\nendstream\n", "EndStream")] + [InlineData(" \n %a comment\r\n\t \fendstream endobj", "EndStream")] + [InlineData("\nendstream", "Undecided")] + [InlineData("x\nendstream", "None")] + [InlineData("\nendstrean", "None")] + [InlineData(" ", "Undecided")] + [InlineData("\nendst", "Undecided")] + [InlineData("\n%a comment the bytes end inside", "Undecided")] + public void Reads_endstream_after_any_white_space_and_comments_or_says_the_bytes_cannot_tell(string bytes, string expected) { - PdfObjectParser.IsEndStreamAt("\nendstream"u8, position).Should().Be(expected); + StreamEnds.ReadEnd(Encoding.ASCII.GetBytes(bytes), reachesEndOfFile: false).Kind.ToString().Should().Be(expected); + } + + [Theory] + [InlineData(" ")] + [InlineData("\nendst")] + [InlineData("\n%a comment the file ends inside")] + public void Bytes_that_end_the_file_before_endstream_hold_none(string bytes) + { + StreamEnds.ReadEnd(Encoding.ASCII.GetBytes(bytes), reachesEndOfFile: true).Kind.Should().Be(StreamEndKind.None); } [Fact] @@ -200,6 +358,38 @@ public void Recovers_a_stream_that_never_ends() diagnostics.Contains(PdfDiagnosticCodes.StreamTruncated).Should().BeTrue(); } + [Fact] + public void A_stream_that_lost_its_endstream_ends_at_its_own_endobj_before_the_next_object() + { + // The parser alone stops the search where the reader does, at the first header its bytes hold, and the object's + // endobj before it ends the data (#174). + var diagnostics = new PdfDiagnostics(); + var stream = ParseStream("<< /Length 3 >>\nstream\nhello\nendobj\n6 0 obj\n<< /Length 5 >>\nstream\nworld\nendstream\nendobj", diagnostics); + + Encoding.ASCII.GetString(stream.GetRawBytes().Span).Should().Be("hello"); + diagnostics.Should().ContainSingle().Which.Code.Should().Be(PdfDiagnosticCodes.StreamTruncated); + } + + [Fact] + public void A_stream_that_lost_its_endstream_runs_to_the_header_its_bytes_hold_when_no_endobj_precedes_it() + { + var diagnostics = new PdfDiagnostics(); + var stream = ParseStream("<< /Length 99 >>\nstream\nhello\n6 0 obj\n<< /Length 5 >>\nstream\nworld\nendstream\nendobj", diagnostics); + + Encoding.ASCII.GetString(stream.GetRawBytes().Span).Should().Be("hello\n"); + diagnostics.Should().ContainSingle().Which.Code.Should().Be(PdfDiagnosticCodes.StreamLengthInvalid); + } + + [Fact] + public void A_window_given_to_the_parser_alone_that_cannot_say_where_a_stream_ends_asks_for_more() + { + // No endstream, no header, no endobj in the window: a larger one may hold them. + var parser = new PdfObjectParser(Encoding.ASCII.GetBytes("<< /Length 300 >>\nstream\n" + new string('d', 100)), endsData: false); + + parser.ParseObject().Should().BeOfType(); + parser.IsTruncated.Should().BeTrue(); + } + [Fact] public void Resolves_a_length_given_as_an_indirect_reference() { @@ -585,6 +775,23 @@ public void Keeps_what_the_end_of_the_data_cut_short_as_far_as_it_was_read() hexadecimal.Bytes.ToArray().Should().Equal((byte)0x41, 0x42, 0x40); } + [Theory] + [InlineData("5 0 << /A 6 0 o")] + [InlineData("5 0 << /A 6 0 ob")] + public void Reports_the_keys_of_a_member_a_window_s_edge_leaves_as_a_header_none_can_be(string text) + { + // An object stream's member has no header after it: its window's edge leaves none undecided, and the key 0 that is no + // name is the file's, as in the whole member (#230). + var diagnostics = new PdfDiagnostics(); + var parser = PdfObjectParser.ForObjectStreamMember( + Encoding.ASCII.GetBytes(text), 9, 1000, 5, ObjectSourceReturning(new PdfObjectId(1), PdfNull.Instance), diagnostics, endsData: false); + parser.Position = 4; + + _ = parser.ParseObject(); + + diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.SyntaxUnexpectedToken); + } + [Fact] public void Says_an_object_stream_s_data_rather_than_the_file_ended_inside_a_member_s_construct() { diff --git a/tests/AdCodicem.Pdf.Tests/ReadOrderTests.cs b/tests/AdCodicem.Pdf.Tests/ReadOrderTests.cs index 387015c2..a1147d4c 100644 --- a/tests/AdCodicem.Pdf.Tests/ReadOrderTests.cs +++ b/tests/AdCodicem.Pdf.Tests/ReadOrderTests.cs @@ -1911,4 +1911,43 @@ private static List Findings(PdfDocument document, int validations) return first; } } + + [Fact] + public void A_stream_read_again_after_a_rebuild_changed_its_length_ends_as_one_read_after_the_rebuild_does() + { + // Object 4 lost its endstream, its endobj after its 10 bytes; its /Length, object 5, gives 10 where the table places + // it, and 3 where a second definition, which only a rebuild finds, gives it. Read before the rebuild, then after it, + // object 4 was served 3 bytes, taken as the length its endobj follows — what was read of the endobj after the first + // length was kept with the search. Read again, it ends as it does when first read after the rebuild: where its + // object's endobj ends it, after 10 bytes (#174). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + rows[4] = file.Position; + file.Raw("4 0 obj\n<< /Length 5 0 R >>\nstream\nABCDEFGHIJ\nendobj\n"); + rows[5] = file.Object(5, "10"); + rows[7] = file.Padding(7, 2000); + file.Object(6, "(six)"); + file.Object(5, "3"); + rows[6] = rows[1] + 1; + file.Table(8, rows, "/Size 8 /Root 1 0 R"); + + using var before = PdfDocument.Open(file.Bytes); + before.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal("ABCDEFGHIJ"u8.ToArray()); + before.GetObject(new PdfObjectId(6)); + before.WasRepaired.Should().BeTrue(); + var again = before.GetObject(new PdfObjectId(4)).AsStream().Required(); + + using var after = PdfDocument.Open(file.Bytes); + after.GetObject(new PdfObjectId(6)); + var first = after.GetObject(new PdfObjectId(4)).AsStream().Required(); + + first.GetRawBytes().ToArray().Should().Equal("ABCDEFGHIJ"u8.ToArray()); + again.GetRawBytes().ToArray().Should().Equal("ABCDEFGHIJ"u8.ToArray()); + before.Reader.TryGetStreamLengthFault(4, out var fault).Should().BeTrue(); + fault.Taken.Should().Be(10); + } } diff --git a/tests/AdCodicem.Pdf.Tests/ReaderLimitsTests.cs b/tests/AdCodicem.Pdf.Tests/ReaderLimitsTests.cs index dc1b152e..f5d35a55 100644 --- a/tests/AdCodicem.Pdf.Tests/ReaderLimitsTests.cs +++ b/tests/AdCodicem.Pdf.Tests/ReaderLimitsTests.cs @@ -22,7 +22,7 @@ public class ReaderLimitsTests /// A guard for each bound of , the trailer's reached two ways. public static TheoryData Guards => - ["decoded stream", "object", "section length", "section count", "trailer", "cross-reference stream", "index entries", "filter chain"]; + ["decoded stream", "stream length", "object", "section length", "section count", "trailer", "cross-reference stream", "index entries", "filter chain"]; [Fact] public void The_defaults_are_the_bounds_the_reader_kept_before_they_were_options() @@ -30,6 +30,7 @@ public void The_defaults_are_the_bounds_the_reader_kept_before_they_were_options var limits = PdfReaderLimits.Default; limits.MaxDecodedStreamLength.Should().Be(256 * 1024 * 1024); + limits.MaxStreamLength.Should().Be(256 * 1024 * 1024); limits.MaxObjectLength.Should().Be(16 * 1024 * 1024); limits.MaxXRefSectionLength.Should().Be(64 * 1024 * 1024); limits.MaxXRefSectionCount.Should().Be(1024); @@ -46,6 +47,7 @@ public void Unbounded_takes_every_guard_to_the_most_the_implementation_holds() var limits = PdfReaderLimits.Unbounded; limits.MaxDecodedStreamLength.Should().Be(Array.MaxLength); + limits.MaxStreamLength.Should().Be(Array.MaxLength); limits.MaxObjectLength.Should().Be(Array.MaxLength); limits.MaxXRefSectionLength.Should().Be(Array.MaxLength); limits.MaxXRefSectionCount.Should().Be(int.MaxValue); @@ -54,8 +56,24 @@ public void Unbounded_takes_every_guard_to_the_most_the_implementation_holds() limits.MaxFilterChainLength.Should().Be(int.MaxValue); } + [Fact] + public void MaxTrailerLength_bounds_a_cross_reference_stream_s_dictionary_and_not_its_data() + { + // The data of a cross-reference stream whose /Length only it indexes was searched for its endstream within the + // window that read the dictionary, which MaxTrailerLength bounds: a kilobyte of rows past 256 bytes reached the guard. + // The data is searched in the file, as any stream's whose length is not confirmed (#220). + var file = ChainFiles.AloneWithIndirectLength(padding: 150); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxTrailerLength = 256 }, ThrowOnLimit = true }; + + using var document = PdfDocument.Open(file, options); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.Index.Count.Should().Be(156); + } + [Theory] [InlineData(nameof(PdfReaderLimits.MaxDecodedStreamLength), 0)] + [InlineData(nameof(PdfReaderLimits.MaxStreamLength), 0)] [InlineData(nameof(PdfReaderLimits.MaxObjectLength), -1)] [InlineData(nameof(PdfReaderLimits.MaxXRefSectionLength), int.MinValue)] [InlineData(nameof(PdfReaderLimits.MaxXRefSectionCount), 0)] @@ -75,6 +93,7 @@ public void A_length_beyond_what_an_array_holds_is_taken_as_that_length() var limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = int.MaxValue, + MaxStreamLength = int.MaxValue, MaxObjectLength = int.MaxValue, MaxXRefSectionLength = int.MaxValue, MaxXRefSectionCount = int.MaxValue, @@ -112,6 +131,7 @@ PdfReaderLimits.Default with throwOnLimit: false); unbounded.Reach(PdfLimit.DecodedStream, diagnostics, "Cut.", 7); + unbounded.Reach(PdfLimit.StreamLength, diagnostics, "Cut.", 7); unbounded.Reach(PdfLimit.XRefSectionCount, diagnostics, "Cut.", 8); justBelow.Reach(PdfLimit.XRefSectionCount, diagnostics, "Cut.", 9); unbounded.Reach(PdfLimit.IndexEntries, diagnostics, "Cut.", 10); @@ -121,6 +141,7 @@ PdfReaderLimits.Default with diagnostics.Select(entry => entry.Message).Should().Equal( "Cut. PdfReaderLimits.MaxDecodedStreamLength is already at the most the reader can hold.", + "Cut. PdfReaderLimits.MaxStreamLength is already at the most the reader can hold.", "Cut. PdfReaderLimits.MaxXRefSectionCount is already at the most the reader can hold.", "Cut. Raise PdfReaderLimits.MaxXRefSectionCount to read past it.", "Cut. PdfReaderLimits.MaxIndexEntries is already at the most the reader can hold.", @@ -234,6 +255,95 @@ public void A_stream_decoded_without_diagnostics_reports_the_bound_to_its_docume document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); } + [Fact] + public void A_stream_cut_at_MaxStreamLength_decodes_what_lies_within_it_as_data_the_reader_cut_and_is_reported_once() + { + // A hundred kilobytes of Flate data, which a bound of 10 KB cuts: the data is read up to the bound however often it + // is asked for, its length stays the file's, and what the Flate data lacks past the cut is the guard's, not damage + // in the file (#170). + var compressed = HostileInputTests.Compress(RandomBytes(100_000)); + var written = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = written.Object(1, Catalog), + [2] = written.Object(2, Pages), + [3] = written.Object(3, string.Create(CultureInfo.InvariantCulture, $"<< /Filter /FlateDecode /Length {compressed.Length} >>\nstream\n{Encoding.Latin1.GetString(compressed)}\nendstream")), + }; + written.Table(4, rows, "/Size 4 /Root 1 0 R"); + var file = written.Bytes; + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxStreamLength = 10 * 1024 } }; + using var document = PdfDocument.Open(file, options); + var stream = document.GetObject(new PdfObjectId(3)).AsStream().Required(); + + stream.GetRawBytes().Length.Should().Be(10 * 1024); + stream.GetRawBytes().ToArray().Should().Equal(compressed.AsSpan(0, 10 * 1024).ToArray()); + stream.Data.Length.Should().Be(compressed.Length, "the length is the file's"); + var decoded = stream.Decode(document.Diagnostics); + + decoded.Length.Should().BeGreaterThan(0).And.BeLessThan(100_000); + var report = document.Diagnostics.Should().ContainSingle().Which; + report.Code.Should().Be(PdfDiagnosticCodes.LimitStreamLength); + report.Message.Should().Be("A stream's data runs past 10 KB; only what lies within it was read. Raise PdfReaderLimits.MaxStreamLength to read past it."); + report.Position.Should().Be(stream.Data.Position); + } + + [Theory] + [InlineData("")] + [InlineData("/Filter 5")] + [InlineData("/Filter []")] + [InlineData("/Filter [null]")] + [InlineData("/Filter /DCTDecode")] + [InlineData("/Filter /JPXDecode")] + [InlineData("/Filter /JBIG2Decode")] + [InlineData("/Filter /CCITTFaxDecode")] + [InlineData("/Filter /FlateDecode")] + [InlineData("/Filter /Crypt")] + [InlineData("/Filter /Foo")] + [InlineData("/Filter [/Foo /FlateDecode]")] + public void Data_decoding_hands_back_as_it_is_is_bounded_by_MaxStreamLength_and_not_by_MaxDecodedStreamLength(string filter) + { + // Fifty thousand bytes that no filter decodes — none named, an image filter, one the library does not know, Flate + // over data that is none —: decoding hands them back as they are, whole under a decoded bound of 1 KB, unreported. + // Their bound is the one on what is read, here 10 KB, which a large uncompressed image raises alone (#170). + var file = new TestPdfBuilder() + .WithObject(1, Catalog) + .WithObject(2, Pages) + .Stream(3, filter, new string('4', 50_000)) + .BuildClassic(rootNumber: 1); + var limits = PdfReaderLimits.Default with { MaxStreamLength = 10 * 1024, MaxDecodedStreamLength = 1024 }; + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = limits }); + var stream = document.GetObject(new PdfObjectId(3)).AsStream().Required(); + + var decoded = stream.Decode(document.Diagnostics); + + decoded.Length.Should().Be(10 * 1024); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.LimitStreamLength); + document.Diagnostics.Should().NotContain(entry => entry.Code == PdfDiagnosticCodes.LimitDecodedStream); + } + + [Theory] + [InlineData(2_147_483_600L)] + [InlineData(2_147_483_647L)] + public void A_length_past_what_an_array_holds_ends_at_the_guard_and_never_in_an_exhausted_memory(long declared) + { + // A file of about 2 GB whose stream declares a length between the most an array holds and the most an int does, + // which the file holds: reading its data threw an OutOfMemoryException each time, and reported nothing (#170). Under + // a bound of 1 MB, a megabyte is read and the guard reported; under no bound the guard's ceiling is the array's. + using var source = SparseSource.Huge(declared); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxStreamLength = 1024 * 1024 } }; + using var document = PdfDocument.Open(source, options, ownsSource: false); + var stream = document.GetObject(new PdfObjectId(3)).AsStream().Required(); + + stream.RawLength.Should().Be((int)declared); + stream.GetRawBytes().Length.Should().Be(1024 * 1024); + stream.Decode(document.Diagnostics).Length.Should().Be(1024 * 1024); + document.Diagnostics.Should().ContainSingle().Which.Code.Should().Be(PdfDiagnosticCodes.LimitStreamLength); + + using var throwing = PdfDocument.Open(source, options with { ThrowOnLimit = true }, ownsSource: false); + var reading = () => throwing.GetObject(new PdfObjectId(3)).AsStream().Required().GetRawBytes(); + reading.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxStreamLength)); + } + [Fact] public void An_object_read_again_after_the_cache_let_it_go_is_reported_once_and_throws_each_time() { @@ -679,9 +789,10 @@ public void A_member_dictionary_whose_stream_keyword_MaxObjectLength_leaves_out_ { // Member 3 is a dictionary of 100 bytes and the stream after it, which an object stream cannot hold. The bound ends // the member's window right after the dictionary, inside "stream", or on its last byte, where it may be a longer - // keyword: what follows the dictionary is unknown, as it is of an object written in the file whose window ends - // there, and the member is cut at the guard, not read as a dictionary in silence. A byte past "stream", the member - // is a stream, and reads as null. + // keyword: what follows the dictionary is unknown, and the member is cut at the guard, not read as a dictionary in + // silence — on the last byte, though the byte past it lies in the decoded data, which #282 is to ask, as an object + // written in the file whose window ends there asks the file. A byte past "stream", the member is a stream, and + // reads as null. const string Member = "<< /A (" + "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" + ") >>\nstream\nabc\nendstream"; Member.IndexOf(">>", StringComparison.Ordinal).Should().Be(98); var file = new TestPdfBuilder() @@ -795,6 +906,35 @@ public void A_member_whose_value_ends_where_MaxObjectLength_ends_its_window_is_r document.Diagnostics.Should().AllSatisfy(diagnostic => diagnostic.Message.Should().StartWith("An array was closed by a dictionary end. It was met in object 4,")); } + [Fact] + public void A_member_whose_reference_ends_where_MaxObjectLength_ends_its_window_is_the_reference() + { + // Member 4's window ends where MaxObjectLength does, right after the R of "6 0 R": the decoded byte past it says the + // R is whole, as the file says of an object's, so the member is the reference the window the member would have + // without the guard reads, and nothing is reported, under ThrowOnLimit as without. + using var document = PdfDocument.Open( + MemberReachingItsBound("6 0 R"), + new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = 205 }, ThrowOnLimit = true }); + + document.GetObject(new PdfObjectId(4)).Should().BeOfType().Which.Id.Should().Be(new PdfObjectId(6)); + document.Reader.IsCutAtLimit(4).Should().BeFalse(); + document.Diagnostics.Should().BeEmpty(); + } + + [Fact] + public void A_member_whose_R_the_byte_past_MaxObjectLength_continues_is_cut() + { + // The byte past member 4's window continues the R, "RG": what the window holds may be no reference, and the member + // is one the guard cut. + using var document = PdfDocument.Open( + MemberReachingItsBound("6 0 RG"), new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = 205 } }); + + _ = document.GetObject(new PdfObjectId(4)); + + document.Reader.IsCutAtLimit(4).Should().BeTrue(); + document.Diagnostics.Select(diagnostic => diagnostic.Code).Should().Equal(PdfDiagnosticCodes.LimitObject); + } + [Fact] public void A_member_whose_array_ends_where_a_guard_cuts_the_data_is_read_whole() { @@ -819,6 +959,41 @@ public void A_member_whose_array_ends_where_a_guard_cuts_the_data_is_read_whole( document.Diagnostics.Select(diagnostic => diagnostic.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); } + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_member_whose_R_ends_where_a_guard_cuts_the_data_is_cut(bool throwOnLimit) + { + // Member 10, "6 0 R", ends on byte 10, where MaxDecodedStreamLength cuts the data: what follows the R, which would + // say whether it ends a reference, is the guard's, and nothing past the data is asked. The member reads as the + // number before it, cut, or the guard is thrown. + const string Data = "10 0 6 0 R (more data) "; + var hex = Convert.ToHexString(Encoding.Latin1.GetBytes(Data)) + ">"; + var written = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, written.Object(1, Catalog), 0), + [2] = (1, written.Object(2, Pages), 0), + [3] = (1, written.Object(3, string.Create(CultureInfo.InvariantCulture, $"<< /Type /ObjStm /N 1 /First 5 /Filter /ASCIIHexDecode /Length {hex.Length} >>\nstream\n{hex}\nendstream")), 0), + [10] = (2, 3, 0), + }; + written.CrossReferenceStream(12, 13, rows, "/Root 1 0 R"); + var options = new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = 10 }, ThrowOnLimit = throwOnLimit }; + + using var document = PdfDocument.Open(written.Bytes, options); + + if (throwOnLimit) + { + var reading = () => document.GetObject(new PdfObjectId(10)); + reading.Should().Throw().Which.LimitName.Should().Be(nameof(PdfReaderLimits.MaxDecodedStreamLength)); + return; + } + + document.GetObject(new PdfObjectId(10)).Should().BeOfType().Which.Value.Should().Be(6); + document.Reader.IsCutAtLimit(10).Should().BeTrue(); + document.Diagnostics.Select(diagnostic => diagnostic.Code).Should().Equal(PdfDiagnosticCodes.LimitDecodedStream); + } + [Fact] public void A_member_left_open_where_a_guard_cuts_the_data_at_the_next_member_s_start_is_the_file_s_fault() { @@ -1618,9 +1793,9 @@ public void A_rebuild_that_fills_the_index_keeps_no_member_a_guard_hides_past_it [InlineData("<< /Type /ObjStm /N 0 /First 4 >>")] public void An_object_stream_refused_for_its_own_fault_hides_no_member_where_a_limit_also_cut_its_data(string dictionary) { - // N2-02: object stream 10's dictionary, read whole, gives no usable /N; its data, with no /Length, runs past the - // limit. The stream is refused for the file's fault, and the catalog it holds was taken for one the guard cut: no - // search for catalog 5, file.root-invalid silent, and the validator's probe said the stream was not checked. + // N2-02: object stream 10's dictionary, read whole, gives no usable /N; its data runs past what MaxStreamLength lets + // the reader read. The stream is refused for the file's fault, and the catalog it holds was taken for one the guard + // cut: no search for catalog 5, file.root-invalid silent, and the validator's probe said the stream was not checked. var file = new HandWrittenFile(); var rows = new Dictionary { @@ -1631,9 +1806,10 @@ public void An_object_stream_refused_for_its_own_fault_hides_no_member_where_a_l [1] = (2, 10, 0), }; file.CrossReferenceStream(11, 12, rows, "/Root 1 0 R"); - using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = 256 } }); + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxStreamLength = 256 } }); - document.Reader.IsCutAtLimit(10).Should().BeTrue(); + document.GetObject(new PdfObjectId(10)).AsStream().Required().GetRawBytes().Length.Should().Be(256); + document.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.LimitStreamLength); document.Reader.IsCutAtLimit(1).Should().BeFalse(); document.Reader.Structure.RootCutByGuard.Should().BeFalse(); document.Reader.CatalogFoundAs.Should().Be(5); @@ -3048,6 +3224,25 @@ static string Row(int type, long offset, int generation) => private static int[] Numbers(int first, int count) => [.. Enumerable.Range(first, count)]; + /// + /// Writes a file whose object stream 3 holds member 4, 200 spaces, then and 300 spaces, and + /// member 6, 42: a MaxObjectLength of 205 ends member 4's window right after the R of a reference. + /// + private static byte[] MemberReachingItsBound(string value) + { + var written = new HandWrittenFile(version: "1.5"); + var rows = new Dictionary + { + [1] = (1, written.Object(1, Catalog), 0), + [2] = (1, written.Object(2, Pages), 0), + [3] = (1, written.ObjectStream(3, [(4, new string(' ', 200) + value + new string(' ', 300)), (6, "42")]), 0), + [4] = (2, 3, 0), + [6] = (2, 3, 1), + }; + written.CrossReferenceStream(7, 8, rows, "/Root 1 0 R"); + return written.Bytes; + } + /// /// Writes a file whose chain places object 4 in object stream 10, whose dictionary runs past 128 bytes, and objects 6 and /// 7 in object stream 11, which holds 6 alone; row 9 places its object inside object 8's string, so that settling @@ -3150,9 +3345,75 @@ private static PdfReaderOptions TrailerCut() => private static string LongArray { get; } = "[" + string.Join(' ', Enumerable.Range(0, 2000)) + "]"; + private static byte[] RandomBytes(int length) + { + var bytes = new byte[length]; + new Random(170).NextBytes(bytes); + return bytes; + } + + /// + /// A source of about 2 GB that holds none of it: a head, white space, and a tail at its end, read as the file would + /// be, so that a stream may declare a length no array holds. + /// + private sealed class SparseSource(byte[] head, long tailAt, byte[] tail) : PdfFileSource + { + public override long Length => tailAt + tail.Length; + + /// + /// A file whose object 3 is a stream of bytes, which an endstream after them + /// confirms, its table and trailer past them. + /// + public static SparseSource Huge(long declared) + { + var text = string.Create( + CultureInfo.InvariantCulture, + $"%PDF-1.7\n1 0 obj\n{Catalog}\nendobj\n2 0 obj\n{Pages}\nendobj\n3 0 obj\n<< /Length {declared} >>\nstream\n"); + var head = Encoding.Latin1.GetBytes(text); + var tailAt = head.Length + declared; + var xref = tailAt + "\nendstream\nendobj\n".Length; + var tail = Encoding.Latin1.GetBytes(string.Create( + CultureInfo.InvariantCulture, + $"\nendstream\nendobj\nxref\n0 4\n0000000000 65535 f\r\n{9:D10} 00000 n\r\n{text.IndexOf("2 0 obj", StringComparison.Ordinal):D10} 00000 n\r\n{text.IndexOf("3 0 obj", StringComparison.Ordinal):D10} 00000 n\r\ntrailer\n<< /Size 4 /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n")); + return new SparseSource(head, tailAt, tail); + } + + public override int Read(long offset, Span buffer) + { + var count = (int)Math.Clamp(Length - offset, 0, buffer.Length); + + for (var i = 0; i < count;) + { + var at = offset + i; + int run; + + if (at < head.Length) + { + run = (int)Math.Min(count - i, head.Length - at); + head.AsSpan((int)at, run).CopyTo(buffer[i..]); + } + else if (at < tailAt) + { + run = (int)Math.Min(count - i, tailAt - at); + buffer.Slice(i, run).Fill((byte)' '); + } + else + { + run = (int)Math.Min(count - i, tail.Length - (at - tailAt)); + tail.AsSpan((int)(at - tailAt), run).CopyTo(buffer[i..]); + } + + i += run; + } + + return count; + } + } + private static PdfReaderLimits With(PdfReaderLimits limits, string limit, int value) => limit switch { nameof(PdfReaderLimits.MaxDecodedStreamLength) => limits with { MaxDecodedStreamLength = value }, + nameof(PdfReaderLimits.MaxStreamLength) => limits with { MaxStreamLength = value }, nameof(PdfReaderLimits.MaxObjectLength) => limits with { MaxObjectLength = value }, nameof(PdfReaderLimits.MaxXRefSectionLength) => limits with { MaxXRefSectionLength = value }, nameof(PdfReaderLimits.MaxXRefSectionCount) => limits with { MaxXRefSectionCount = value }, @@ -3165,6 +3426,7 @@ private static PdfReaderOptions TrailerCut() => private static int Bound(PdfReaderLimits limits, string limit) => limit switch { nameof(PdfReaderLimits.MaxDecodedStreamLength) => limits.MaxDecodedStreamLength, + nameof(PdfReaderLimits.MaxStreamLength) => limits.MaxStreamLength, nameof(PdfReaderLimits.MaxObjectLength) => limits.MaxObjectLength, nameof(PdfReaderLimits.MaxXRefSectionLength) => limits.MaxXRefSectionLength, nameof(PdfReaderLimits.MaxXRefSectionCount) => limits.MaxXRefSectionCount, @@ -3302,6 +3564,7 @@ private sealed record Case( public static Case For(string guard) => guard switch { "decoded stream" => DecodedStream(), + "stream length" => LongStream(), "object" => LongObject(), "section length" => LongSection(), "section count" => ManySections(), @@ -3338,6 +3601,29 @@ static int Decoded(PdfDocument document) => document.GetObject(new PdfObjectId(3)).AsStream().Required().Decode(document.Diagnostics).Length; } + private static Case LongStream() + { + // Ten thousand bytes of data no filter decodes, which a bound of a thousand cuts when they are read. + var file = new TestPdfBuilder() + .WithObject(1, Catalog) + .WithObject(2, Pages) + .Stream(3, string.Empty, new string('A', 10_000)) + .BuildClassic(rootNumber: 1); + + return new Case( + nameof(PdfReaderLimits.MaxStreamLength), + PdfDiagnosticCodes.LimitStreamLength, + file, + PdfReaderLimits.Default with { MaxStreamLength = 1000 }, + PdfReaderLimits.Default, + document => document.GetObject(new PdfObjectId(3)).AsStream().Required().GetRawBytes().Length.ToString(CultureInfo.InvariantCulture), + "10000", + "A stream's data runs past 1,000 bytes; only what lies within it was read.", + OffsetOf(file, "stream\n") + "stream\n".Length, + ThrowsOnOpening: false, + ReportsNothingElse: true); + } + private static Case LongFilterChain() { // Three layers of hex around a word, which a bound of two filters leaves one layer of. @@ -3474,7 +3760,7 @@ private static Case LongXRefStreamDictionary() PdfReaderLimits.Default with { MaxTrailerLength = 1024 * 1024 }, IndexState, "as written", - "A cross-reference stream's dictionary runs past 64 KB; only what lies within it was read.", + "The dictionary of the object read as a cross-reference stream runs past 64 KB; only what lies within it was read.", OffsetOf(file, "<< /Type /XRef") - "4 0 obj\n".Length, ThrowsOnOpening: true, ReportsNothingElse: false); @@ -3546,4 +3832,30 @@ private static byte[] WithIncompleteIndex(byte[] file) var at = trailer + "trailer\n<<".Length; return [.. file.AsSpan(0, at), .. " /Prev 1 0 R"u8, .. file.AsSpan(at)]; } + + [Fact] + public void Decoding_into_diagnostics_of_one_s_own_reports_the_cut_max_stream_length_made_each_time() + { + // Object 4's 2,000 bytes run past MaxStreamLength: the reader reported the cut once, in the document's diagnostics, + // and diagnostics a caller passed to Decode never held it, though limit.decoded-stream reaches them each time. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, "<< /Length 2000 >>\nstream\n" + new string('x', 2000) + "\nendstream"), + }; + file.Table(5, rows, "/Size 5 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxStreamLength = 1024 } }); + var stream = document.GetObject(new PdfObjectId(4)).AsStream().Required(); + var mine = new PdfDiagnostics(); + + stream.Decode(mine).Length.Should().Be(1024); + stream.Decode(mine).Length.Should().Be(1024); + + mine.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.LimitStreamLength, PdfDiagnosticCodes.LimitStreamLength); + mine[0].Message.Should().Be( + "A stream's data runs past 1 KB; only what lies within it was read. Raise PdfReaderLimits.MaxStreamLength to read past it."); + document.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.LimitStreamLength).Should().ContainSingle(); + } } diff --git a/tests/AdCodicem.Pdf.Tests/StreamLengthTests.cs b/tests/AdCodicem.Pdf.Tests/StreamLengthTests.cs index ba886f09..702ee8a7 100644 --- a/tests/AdCodicem.Pdf.Tests/StreamLengthTests.cs +++ b/tests/AdCodicem.Pdf.Tests/StreamLengthTests.cs @@ -104,9 +104,11 @@ public void A_stream_past_the_window_whose_length_is_long_ends_at_the_endstream_ } [Fact] - public void A_stream_past_the_window_with_no_endstream_before_the_next_object_keeps_its_declared_length() + public void A_stream_past_the_window_with_no_endstream_before_the_next_object_ends_at_its_object_s_endobj() { - // The endstream the search would find next is object 6's: the next object bounds it. + // The endstream the search would find next is object 6's: the next object bounds it. No endstream before it, the + // object's own endobj ends the data, its end-of-line left out, rather than a declared length that runs on into + // object 6 (#174). var data = Data(3 * Window); var file = Reachable((StreamNumber, Stream(data.Length + 100, data, end: "\n")), (6, Stream(5, "hello"))); var dataStart = DataStartOf(file, StreamNumber); @@ -115,22 +117,24 @@ public void A_stream_past_the_window_with_no_endstream_before_the_next_object_ke using var document = PdfDocument.Open(file); var stream = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); - stream.Data.Length.Should().Be(data.Length + 100); - document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( - Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows them before the next object, at {next}; the declared length is kept.")); + stream.Data.Length.Should().Be(data.Length + 1); + var diagnostic = document.Diagnostics.Should().ContainSingle().Which; + diagnostic.Code.Should().Be(PdfDiagnosticCodes.StreamTruncated); + diagnostic.Message.Should().Be( + Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows its data before the next object, at {next}: its object's endobj ends it, after {data.Length + 1} bytes.")); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); fault.Should().Be(new StreamLengthFault { Form = StreamLengthForm.Integer, Declared = data.Length + 100, - Taken = data.Length + 100, - EndStream = EndStreamState.MissingBeforeNextObject, + Taken = data.Length + 1, + EndStream = EndStreamState.MissingBeforeEndObj, NextObject = next, DataStart = dataStart, }); - // Where the object ends is unknown, and its endobj is not judged. + // The object ends with its own endobj. document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse(); new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectEndObjMissing).Should().BeFalse(); } @@ -153,20 +157,21 @@ public void A_stream_past_the_window_with_no_endstream_before_the_end_of_the_fil } [Fact] - public void A_stream_past_the_window_with_no_endstream_before_an_object_the_window_holds_keeps_its_declared_length() + public void A_stream_past_the_window_with_no_endstream_before_an_object_the_window_holds_ends_at_its_endobj_unsearched() { - // The next object starts inside the window: the part of the window before it is all the search reads. + // The next object starts inside the window: the part of the window before it is all the search reads, and the + // object's endobj before it ends the data (#174). var file = Document((StreamNumber, Stream(3 * Window, "short data", end: string.Empty)), (6, "(" + new string('p', 4 * Window) + ")")); using var source = new StrictCountingSource(file); using var document = PdfDocument.Open(source, options: null, ownsSource: false); var beforeLoading = source.BytesRead; - document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(3 * Window); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be("short data".Length); (source.BytesRead - beforeLoading).Should().Be(Window + PdfObjectParser.EndObjLookahead, "the file past the window is not searched"); document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( - Invariant($"The stream declared {3 * Window} bytes, and no endstream follows them before the next object, at {HeaderOf(file, 6)}; the declared length is kept.")); + Invariant($"The stream declared {3 * Window} bytes, and no endstream follows its data before the next object, at {HeaderOf(file, 6)}: its object's endobj ends it, after 10 bytes.")); } [Fact] @@ -186,6 +191,108 @@ public void A_stream_whose_declared_length_ends_the_file_without_endstream_keeps document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse("where the object ends is unknown"); } + [Theory] + [InlineData(" ", false)] + [InlineData(" ", true)] + [InlineData("\r\n% a comment between the data and its keyword\r\n", false)] + [InlineData("\r\n% a comment between the data and its keyword\r\n", true)] + [InlineData("\t\f\0 \r\n ", false)] + [InlineData("\t\f\0 \r\n ", true)] + public void A_length_any_white_space_or_comment_separates_from_endstream_is_confirmed(string gap, bool pastWindow) + { + // ISO 32000-1 (7.3.8.1) only says an end-of-line should precede endstream, and 7.2.3 makes a comment white space: + // five spaces, or a comment, after a length that was right made it a wrong one, the gap taken into the data, and + // the file past the window searched (#174). Past the window, the bytes after the declared length are all it reads. + var data = Data(pastWindow ? 3 * Window : 100); + var file = Document((StreamNumber, Stream(data.Length, data, end: gap + "endstream")), (6, "(six)")); + + using var source = new StrictCountingSource(file); + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + var beforeLoading = source.BytesRead; + var stream = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); + + stream.Data.Length.Should().Be(data.Length); + document.Diagnostics.Should().BeEmpty(); + document.Reader.TryGetStreamLengthFault(StreamNumber, out _).Should().BeFalse(); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse(); + + if (pastWindow) + { + (source.BytesRead - beforeLoading).Should().BeLessThanOrEqualTo(Window + (2 * PdfObjectParser.EndObjLookahead) + 8, "the window, then the bytes after the declared length"); + } + } + + [Theory] + [InlineData(3 * 8192, 100)] + [InlineData(3 * 8192, 5_000)] + [InlineData(3 * 8192, 200_000)] + [InlineData(8192 - 200, 300)] + public void White_space_before_endstream_that_runs_past_the_first_read_is_read_on_and_confirms_the_length(int length, int spaces) + { + // White space after the data longer than the reader reads at first, past the window or across its edge: the file is + // read on as far as it runs, and the endstream after it confirms the length (#174). + var data = Data(length); + var file = Document((StreamNumber, Stream(data.Length, data, end: new string(' ', spaces) + "endstream")), (6, "(six)")); + + using var document = PdfDocument.Open(file); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length); + + document.Diagnostics.Should().BeEmpty(); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse(); + } + + [Theory] + [InlineData("<< /Length 3 >>", false)] + [InlineData("<< /Length 3 >>", true)] + [InlineData("<< /Length -1 >>", false)] + [InlineData("<< /Length 2147483647 >>", false)] + [InlineData("<< /Length 2147483647 >>", true)] + [InlineData("<< /Length 20000 >>", false)] + [InlineData("<< /Length 9 0 R >>", false)] + [InlineData("<< >>", false)] + [InlineData("<< >>", true)] + public void A_stream_that_lost_its_endstream_takes_nothing_of_the_object_after_it(string dictionary, bool pastWindow) + { + // Object 5 lost its endstream, and object 6, a stream, follows it. The search in the window took object 5's endobj, + // object 6's header and its data up to its endstream — 49 bytes for "hello" —, and a declared length past it was + // kept, whatever the search found. The search stops at object 6, the next object, and object 5's own endobj ends + // its data, its end-of-line left out, wherever its /Length points (#174). + var data = pastWindow ? Data(3 * Window) : "hello"; + var file = Document( + (StreamNumber, dictionary + "\nstream\n" + data), + (6, Stream(5, "world")), + (7, "(" + new string('p', 30_000) + ")")); + + using var document = PdfDocument.Open(file); + var stream = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); + + Encoding.ASCII.GetString(stream.GetRawBytes().Span).Should().Be(data); + var diagnostic = document.Diagnostics.Should().ContainSingle().Which; + diagnostic.Code.Should().Be(PdfDiagnosticCodes.StreamTruncated); + diagnostic.Message.Should().EndWith(Invariant($"before the next object, at {HeaderOf(file, 6)}: its object's endobj ends it, after {data.Length} bytes.")); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse("the object ends with its own endobj"); + + Encoding.ASCII.GetString(document.GetObject(new PdfObjectId(6)).AsStream().Required().GetRawBytes().Span).Should().Be("world"); + document.Diagnostics.Should().ContainSingle("object 6 is read alone, whole"); + } + + [Fact] + public void A_stream_that_lost_its_endstream_runs_to_a_header_its_object_has_no_endobj_before_and_lacks_its_endobj() + { + // No endstream, no endobj: the data runs to the next object's header, which no declared length passes, and the + // object ends without its endobj (#174). + var data = Data(3 * Window); + var file = Document((StreamNumber, Stream(5 * Window, data, end: "\n7 0 obj\n(seven)")), (6, "(six)")); + var seven = Encoding.Latin1.GetString(file).IndexOf("\n7 0 obj", StringComparison.Ordinal) + 1; + + using var document = PdfDocument.Open(file); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length + 1); + + document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( + Invariant($"The stream declared {5 * Window} bytes, and no endstream follows its data before the next object, at {seven}; the {data.Length + 1} bytes up to that object are taken as its data.")); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeTrue(); + } + [Fact] public void The_search_stops_at_an_object_the_index_as_written_places_though_a_rebuilt_index_lacks_it() { @@ -218,7 +325,8 @@ public void The_search_stops_at_an_object_the_index_as_written_places_though_a_r [Fact] public void The_search_stops_at_the_next_object_of_an_index_rebuilt_as_the_document_opened() { - // No index was written: the one the rebuild made is the only one, and object 6's endstream is past its header. + // No index was written: the one the rebuild made is the only one, and object 6's endstream is past its header. The + // object's endobj before it ends the data (#174). var data = Data(3 * Window); var file = new TestPdfBuilder() .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") @@ -231,9 +339,9 @@ public void The_search_stops_at_the_next_object_of_an_index_rebuilt_as_the_docum document.WasRepaired.Should().BeTrue(); var stream = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); - stream.Data.Length.Should().Be(data.Length - 7); - document.Diagnostics.Where(d => d.Code == PdfDiagnosticCodes.StreamLengthInvalid).Should().ContainSingle() - .Which.Message.Should().EndWith(Invariant($"before the next object, at {HeaderOf(file, 6)}; the declared length is kept.")); + stream.Data.Length.Should().Be(data.Length + 1); + document.Diagnostics.Where(d => d.Code == PdfDiagnosticCodes.StreamTruncated).Should().ContainSingle() + .Which.Message.Should().EndWith(Invariant($"before the next object, at {HeaderOf(file, 6)}: its object's endobj ends it, after {data.Length + 1} bytes.")); } [Fact] @@ -241,7 +349,7 @@ public void The_search_stops_at_an_object_header_the_bytes_hold_before_the_next_ { // Object 5 lost its endstream, and object 12's header follows it, an object no index places: the index as written // places object 6 next, past object 12's endstream. The header the bytes hold is the nearer, and the search stops - // there rather than take in object 12. + // there rather than take in object 12; object 5's endobj before it ends the data (#174). var data = Data(3 * Window); var file = Document( (StreamNumber, Stream(data.Length + 400, data, end: "\nendstreax\nendobj\n12 0 obj\n<< /Length 5 >>\nstream\nhello\nendstream")), @@ -252,20 +360,20 @@ public void The_search_stops_at_an_object_header_the_bytes_hold_before_the_next_ using var document = PdfDocument.Open(file); var stream = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); - stream.Data.Length.Should().Be(data.Length + 400); + stream.Data.Length.Should().Be(data.Length + "\nendstreax".Length); HeaderOf(file, 6).Should().BeGreaterThan(twelve); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); fault.Should().Be(new StreamLengthFault { Form = StreamLengthForm.Integer, Declared = data.Length + 400, - Taken = data.Length + 400, - EndStream = EndStreamState.MissingBeforeNextObject, + Taken = data.Length + "\nendstreax".Length, + EndStream = EndStreamState.MissingBeforeEndObj, NextObject = twelve, DataStart = dataStart, }); document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( - Invariant($"The stream declared {data.Length + 400} bytes, and no endstream follows them before the next object, at {twelve}; the declared length is kept.")); + Invariant($"The stream declared {data.Length + 400} bytes, and no endstream follows its data before the next object, at {twelve}: its object's endobj ends it, after {data.Length + "\nendstreax".Length} bytes.")); } [Theory] @@ -304,7 +412,8 @@ public void Text_that_follows_a_lost_endstream_stops_the_search_when_it_is_an_ob { // An object header is "N G obj" at a token boundary, as the bytes hold it: a number and a generation whose values // the parser takes, however many zeros lead them, and white space between its tokens, however much. Text that is - // one stops the search before the endstream after it, and the declared length is kept; text that only looks like + // one stops the search before the endstream after it, and the data, which no declared length runs past, ends there + // (#174); text that only looks like // one — endobj, obj glued to what follows or precedes it, digits after a regular character, a number or a // generation out of range, a comment between its tokens — is read past, and the endstream after it ends the data. var data = Data(3 * Window); @@ -321,11 +430,11 @@ public void Text_that_follows_a_lost_endstream_stops_the_search_when_it_is_an_ob if (header is { } at) { var start = dataStart + data.Length + 1 + at; - stream.Data.Length.Should().Be(data.Length + 400); + stream.Data.Length.Should().Be(data.Length + 1 + at); fault.EndStream.Should().Be(EndStreamState.MissingBeforeNextObject); fault.NextObject.Should().Be(start); document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( - Invariant($"The stream declared {data.Length + 400} bytes, and no endstream follows them before the next object, at {start}; the declared length is kept.")); + Invariant($"The stream declared {data.Length + 400} bytes, and no endstream follows its data before the next object, at {start}; the {data.Length + 1 + at} bytes up to that object are taken as its data.")); } else { @@ -357,7 +466,7 @@ public void The_start_and_the_end_of_the_search_bound_an_object_header_as_white_ } using var document = PdfDocument.Open(file); - document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length + 400); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be((int)(header - dataStart), "the data runs to the header"); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); fault.NextObject.Should().Be(header); @@ -378,7 +487,7 @@ public void A_header_the_edge_between_two_reads_of_the_search_cuts_is_decided_wh // the same with a regular character before or after it, or a number or a generation out of range, which are none. // What a read's start cuts of a header, the read before carries into it. The start of a read is no boundary before // a number, nor its end one after obj, unless the search starts or ends there. A header missed, the endstream after - // it would end the data; text taken for one, the declared length would be kept. + // it would end the data; text taken for one, the data would end there. var prefix = Invariant($"{StreamNumber} 0 obj\n") + Stream(0, string.Empty, end: string.Empty, width: 7); var read = PdfFileReader.EndStreamSearchFirstRead(Window - prefix.Length); var first = Window - PdfFileReader.EndStreamSearchOverlap + read; @@ -402,7 +511,7 @@ public void A_header_the_edge_between_two_reads_of_the_search_cuts_is_decided_wh if (header) { - stream.Data.Length.Should().Be(length + 400, because); + stream.Data.Length.Should().Be(length + 1, because); fault.NextObject.Should().Be(HeaderOf(file, StreamNumber) + at, because); } else @@ -443,7 +552,7 @@ public void A_header_whose_runs_span_whole_reads_of_the_search_is_decided_as_its if (header) { - stream.Data.Length.Should().Be(declared); + stream.Data.Length.Should().Be(data.Length + 1, "the data runs to the header"); fault.NextObject.Should().Be(dataStart + data.Length + 1, "the header starts where its number does, its zeros with it"); } else @@ -475,7 +584,8 @@ public void Streams_only_a_rebuilt_index_places_stop_at_the_next_header_and_hold // all, declares 100 bytes more than its data, which its endstream follows. Bounded by the index as written alone, // each of 10 to 38 ran to object 39's endstream: the searches read the file over and over and spent the document's // budget, object 50 was searched only when read before the rebuild, and it held 30,000 bytes or 30,100 according - // to the order. Each stream now stops at the header of the one after it, and holds the same data in either order, + // to the order. Each stream now stops at the header of the one after it, ends at its own endobj before that header + // (#174), and holds the same data in either order, // however the headers are written that both the rebuild and the parser take: more white space than a read of the // search repeats, zeros leading the generation. The searches cannot know where that header is before they read // it: reading the file 64 KB at a time, or 16 KB at first whatever window the object was parsed through, the @@ -521,9 +631,9 @@ public void Streams_only_a_rebuilt_index_places_stop_at_the_next_header_and_hold for (var number = 10; number < 39; number++) { - document.GetObject(new PdfObjectId(number)).AsStream().Required().Data.Length.Should().Be(data.Length + 100); + document.GetObject(new PdfObjectId(number)).AsStream().Required().Data.Length.Should().Be(data.Length); document.Reader.TryGetStreamLengthFault(number, out var fault).Should().BeTrue(); - fault.EndStream.Should().Be(EndStreamState.MissingBeforeNextObject, "the document's searches have read far less than they may"); + fault.EndStream.Should().Be(EndStreamState.MissingBeforeEndObj, "the document's searches have read far less than they may"); fault.NextObject.Should().Be(text.IndexOf("\n" + Header(header, number + 1) + "\n", StringComparison.Ordinal) + 1); } @@ -623,16 +733,20 @@ 0 6 new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectEndObjMissing).Should().Be(missing); } - [Fact] - public void What_follows_the_endstream_of_a_stream_past_the_window_is_unseen_past_a_few_bytes() + [Theory] + [InlineData("\nendobj\n", false)] + [InlineData("\n8 0 obj\n", true)] + public void What_follows_the_endstream_of_a_stream_past_the_window_is_seen_however_far_white_space_runs(string next, bool missing) { + // Past a few bytes of white space after endstream, what followed went unseen, and the object's endobj unjudged: the + // file is read on as far as white space runs (#174). var data = Data(3 * Window); - var file = Document((StreamNumber, Stream(data.Length, data, end: "\nendstream" + new string(' ', PdfObjectParser.EndObjLookahead)))); + var file = Document((StreamNumber, Stream(data.Length, data, end: "\nendstream" + new string(' ', 200) + next + "(unwritten)"))); using var document = PdfDocument.Open(file); - document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length); - document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse("what follows is not seen, and not judged"); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().Be(missing); } [Fact] @@ -969,9 +1083,39 @@ public void A_parser_without_diagnostics_reads_a_stream_whose_length_is_wrong_to [Theory] [InlineData(false)] [InlineData(true)] - public void A_stream_that_endobj_follows_without_its_endstream_says_what_it_takes(bool pastWindow) + public void A_stream_that_endobj_follows_without_its_endstream_keeps_its_declared_length(bool pastWindow) + { + // iPRES's t02-05-01-014: its /Length is right, and its object's endobj follows the data without an endstream. The data + // ran to the end of the file, through object 6, the table and the trailer; the endobj plays the endstream's part now, + // and the declared length is kept, in the window or past it (#174). + var data = Data(pastWindow ? 3 * Window : 68); + var file = Encoding.ASCII.GetBytes( + "%PDF-1.7\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n2 0 obj\n<< /Type /Pages /Kids [] /Count 0 >>\nendobj\n" + + Invariant($"5 0 obj\n<< /Length {data.Length} >>\nstream\n{data}\r\n\n\nendobj \n6 0 obj\n(six)\nendobj\ntrailer\n<< /Root 1 0 R >>\n%%EOF\n")); + var dataStart = DataStartOf(file, StreamNumber); + + using var document = PdfDocument.Open(file); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length); + + var diagnostic = document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamTruncated).Which; + diagnostic.Message.Should().Be(Invariant($"The stream has no endstream: its object's endobj follows the {data.Length} bytes it declared, which are taken as its data.")); + diagnostic.Position.Should().Be(dataStart); + document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); + fault.EndStream.Should().Be(EndStreamState.MissingBeforeEndObj); + fault.Taken.Should().Be(data.Length); + fault.Found.Should().BeNull(); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse("the object ends with its endobj"); + document.GetObject(new PdfObjectId(6)).Should().BeOfType(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_stream_whose_endobj_the_trailer_follows_runs_to_the_end_of_the_file_and_leaves_its_endobj_unjudged(bool pastWindow) { - // iPRES's t02-05-01-014: the data runs to the end of the file, which the endobj after it does not stop. + // A /Length 20,000 bytes too long, an endobj after the data, then the trailer: no endstream, no object after it, and + // the endobj does not end the file, which is where the search stops. The data runs to the end of the file, and + // holds that endobj, which may be the object's own: whether the object ends with one is not judged (#174). var data = Data(pastWindow ? 3 * Window : 60); var file = Encoding.ASCII.GetBytes( "%PDF-1.7\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n2 0 obj\n<< /Type /Pages /Kids [] /Count 0 >>\nendobj\n" + @@ -983,12 +1127,11 @@ public void A_stream_that_endobj_follows_without_its_endstream_says_what_it_take document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be((int)rest); var diagnostic = document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamTruncated).Which; - diagnostic.Message.Should().Be(Invariant($"The stream has no endstream before the endobj that follows its data; the {rest} bytes to the end of the file are taken as its data.")); - diagnostic.Position.Should().Be(dataStart); + diagnostic.Message.Should().Be(Invariant($"The stream declared {data.Length + 20000} bytes, and no endstream follows its data before the end of the file; the {rest} bytes to the end of the file are taken as its data.")); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); - fault.EndStream.Should().Be(EndStreamState.MissingBeforeEndObj); + fault.EndStream.Should().Be(EndStreamState.MissingBeforeEndOfFile); fault.Taken.Should().Be((int)rest); - fault.Found.Should().BeNull(); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse("the data holds an endobj, which may be the object's own"); } [Fact] @@ -1003,7 +1146,7 @@ public void A_stream_the_end_of_the_file_cuts_records_what_it_took() document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length); document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamTruncated) - .Which.Message.Should().Be("A stream ran past the end of the file."); + .Which.Message.Should().Be("The stream declared 20000 bytes, and no endstream follows its data before the end of the file; the 200 bytes to the end of the file are taken as its data."); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); fault.Should().Be(new StreamLengthFault { @@ -1016,17 +1159,41 @@ public void A_stream_the_end_of_the_file_cuts_records_what_it_took() } [Fact] - public void A_stream_a_limit_cuts_records_nothing_of_its_length() + public void A_stream_s_data_counts_for_nothing_toward_MaxObjectLength_and_its_fault_is_the_file_s() { - // The window stopped at the guard, not at the end of the file: what the parser found there is the limit's. + // A stream with no /Length, its data three windows long, under a MaxObjectLength of one window: the window grew to + // find its endstream, reached the guard, and the guard was reported in place of the stream's fault. Stream data is + // searched past the window now, and counts for nothing toward the object's length (#174). var file = Document((StreamNumber, "<< >>\nstream\n" + Data(3 * Window) + "\nendstream")); var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = Window } }; using var document = PdfDocument.Open(file, options); - document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(3 * Window); + + document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be( + Invariant($"The stream has no /Length; its data ends after {3 * Window} bytes.")); + document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); + fault.EndStream.Should().Be(EndStreamState.Found); + document.Reader.IsCutAtLimit(StreamNumber).Should().BeFalse(); + } + + [Fact] + public void A_stream_whose_keyword_a_limit_cuts_records_nothing_of_its_length() + { + // The window MaxObjectLength allows ends inside the stream keyword: whether a stream follows the dictionary is past + // it, the guard's, and what the parser found of the object is the limit's. A window that ends on the whole keyword + // reads the stream as a larger one does: its data counts toward no window (#174). + var header = Invariant($"{StreamNumber} 0 obj\n"); + var dictionary = "<< /Pad (" + new string('x', 1024 - header.Length - "<< /Pad () >>\nstream".Length) + ") >>\nstream"; + var file = Document((StreamNumber, dictionary + "\n" + Data(100) + "\nendstream")); + var options = PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = 1024 - "eam".Length } }; + + using var document = PdfDocument.Open(file, options); + document.GetObject(new PdfObjectId(StreamNumber)); document.Diagnostics.Should().ContainSingle().Which.Code.Should().Be(PdfDiagnosticCodes.LimitObject); document.Reader.TryGetStreamLengthFault(StreamNumber, out _).Should().BeFalse(); + document.Reader.IsCutAtLimit(StreamNumber).Should().BeTrue(); } [Fact] @@ -1064,7 +1231,8 @@ public void The_next_object_a_search_stops_at_is_placed_from_the_header_junk_pre // Six hundred bytes of junk come before %PDF-, and the index counts from there. Declared 300 bytes short, the // stream's endstream lies past where object 6's entry would place it counted from the start of the file: it is // found only when the bound is placed from the header. Without an endstream, the report names where object 6 - // starts in the file, and the search stops there rather than at object 6's endstream. + // starts in the file, and the search stops there rather than at object 6's endstream: object 5's endobj before it + // ends the data (#174). const int Junk = 600; var data = Data(3 * Window); var built = endStreamFollowsData @@ -1081,7 +1249,8 @@ public void The_next_object_a_search_stops_at_is_placed_from_the_header_junk_pre document.Reader.HeaderOffset.Should().Be(Junk); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); - var message = document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which.Message; + var code = endStreamFollowsData ? PdfDiagnosticCodes.StreamLengthInvalid : PdfDiagnosticCodes.StreamTruncated; + var message = document.Diagnostics.Should().ContainSingle(d => d.Code == code).Which.Message; if (endStreamFollowsData) { @@ -1090,10 +1259,10 @@ public void The_next_object_a_search_stops_at_is_placed_from_the_header_junk_pre } else { - stream.Data.Length.Should().Be(data.Length + 100); + stream.Data.Length.Should().Be(data.Length + 1); fault.NextObject.Should().Be(next); message.Should().Be( - Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows them before the next object, at {next}; the declared length is kept.")); + Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows its data before the next object, at {next}: its object's endobj ends it, after {data.Length + 1} bytes.")); } } @@ -1188,7 +1357,8 @@ public void A_stream_past_the_window_holds_the_same_data_whether_it_is_read_befo // Object 5 lost its endstream. Object 7, which the index as written marks free, follows it; object 8's entry // leads nowhere near it, and asking for it rebuilds the index, which places object 7. The index as written places // nothing between object 5 and object 6, but the bytes hold object 7's header: read before the rebuild or after - // it, the search stops there, and the stream keeps its declared length rather than object 7's header and data. The + // it, the search stops there, and the stream ends at its own endobj before it rather than take in object 7's header + // and data (#174). The // rebuilt index is not asked, which would have bounded a stream read after the rebuild only: its data would depend // on what was read first. var data = Data(3 * Window); @@ -1217,13 +1387,13 @@ public void A_stream_past_the_window_holds_the_same_data_whether_it_is_read_befo document.WasRepaired.Should().BeTrue(); var again = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); - first.Data.Length.Should().Be(data.Length + 400); - again.Data.Length.Should().Be(data.Length + 400); + first.Data.Length.Should().Be(data.Length); + again.Data.Length.Should().Be(data.Length); document.GetObject(new PdfObjectId(7)).AsStream().Required().Data.Length.Should().Be(5, "the rebuilt index serves object 7 in its own right"); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); fault.NextObject.Should().Be(seven); - document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which.Message.Should().Be( - Invariant($"The stream declared {data.Length + 400} bytes, and no endstream follows them before the next object, at {seven}; the declared length is kept.")); + document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamTruncated).Which.Message.Should().Be( + Invariant($"The stream declared {data.Length + 400} bytes, and no endstream follows its data before the next object, at {seven}: its object's endobj ends it, after {data.Length} bytes.")); } [Theory] @@ -1234,7 +1404,8 @@ public void A_stream_past_the_window_stops_where_the_index_as_written_places_the // As the IBM QMF manual's entries do, object 16's points one byte into its header, at "6 0 obj": reading object 16 // finds it a byte before, and corrects the index the reader reads with. The search for object 5's endstream, which // it lost, is bounded by the index as the file wrote it, and stops at the same place whether object 16 was read and - // found first or not. + // found first or not. Its endobj lies before that place, but a byte of object 16's header lies between: the data + // runs to the place the index gives, and holds the endobj, whether the object ends there unknown (#174). var data = Data(3 * Window); var file = Document((StreamNumber, Stream(data.Length + 100, data, end: "\n")), (16, "(" + new string('p', 200) + ")")); var header = HeaderOf(file, 16); @@ -1252,11 +1423,13 @@ public void A_stream_past_the_window_stops_where_the_index_as_written_places_the var stream = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); - stream.Data.Length.Should().Be(data.Length + 100); + var taken = (int)(header + 1 - DataStartOf(file, StreamNumber)); + stream.Data.Length.Should().Be(taken); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); fault.NextObject.Should().Be(header + 1, "the index as written places object 16 there"); document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which.Message.Should().Be( - Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows them before the next object, at {header + 1}; the declared length is kept.")); + Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows its data before the next object, at {header + 1}; the {taken} bytes up to that object are taken as its data.")); + document.Reader.IsEndObjMissing(StreamNumber, out _).Should().BeFalse("the data holds an endobj, which may be the object's own"); } [Theory] @@ -1296,10 +1469,10 @@ public void An_index_rebuilt_as_the_document_opened_bounds_the_searches_as_it_st document.GetObject(new PdfObjectId(number)); } - document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length + 100); + document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required().Data.Length.Should().Be(data.Length + 1); document.GetObject(new PdfObjectId(7)).AsStream().Required().Data.Length.Should().Be(5); - document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which.Message.Should().Be( - Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows them before the next object, at {HeaderOf(file, 6)}; the declared length is kept.")); + document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamTruncated).Which.Message.Should().Be( + Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows its data before the next object, at {HeaderOf(file, 6)}: its object's endobj ends it, after {data.Length + 1} bytes.")); source.Searches(dataStart, data.Length).Should().Be(0, "the stream was searched as the document opened"); document.Reader.Index.Entries.Should().Equal(opened, "nothing read after the document opened changes an index rebuilt as it opened"); } @@ -1315,7 +1488,8 @@ public void The_next_object_a_search_stops_at_is_where_the_index_as_written_plac // its endstream (#182). Once the chain is read, the /Length is read and checked: object 10 is found near its entry, // which is corrected behind a copy of the index as the file wrote it. Object 5 lost its endstream; its search stops // where that index places the next object, 2 bytes before object 10's header, whether reading object 6 — whose entry - // points one byte into its header — corrected the index the reader reads with since, or not. + // points one byte into its header — corrected the index the reader reads with since, or not. That place cuts object + // 5's endobj: the data runs to it (#174). var data = Data(3 * Window); var file = LengthObjectOffItsEntry(data, out var tenAt); @@ -1332,12 +1506,13 @@ public void The_next_object_a_search_stops_at_is_where_the_index_as_written_plac var stream = document.GetObject(new PdfObjectId(StreamNumber)).AsStream().Required(); - stream.Data.Length.Should().Be(data.Length + 100); + var taken = (int)(tenAt - 2 - DataStartOf(file, StreamNumber)); + stream.Data.Length.Should().Be(taken); document.Reader.TryGetStreamLengthFault(StreamNumber, out var fault).Should().BeTrue(); fault.NextObject.Should().Be(tenAt - 2); document.Diagnostics.Should().ContainSingle(d => d.Code == PdfDiagnosticCodes.StreamLengthInvalid && d.Position == DataStartOf(file, StreamNumber)) .Which.Message.Should().Be( - Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows them before the next object, at {tenAt - 2}; the declared length is kept.")); + Invariant($"The stream declared {data.Length + 100} bytes, and no endstream follows its data before the next object, at {tenAt - 2}; the {taken} bytes up to that object are taken as its data.")); } [Theory] @@ -1374,30 +1549,7 @@ public void Streams_whose_data_share_a_stretch_read_the_file_a_bounded_number_of // it a few times over together; then a stream keeps its declared length without a search, and says so. const int Streams = 30; const int Declared = 100_000; - var nested = Invariant($"{Streams + 2} 0 obj<>stream\n"); - - for (var number = Streams + 1; number >= 3; number--) - { - nested = Invariant($"{number} 0 obj<>stream\n"); - } - - var head = "%PDF-1.7\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n2 0 obj\n<< /Type /Pages /Kids [] /Count 0 >>\nendobj\n" + nested; - var offsets = new List { 0, 9, head.IndexOf("2 0 obj", StringComparison.Ordinal) }; - - for (var number = 3; number <= Streams + 2; number++) - { - offsets.Add(head.IndexOf(Invariant($"{number} 0 obj<<"), StringComparison.Ordinal)); - } - - var body = head + new string('x', 1024 * 1024) + "\n"; - var xref = new StringBuilder(Invariant($"xref\n0 {offsets.Count}\n0000000000 65535 f\r\n")); - - foreach (var offset in offsets.Skip(1)) - { - xref.Append(Invariant($"{offset:D10} 00000 n\r\n")); - } - - var file = Encoding.Latin1.GetBytes(body + xref + Invariant($"trailer\n<< /Size {offsets.Count} /Root 1 0 R >>\nstartxref\n{body.Length}\n%%EOF\n")); + var (file, head) = NestedStreams(Streams, Declared); using var source = new StrictCountingSource(file); using var document = PdfDocument.Open(source, options: null, ownsSource: false); @@ -1594,6 +1746,26 @@ public void A_remote_document_reads_the_stream_whose_length_is_wrong_as_the_meas .Which.Message.Should().Be(message); } + [Fact] + public void IPRES_s_stream_that_lost_its_endstream_keeps_the_length_its_endobj_follows() + { + // t02-05-01-014: the 68 bytes its /Length declares, which its endobj follows without an endstream. It took the 310 + // bytes to the end of the file, through object 5, the table and the trailer (#174). + const string File = "remote/ipres2017/t02-05-01-014-endstream-keyword-missing.pdf"; + Assert.SkipUnless(Corpus.Paths.Contains(File), $"{File} is a remote document this machine did not fetch"); + using var document = PdfDocument.Open(Corpus.Read(File), CorpusOptions(Corpus.Get(File))); + + foreach (var each in document.ObjectNumbers.ToList()) + { + document.GetObject(new PdfObjectId(each)); + } + + document.Reader.TryGetStreamLengthFault(4, out var fault).Should().BeTrue(); + fault.Taken.Should().Be(68); + document.Diagnostics.Should().ContainSingle(d => d.Position == fault.DataStart).Which.ToString().Should().Be( + "Warning stream.truncated at 315: The stream has no endstream: its object's endobj follows the 68 bytes it declared, which are taken as its data."); + } + private static readonly (string File, int Number, long DataStart, int Length, string Message)[] RemoteRows = [ ("remote/opf-format-corpus/ibm-id-workbench-xpp-qmf-manual.pdf", 93, 89018, 115264, "The stream declared 115266 bytes but ended after 115264."), @@ -1609,6 +1781,7 @@ private static readonly (string File, int Number, long DataStart, int Length, st ("remote/opf-format-corpus/quartz-word-samhsa-prevention-pathways-fact-sheet.pdf", 72, 145068, 8458, "The stream declared 8434 bytes but ended after 8458."), ("remote/opf-format-corpus/jhove-hul-35-atypon-pdfplus-journal-article.pdf", 49, 249330, 1647, "The stream declared 62065 bytes but ended after 1647."), ("remote/opf-format-corpus/jhove-hul-35-pdfium-urban-planning-report.pdf", 695, 19901349, 202154, "The stream declared 202154 bytes, and no endstream follows them before the next object, at 20103524; the declared length is kept."), + ("remote/pdfbox/distiller6-zeroed-object-stream-pdfbox3947.pdf", 417, 1670741, 1302, "The stream declared 1302 bytes, and no endstream follows them before the next object, at 1672062; the declared length is kept."), ("remote/ipres2017/t02-05-01-015-invalid-stream-length-in-dictionary.pdf", 4, 316, 68, "The stream's /Length is a real number, 61.5, not a non-negative integer; its data ends after 68 bytes."), ("remote/ipres2017/t02-05-01-016-stream-length-missing-in-dictionary.pdf", 4, 304, 68, "The stream has no /Length; its data ends after 68 bytes."), ("remote/opf-format-corpus/jhove-hul-157-tiff2pdf-1996-journal-article-scan.pdf", 16, 542577, 5555, "The stream's /Length names object 18 0, which holds a stream, not a non-negative integer; its data ends after 5555 bytes."), @@ -1631,6 +1804,7 @@ private static PdfReaderOptions CorpusOptions(CorpusDocument entry) => entry.Rea Limits = PdfReaderLimits.Default with { MaxDecodedStreamLength = raised.MaxDecodedStreamLength ?? PdfReaderLimits.Default.MaxDecodedStreamLength, + MaxStreamLength = raised.MaxStreamLength ?? PdfReaderLimits.Default.MaxStreamLength, MaxObjectLength = raised.MaxObjectLength ?? PdfReaderLimits.Default.MaxObjectLength, MaxXRefSectionLength = raised.MaxXRefSectionLength ?? PdfReaderLimits.Default.MaxXRefSectionLength, MaxXRefSectionCount = raised.MaxXRefSectionCount ?? PdfReaderLimits.Default.MaxXRefSectionCount, @@ -1873,4 +2047,397 @@ public override int Read(long offset, Span buffer) return count; } } + + [Theory] + [InlineData("stream")] + [InlineData("stream\r")] + [InlineData("stream\n")] + [InlineData("stream\r\n")] + public void A_file_that_ends_on_the_stream_keyword_ends_the_stream_s_data_there_and_says_so(string tail) + { + // A buffer that ends on the keyword, or on the carriage return of a CR LF, cannot say where the data starts, and was + // left for a larger one, which a buffer that ends the file has none of: the stream read as no data, in silence. + // Its data starts where the file ends, holding nothing, as with the line feed after the keyword (#174). + var text = "%PDF-1.7\n1 0 obj << /Type /Catalog /Pages 2 0 R /Metadata 5 0 R >> endobj\n" + + "2 0 obj << /Type /Pages /Kids [] /Count 0 >> endobj\n5 0 obj << /Length 10 >>\n" + tail; + using var document = PdfDocument.Open(Encoding.ASCII.GetBytes(text)); + + document.GetObject(new PdfObjectId(5)).AsStream().Required().RawLength.Should().Be(0); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated).Which.Message.Should().Be( + "The stream declared 10 bytes, and no endstream follows its data before the end of the file; the 0 bytes to the end of the file are taken as its data."); + document.Reader.TryGetStreamLengthFault(5, out var fault).Should().BeTrue(); + fault.EndStream.Should().Be(EndStreamState.MissingBeforeEndOfFile); + } + + [Fact] + public void A_percent_sign_where_a_short_length_ends_is_data_and_no_comment_that_confirms_the_length() + { + // A /Length 7 bytes short ends on the "%" of the data: the rest of the line read as a comment before the endstream, + // which confirmed the length, and the 7 bytes were lost in silence. A comment counts only after white space. + using var document = PdfDocument.Open(OneStream("<< /Length 3 >>\nstream\nabc%\u00e9\u00ff\u0001xyz\nendstream")); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(10); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid) + .Which.Message.Should().Be("The stream declared 3 bytes but ended after 10."); + } + + [Theory] + [InlineData("abc\n% a note\nendstream")] + [InlineData("abc %a note\r\nendstream")] + public void A_comment_after_white_space_where_the_length_ends_confirms_it(string data) + { + // A comment is white space between the data and its endstream (ISO 32000-1, 7.2.3), once white space comes before it. + using var document = PdfDocument.Open(OneStream("<< /Length 3 >>\nstream\n" + data)); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(3); + document.Diagnostics.Should().BeEmpty(); + } + + [Fact] + public void A_percent_sign_where_a_short_length_ends_is_data_before_the_object_s_endobj_too() + { + // With no endstream, the endobj after the declared length kept "abc", the comment it skipped lost "%xyz": the endobj + // after the data ends it instead. + using var document = PdfDocument.Open(OneStream("<< /Length 3 >>\nstream\nabc%xyz")); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(7); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated) + .Which.Message.Should().EndWith("its object's endobj ends it, after 7 bytes."); + } + + [Theory] + [InlineData("\nendobj\n% a note\n")] + [InlineData("\nendobj % a note\n")] + [InlineData("\nendobj%a note\r")] + [InlineData("\r\nendobj\n%a\n % b endobj x\n\n")] + [InlineData("\nendobj\n% a note")] + public void An_object_s_endobj_ends_data_with_no_endstream_whatever_comments_follow_it(string tail) + { + // A comment between the object's endobj and the next object kept the endobj from ending the data: the declared + // length was kept, or the data took in the endobj. A comment after it is white space (ISO 32000-1, 7.2.3). + foreach (var length in new[] { 3, 20_000 }) + { + using var document = PdfDocument.Open(OneStream( + "<< /Length " + length.ToString(CultureInfo.InvariantCulture) + " >>\nstream\nhello" + tail, endObj: false)); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(5); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated) + .Which.Message.Should().EndWith("its object's endobj ends it, after 5 bytes."); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + } + } + + [Theory] + [InlineData("\nendobj\n% a note\nfoo\n")] + [InlineData("\nendobj % a note endobj\nx")] + public void An_endobj_a_token_follows_past_its_comments_ends_no_data(string tail) + { + // What follows the endobj past its comments is a token: the endobj is the data's, and the data runs to the next object. + using var document = PdfDocument.Open(OneStream("<< /Length 20000 >>\nstream\nhello" + tail, endObj: false)); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().BeGreaterThan(5); + document.Diagnostics.Should().NotContain(entry => entry.Message.EndsWith("after 5 bytes.", StringComparison.Ordinal)); + } + + /// + /// A document whose object 4 is , followed by its endobj when , and + /// then by object 6. + /// + private static byte[] OneStream(string body, bool endObj = true) + { + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + rows[4] = file.Position; + file.Raw("4 0 obj\n" + body + (endObj ? "\nendobj\n" : string.Empty)); + rows[6] = file.Object(6, "(six)"); + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + return file.Bytes; + } + + /// + /// A document of streams, objects 3 on, each declaring bytes, whose + /// headers each lie inside the dictionary of the one before, as a string, followed by a megabyte of data no endstream + /// ends, then by , objects numbered on from the last stream; and the text before that megabyte. + /// + private static (byte[] File, string Head) NestedStreams(int streams, int declared, params string[] after) + { + var nested = Invariant($"{streams + 2} 0 obj<>stream\n"); + + for (var number = streams + 1; number >= 3; number--) + { + nested = Invariant($"{number} 0 obj<>stream\n"); + } + + var head = "%PDF-1.7\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n2 0 obj\n<< /Type /Pages /Kids [] /Count 0 >>\nendobj\n" + nested; + var offsets = new List { 0, 9, head.IndexOf("2 0 obj", StringComparison.Ordinal) }; + + for (var number = 3; number <= streams + 2; number++) + { + offsets.Add(head.IndexOf(Invariant($"{number} 0 obj<<"), StringComparison.Ordinal)); + } + + var body = head + new string('x', 1024 * 1024) + "\n"; + + foreach (var text in after) + { + offsets.Add(body.Length); + body += text; + } + + var xref = new StringBuilder(Invariant($"xref\n0 {offsets.Count}\n0000000000 65535 f\r\n")); + + foreach (var offset in offsets.Skip(1)) + { + xref.Append(Invariant($"{offset:D10} 00000 n\r\n")); + } + + var file = Encoding.Latin1.GetBytes(body + xref + Invariant($"trailer\n<< /Size {offsets.Count} /Root 1 0 R >>\nstartxref\n{body.Length}\n%%EOF\n")); + + return (file, head); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void An_endstream_the_data_spells_past_which_the_searches_may_read_no_more_ends_data_that_is_the_guard_s(bool endObj) + { + // Ten streams whose data share a megabyte read as much of the file as the searches may. Then stream 13, whose + // /Length, object 14, MaxObjectLength cuts, spells an endstream in a string, a long comment after it, then an endobj + // or not: what follows the comment lies past what the searches may read, and nothing proves the endstream the + // stream's own. The data was taken for the file's, cut short unmarked, its object said to lack its endobj where the + // data spells none; it is the guard's (#224). + const int Streams = 10; + var data = "BT (a\nendstream\n%" + new string('c', 80) + (endObj ? "\nendobj\n" : "\n") + ") Tj ET"; + var (file, _) = NestedStreams( + Streams, + 100_000, + Invariant($"{Streams + 3} 0 obj\n<< /Length {Streams + 4} 0 R >>\nstream\n{data}\nendstream\nendobj\n"), + Invariant($"{Streams + 4} 0 obj\n{new string(' ', 2000)}{data.Length}\nendobj\n")); + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = 1024 } }); + + for (var number = 3; number <= Streams + 2; number++) + { + document.GetObject(new PdfObjectId(number)); + } + + var stream = document.GetObject(new PdfObjectId(Streams + 3)).AsStream().Required(); + + stream.GetRawBytes().ToArray().Should().Equal("BT (a"u8.ToArray()); + stream.Data.CutByGuard.Should().BeTrue(); + document.Reader.IsEndObjMissing(Streams + 3, out _).Should().BeFalse(); + } + + [Theory] + [InlineData("cut")] + [InlineData("left out")] + public void A_stream_left_unsearched_whose_length_a_guard_hid_is_the_guard_s(string guard) + { + // As above, stream 13's data running 20,000 bytes past the bytes its object is parsed with, its /Length cut by + // MaxObjectLength or left out of the index by MaxIndexEntries: once the searches have read as much as they may, it is + // not searched past them, and where it ends is unknown — the length that would say is the guard's. The data up to the + // next object, its endstream and endobj taken in, or to the end of the file, was the file's, reported as a length the + // reader did not search past; it is the guard's, as an endstream found and not proven is (#224). + const int Streams = 10; + var data = "BT " + new string('x', 20_000) + " ET"; + var (file, _) = NestedStreams( + Streams, + 100_000, + Invariant($"{Streams + 3} 0 obj\n<< /Length {Streams + 4} 0 R >>\nstream\n{data}\nendstream\nendobj\n"), + Invariant($"{Streams + 4} 0 obj\n{(guard == "cut" ? new string(' ', 2000) : string.Empty)}{data.Length}\nendobj\n")); + var limits = guard == "cut" + ? PdfReaderLimits.Default with { MaxObjectLength = 1024 } + : PdfReaderLimits.Default with { MaxIndexEntries = Streams + 4 }; + using var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = limits }); + + for (var number = 3; number <= Streams + 2; number++) + { + document.GetObject(new PdfObjectId(number)); + } + + var reports = document.Diagnostics.Count; + var stream = document.GetObject(new PdfObjectId(Streams + 3)).AsStream().Required(); + + stream.Data.CutByGuard.Should().BeTrue(); + document.Reader.IsEndObjMissing(Streams + 3, out _).Should().BeFalse(); + document.Diagnostics.Skip(reports).Select(entry => entry.Code).Should().Equal( + guard == "cut" ? [PdfDiagnosticCodes.LimitObject] : []); + } + + [Fact] + public void A_stream_left_unsearched_whose_length_runs_past_the_file_takes_the_data_its_object_holds() + { + // As streams whose data share a stretch exhaust what the searches may read, the last is left unsearched; its declared + // length runs past the end of the file, and the data up to it is taken, without a search, and said so. + const int Streams = 30; + const int Declared = 10_000_000; + var (file, _) = NestedStreams(Streams, Declared); + using var document = PdfDocument.Open(file); + + for (var number = 3; number <= Streams + 2; number++) + { + document.GetObject(new PdfObjectId(number)); + } + + document.Reader.TryGetStreamLengthFault(Streams + 2, out var last).Should().BeTrue(); + last.EndStream.Should().Be(EndStreamState.NotSearched); + last.Taken.Should().Be((int)(file.Length - last.DataStart)); + document.Diagnostics.Should().ContainSingle(d => d.Position == last.DataStart).Which.Message.Should().Be( + Invariant($"The stream declared {Declared} bytes, and no endstream follows its data in the bytes read with its object; the {last.Taken} bytes to the end of the file are taken as its data without a search, the document's searches for endstream having read as much of the file as they may.")); + } + + [Fact] + public void A_stream_left_unsearched_is_reported_once_however_often_it_is_parsed() + { + // The last stream is left unsearched, as above; seventy objects read after it let the cache drop it, and parsed again, + // it keeps its data and says nothing more: its fault was reported the first time. + const int Streams = 30; + var after = Enumerable.Range(Streams + 3, 70).Select(number => Invariant($"{number} 0 obj\n({number})\nendobj\n")).ToArray(); + var (file, _) = NestedStreams(Streams, 10_000_000, after); + using var document = PdfDocument.Open(file, PdfReaderOptions.Default with { ObjectCacheCapacity = 64 }); + + for (var number = 3; number <= Streams + 2; number++) + { + document.GetObject(new PdfObjectId(number)); + } + + document.Reader.TryGetStreamLengthFault(Streams + 2, out var last).Should().BeTrue(); + last.EndStream.Should().Be(EndStreamState.NotSearched); + var length = document.GetObject(new PdfObjectId(Streams + 2)).AsStream().Required().RawLength; + + for (var number = Streams + 3; number < Streams + 73; number++) + { + document.GetObject(new PdfObjectId(number)); + } + + document.GetObject(new PdfObjectId(Streams + 2)).AsStream().Required().RawLength.Should().Be(length); + document.Diagnostics.Should().ContainSingle(d => d.Position == last.DataStart); + } + + [Fact] + public void A_comment_in_the_gap_that_runs_past_the_window_is_read_on_and_its_check_kept_for_a_second_reading() + { + // The declared length ends 20 bytes short of the window's edge, and a comment after it runs past the edge and past the + // first read after it: the reader reads on through it to the endstream, and keeps what it found for the stream's next + // reading, after the cache let the stream go, which reads nothing past its window and the first read after its data. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + rows[4] = file.Position; + const string Before = "4 0 obj\n<< /Length 0000 >>\nstream\n"; + var length = PdfFileReader.InitialObjectWindow - 20 - Before.Length; + file.Raw(Before.Replace("0000", Invariant($"{length:D4}"), StringComparison.Ordinal) + new string('a', length) + + "\n% " + new string('c', 200) + "\nendstream\nendobj\n"); + rows[6] = file.Object(6, "(six)"); + + for (var number = 10; number < 80; number++) + { + rows[number] = file.Object(number, Invariant($"({number})")); + } + + file.Table(80, rows, "/Size 80 /Root 1 0 R"); + using var source = new StrictCountingSource(file.Bytes); + using var document = PdfDocument.Open(source, PdfReaderOptions.Default with { ObjectCacheCapacity = 64 }, ownsSource: false); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(length); + + for (var number = 10; number < 80; number++) + { + document.GetObject(new PdfObjectId(number)); + } + + var before = source.BytesRead; + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(length); + (source.BytesRead - before).Should().BeLessThanOrEqualTo(PdfFileReader.InitialObjectWindow + PdfObjectParser.EndObjLookahead); + document.Diagnostics.Should().BeEmpty(); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + } + + [Fact] + public void An_endstreamx_read_on_past_the_window_confirms_the_length_and_is_no_endobj() + { + // White space after the declared length runs past the window's edge and the first read after it, to "endstreamx": + // read as the parser reads what follows a stream, the length is confirmed, and its object lacks its endobj. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + rows[4] = file.Position; + const string Before = "4 0 obj\n<< /Length 0000 >>\nstream\n"; + var length = PdfFileReader.InitialObjectWindow - 20 - Before.Length; + file.Raw(Before.Replace("0000", Invariant($"{length:D4}"), StringComparison.Ordinal) + new string('a', length) + + "\n" + new string(' ', 100) + "endstreamx\nendobj\n"); + rows[6] = file.Object(6, "(six)"); + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(length); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + } + + [Theory] + [InlineData("", false)] + [InlineData("endstream\nendobj", true)] + public void White_space_after_a_length_past_the_window_is_read_on_to_the_end_of_the_file(string end, bool ended) + { + // A file with no index, whose last object is a stream of 9,000 bytes followed by white space past the first read after + // it — to the end of the file, or to an endstream and an endobj that end it. + const int Length = 9000; + var text = "%PDF-1.7\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n2 0 obj\n<< /Type /Pages /Kids [] /Count 0 >>\nendobj\n" + + Invariant($"4 0 obj\n<< /Length {Length} >>\nstream\n") + new string('a', Length) + new string(' ', 200) + end; + using var document = PdfDocument.Open(Encoding.ASCII.GetBytes(text)); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(Length); + + if (ended) + { + document.Diagnostics.Should().NotContain(entry => entry.Code.StartsWith("stream.", StringComparison.Ordinal)); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + } + else + { + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which.Message.Should().Be( + "The stream declared 9000 bytes, and no endstream follows them before the end of the file; the declared length is kept."); + } + } + + [Fact] + public void A_stream_read_once_the_searches_read_as_much_as_they_may_is_not_read_on_past_its_window() + { + // Once streams whose data share a stretch have read as much of the file as the searches may, white space after a + // declared length that runs past the window is not read on: a length the first read after the window cannot confirm + // is kept without a search, and one whose endstream that read ends after leaves what follows the keyword unseen. + const int Streams = 30; + var unread = Gapped(Streams + 3, new string(' ', 200) + "endstream\nendobj\n", out var length); + var unseen = Gapped(Streams + 4, new string(' ', 50) + "endstream endobj\n", out _); + var (file, _) = NestedStreams(Streams, 100_000, unread, unseen); + using var document = PdfDocument.Open(file); + + for (var number = 3; number <= Streams + 2; number++) + { + document.GetObject(new PdfObjectId(number)); + } + + document.GetObject(new PdfObjectId(Streams + 3)).AsStream().Required().RawLength.Should().Be(length); + document.Reader.TryGetStreamLengthFault(Streams + 3, out var fault).Should().BeTrue(); + fault.EndStream.Should().Be(EndStreamState.NotSearched); + document.GetObject(new PdfObjectId(Streams + 4)).AsStream().Required().RawLength.Should().Be(length); + document.Reader.TryGetStreamLengthFault(Streams + 4, out _).Should().BeFalse(); + + // An object whose declared length ends 20 bytes short of its window's edge, followed by what follows it. + static string Gapped(int number, string after, out int length) + { + var head = Invariant($"{number} 0 obj\n<< /Length 0000 >>\nstream\n"); + length = PdfFileReader.InitialObjectWindow - 20 - head.Length; + return head.Replace("0000", Invariant($"{length:D4}"), StringComparison.Ordinal) + new string('a', length) + after; + } + } } diff --git a/tests/AdCodicem.Pdf.Tests/SyntaxFaultTests.cs b/tests/AdCodicem.Pdf.Tests/SyntaxFaultTests.cs index 16e98f02..e831cb43 100644 --- a/tests/AdCodicem.Pdf.Tests/SyntaxFaultTests.cs +++ b/tests/AdCodicem.Pdf.Tests/SyntaxFaultTests.cs @@ -144,6 +144,418 @@ public void A_container_an_endobj_ends_takes_nothing_of_the_objects_after_it(str new PdfValidator().Validate(document).Findings.Should().BeEmpty(); } + [Theory] + [InlineData("[1 2 3", "an array, which was never closed")] + [InlineData("<< /A 1", "a dictionary, which was never closed")] + [InlineData("<< /A", "a dictionary, which was never closed, before the value of its last key")] + public void An_object_that_lost_its_closing_delimiter_and_its_endobj_ends_at_the_next_object_s_header(string value, string what) + { + // Object 4 lost both its closing ] or >> and its endobj: its container took in object 5's header and value up to + // object 5's endobj, and the object read as ending with one, its missing endobj unreported (#230). The next object's + // header ends it, the cut reported once, and the object lacks its endobj. + var file = FourLostItsEnd(value); + using var document = PdfDocument.Open(file); + + var read = document.GetObject(new PdfObjectId(4)); + + (read is PdfArray { Count: 3 } || read is PdfDictionary { Count: <= 1 }).Should().BeTrue(); + (read is not PdfDictionary dictionary || dictionary.ContainsKey(PdfName.Get("A")) == !value.EndsWith("/A", StringComparison.Ordinal)).Should().BeTrue(); + document.GetObject(new PdfObjectId(5)).AsDictionary().Required().Should().HaveCount(2); + var report = document.Diagnostics.Should().ContainSingle().Subject; + report.Code.Should().Be(PdfDiagnosticCodes.SyntaxTruncatedObject); + report.Position.Should().Be(PdfTemplate.OffsetOf(file, "4 0 obj\n") + "4 0 obj\n".Length); + report.Message.Should().Be($"An object header ended the object inside {what}."); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + new PdfValidator().Validate(document).Findings.Should().ContainSingle() + .Which.RuleId.Should().Be(PdfValidationRuleIds.ObjectEndObjMissing); + } + + [Fact] + public void The_next_object_s_header_ends_every_container_left_open_and_is_reported_at_the_innermost() + { + // The innermost dictionary took object 5's /Foo and /Author and closed at its >>: it ends at the header, which is + // reported where it opens, and the containers around it end there too (#230). + var file = FourLostItsEnd("<< /A [1 << /B 2"); + using var document = PdfDocument.Open(file); + + var outer = document.GetObject(new PdfObjectId(4)).AsDictionary().Required(); + var array = outer.GetArray(PdfName.Get("A")).Required(); + array.Should().HaveCount(2); + array[1].Should().BeOfType().Which.Should().HaveCount(1, "the inner dictionary takes nothing of object 5"); + var report = document.Diagnostics.Should().ContainSingle().Subject; + report.Position.Should().Be(PdfTemplate.OffsetOf(file, "<< /B 2")); + report.Message.Should().Be("An object header ended the object inside a dictionary, which was never closed; the 2 arrays or dictionaries around it were never closed either."); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + } + + [Theory] + [InlineData("[1 (x 6 0 obj y) 2", 3, 0)] + [InlineData("[1 6 0 R", 2, 0)] + [InlineData("[1 6 65536 obj", 4, 1)] + [InlineData("[1 2147483648 0 obj", 4, 1)] + [InlineData("[1 6 %a comment\n0 obj", 4, 1)] + [InlineData("[1 +6 0 obj", 4, 1)] + public void Text_that_is_no_object_header_ends_no_container(string value, int count, int strays) + { + // A header in a string is no token; a reference is decided first; a generation or a number out of range, a comment + // between the tokens, a sign before the number: none is a header, as the search for a stream's endstream takes + // one, and the array goes on, a stray obj reported, to object 5's header (#230). + var file = FourLostItsEnd(value); + using var document = PdfDocument.Open(file); + + document.GetObject(new PdfObjectId(4)).AsArray().Required().Should().HaveCount(count); + document.Diagnostics.Count(entry => entry.Code == PdfDiagnosticCodes.SyntaxUnexpectedToken).Should().Be(strays); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.SyntaxTruncatedObject) + .Which.Message.Should().Be("An object header ended the object inside an array, which was never closed."); + } + + [Fact] + public void A_header_inside_a_closed_container_ends_it_there_and_its_endobj_is_judged_missing() + { + // The rule's cost: text an object holds that is a header ends the object there, though its container closes and + // its endobj follows; what follows the header is not read (#230). No document of the corpus holds one. + var file = PdfTemplate.Build(PdfTemplate.Sound + .Replace("<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Catalog /Pages 2 0 R /Test 4 0 R >>", StringComparison.Ordinal) + .Replace("xref\n0 4\n", "4 0 obj\n[1 2 6 0 obj 3]\nendobj\nxref\n0 5\n", StringComparison.Ordinal) + .Replace("{row:3}\n", "{row:3}\n{row:4}\n", StringComparison.Ordinal) + .Replace("/Size 4", "/Size 5", StringComparison.Ordinal)); + using var document = PdfDocument.Open(file); + + document.GetObject(new PdfObjectId(4)).AsArray().Required().Should().HaveCount(2); + document.Diagnostics.Should().ContainSingle().Which.Message.Should().Be("An object header ended the object inside an array, which was never closed."); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + } + + [Fact] + public void An_object_stream_member_holding_a_header_reads_it_as_tokens() + { + // A member has no header, and is followed by none: text in it that reads as one is a stray obj, as it was (#230). + var file = Packed("[1 2 5 0 obj 3]", last: true); + using var document = PdfDocument.Open(file); + + document.GetObject(new PdfObjectId(4)).AsArray().Required().Should().HaveCount(6); + document.Diagnostics.Should().ContainSingle().Which.Code.Should().Be(PdfDiagnosticCodes.SyntaxUnexpectedToken); + } + + [Fact] + public void Nesting_deeper_than_the_reader_follows_ends_at_the_next_object_s_header_too() + { + // Past the depth the parser follows, the containers are skipped to their end: an endobj, and now the next object's + // header, ends them (#230). + var file = FourLostItsEnd(new string('[', 200)); + using var document = PdfDocument.Open(file); + + document.GetObject(new PdfObjectId(4)).Should().BeOfType(); + document.GetObject(new PdfObjectId(5)).AsDictionary().Required().Should().HaveCount(2); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(PdfDiagnosticCodes.SyntaxDepthExceeded, PdfDiagnosticCodes.SyntaxTruncatedObject); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + } + + [Fact] + public void A_trailer_that_lost_its_closing_delimiter_takes_nothing_of_the_update_after_it() + { + // An older trailer that lost its >>: it took the update's object 4, /Type and /Info, into the trailer the reader + // merged, and /Info named an object the file lacks. The update's header ends it (#230). + var file = TestPdfBuilder.AppendIncrementalUpdate( + new TestPdfBuilder().WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>").WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>").BuildClassic(rootNumber: 1), + rootNumber: 1, + [(4, "<< /Type /Foo /Info 9 0 R >>")]); + var text = Encoding.Latin1.GetString(file); + var trailer = text.IndexOf("/Root 1 0 R >>", StringComparison.Ordinal) + "/Root 1 0 R".Length; + Encoding.ASCII.GetBytes(" ").CopyTo(file, trailer); + + using var document = PdfDocument.Open(file); + + document.Trailer.ContainsKey(PdfName.Info).Should().BeFalse(); + document.Trailer.ContainsKey(PdfName.Type).Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectReferenceMissing).Should().BeFalse(); + + // A trailer is no object: the report said the header ended "the object" at the trailer's <<. + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.SyntaxTruncatedObject) + .Which.Message.Should().Be("An object header came inside a dictionary, which was never closed."); + } + + [Theory] + [InlineData(" /A 1\n\n6 0 obj")] + [InlineData(" /A 1\n\n6 0 ")] + [InlineData(" /A 1 /A\n\n6 0 obj")] + public void A_header_the_guard_s_cut_leaves_undecided_is_the_guard_s(string tail) + { + // Object 4 lost its >> and its endobj, and the next object's header comes where MaxObjectLength cuts it: the header's + // number and generation were kept as keys that are not names, or as the value of the key before them, where a larger + // limit reads the header (#230). + const int Limit = 128; + var head = "4 0 obj\n<< /Pad ("; + var body = "<< /Pad (" + new string('x', Limit - head.Length - (")" + tail).Length) + ")" + tail + "\n<< /Foo 7 >>"; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, body), + }; + file.Table(5, rows, "/Size 5 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = Limit } }); + + document.GetObject(new PdfObjectId(4)); + + document.Diagnostics.Should().NotContain(entry => entry.Code == PdfDiagnosticCodes.SyntaxUnexpectedToken); + document.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.LimitObject); + } + + [Fact] + public void A_header_the_guard_s_cut_leaves_undecided_gives_the_key_before_it_no_value() + { + // "6" after the second /A may be the value it is given, or the next object's header, which a larger limit reads, the + // key then left out: the guard's cut decides neither, the key keeps the value the file gave first, and the repeat + // is not said to keep a last one (#230). + var document = CutAt(" /A 1 /A\n\n6 0 obj", 128); + + document.GetObject(new PdfObjectId(4)).AsDictionary().Required().GetInteger(PdfName.Get("A")).Should().Be(1); + document.Diagnostics.Should().NotContain(entry => entry.Code == PdfDiagnosticCodes.SyntaxKeyRepeated); + document.Dispose(); + } + + [Theory] + [InlineData(" /A 1\n\n0 0 obj", 2)] + [InlineData(" /A 1 /B 6 70000 obj", 1)] + [InlineData(" /A 1\n\n+6 0 obj", 2)] + [InlineData(" /A 1\n\n6 %c\n0 obj", 2)] + public void Tokens_no_header_begins_with_are_reported_at_the_guard_s_cut_as_past_it(string tail, int keys) + { + // An object number 0, or a generation past 65,535, rules a header out whatever the cut hides: the keys that are not + // names are the file's, reported as under a larger limit (#175, #230). + using var guarded = CutAt(tail, 128); + using var whole = CutAt(tail, 100_000); + guarded.GetObject(new PdfObjectId(4)); + whole.GetObject(new PdfObjectId(4)); + + var reported = guarded.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.SyntaxUnexpectedToken).Select(entry => entry.Position).ToArray(); + reported.Should().HaveCount(keys); + whole.Diagnostics.Where(entry => entry.Code == PdfDiagnosticCodes.SyntaxUnexpectedToken).Select(entry => entry.Position).Should().Contain(reported); + } + + [Theory] + [InlineData(" /A 1 /B +6 0 obj")] + [InlineData(" /A 1 /B 6 %c\n0 obj")] + public void A_value_no_header_can_begin_with_keeps_its_place_at_the_guard_s_cut(string tail) + { + // A sign, or a comment between the tokens, rules a header out whatever the cut hides: "+6" is the value of /B, as a + // larger limit reads it, not the start of a header the edge left undecided (#230). + using var document = CutAt(tail, 128); + + document.GetObject(new PdfObjectId(4)).AsDictionary().Required().GetInteger(PdfName.Get("B")).Should().Be(6); + } + + [Theory] + [InlineData("<< /Pad ({0}) /A 1 /B 6 0 objx >>", "obj")] + [InlineData("<< /Pad ({0}) /A 1 /B 6 0 objx >>", "ob")] + [InlineData("<< /Pad ({0}) /A 1 /B 6 0 objx >>", "o")] + [InlineData("<< /Pad ({0}) /A 1 /B [3 6 0 objx] >>", "obj")] + [InlineData("<< /Pad ({0}) /A 1 /B << /D 6 0 objx >> >>", "ob")] + public void An_object_a_guard_cuts_at_a_header_it_leaves_undecided_is_not_said_to_lack_its_endobj(string body, string kept) + { + // Object 4 is whole, its endobj after it; MaxObjectLength ends the window inside "objx", which may be the next object's + // header: the guard's cut, which says nothing of what follows the object (#230). + var text = string.Format(CultureInfo.InvariantCulture, body, new string('x', 40)); + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R /Extra 4 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, text), + }; + file.Table(5, rows, "/Size 5 /Root 1 0 R"); + var limit = "4 0 obj\n".Length + text.IndexOf("objx", StringComparison.Ordinal) + kept.Length; + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit } }); + + document.GetObject(new PdfObjectId(4)); + + document.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.LimitObject); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectEndObjMissing).Should().BeFalse(); + } + + [Theory] + [InlineData("6 0 ")] + [InlineData("6 0")] + [InlineData("6 0 R")] + [InlineData("6")] + public void An_object_a_guard_cuts_inside_a_reference_it_holds_alone_is_not_said_to_lack_its_endobj(string kept) + { + // Object 4 holds the reference 6 0 R alone, after 100 spaces, its endobj after it; MaxObjectLength ends the window + // inside it, after its generation, whose look-ahead the edge left undecided: the generation was taken for what follows + // the object, which was said to lack its endobj, at the guard's cut (#224). + var body = new string(' ', 100) + "6 0 R"; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R /Extra 4 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, body), + [6] = file.Object(6, "(six)"), + }; + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + var limit = "4 0 obj\n".Length + 100 + kept.Length; + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit } }); + + document.GetObject(new PdfObjectId(4)); + + document.Diagnostics.Should().Contain(entry => entry.Code == PdfDiagnosticCodes.LimitObject); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + new PdfValidator().Validate(document).Contains(PdfValidationRuleIds.ObjectEndObjMissing).Should().BeFalse(); + } + + [Theory] + [InlineData("(abc)")] + [InlineData("<< /A 1 >>")] + [InlineData("[1 2 3]")] + public void An_object_exactly_as_long_as_MaxObjectLength_reads_whole(string body) + { + // The window MaxObjectLength allows ends right after the object's endobj: the file says no regular character goes on + // past it, and the object is whole, as under a larger limit (#174). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R /Extra 4 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, body), + }; + file.Table(5, rows, "/Size 5 /Root 1 0 R"); + var limit = ("4 0 obj\n" + body + "\nendobj").Length; + using var document = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit } }); + + document.GetObject(new PdfObjectId(4)); + + // The catalog, longer than the bound, is the guard's; object 4 is not. + document.Diagnostics.Should().NotContain(entry => entry.Position == rows[4]); + document.Reader.IsCutAtLimit(4).Should().BeFalse(); + document.Reader.IsEndObjMissing(4, out _).Should().BeFalse(); + } + + [Theory] + [InlineData("Prev", 9)] + [InlineData("XRefStm", 6)] + [InlineData("W", 5)] + [InlineData("W unclosed", 5)] + [InlineData("Prev empty", 9)] + public void An_object_the_chain_reads_exactly_as_long_as_MaxObjectLength_reads_whole(string key, int number) + { + // The chain reads object N — a /Prev, an /XRefStm, a cross-reference stream's /W — through a window MaxObjectLength + // ends right after its endobj: the file says no regular character goes on past it, and the object is whole, as a + // load reads it, whether the endobj follows its value, ends an array it left open, or is all it holds. It ended the + // chain for the guard, which no load then reached (#174, #263). + var file = key switch + { + "Prev" => ChainFiles.UpdateNamingItsPrevious("9 0 R", padding: 200), + "XRefStm" => ChainFiles.HybridNamingItsStream("6 0 R", padding: 200), + "W" => ChainFiles.UnderAnUpdate("W", placed: true, padding: 200), + "W unclosed" => ChainFiles.UnderAnUpdate( + "/Type /XRef /Size 8 /W 5 0 R /Length {length}", "[1 4 2" + new string(' ', 200), compressed: false, predicted: false, placed: true), + _ => Blanked(ChainFiles.UpdateNamingItsPrevious("9 0 R", padding: 200)), + }; + var text = Encoding.Latin1.GetString(file); + var header = text.IndexOf(string.Create(CultureInfo.InvariantCulture, $"\n{number} 0 obj\n"), StringComparison.Ordinal) + 1; + var limit = text.IndexOf("endobj", header, StringComparison.Ordinal) + "endobj".Length - header; + using var whole = PdfDocument.Open(file); + + using (var document = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit } })) + { + document.GetObject(new PdfObjectId(3)).AsDictionary().Should().NotBeNull(); + document.Reader.Structure.Sections.Should().OnlyContain(section => !section.CutByLimit && section.ChainHiddenByGuard == null); + document.Diagnostics.Select(entry => entry.Code).Should().Equal(whole.Diagnostics.Select(entry => entry.Code)); + } + + using (var cut = PdfDocument.Open(file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit - 1 } })) + { + cut.Diagnostics.Select(entry => entry.Code).Should().Contain(PdfDiagnosticCodes.LimitObject); + } + + var throwing = () => PdfDocument.Open( + file, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit }, ThrowOnLimit = true }).Dispose(); + throwing.Should().NotThrow(); + + // Object 9's value, the first table's offset, given to spaces: the object holds nothing but its endobj. + static byte[] Blanked(byte[] file) + { + var text = Encoding.Latin1.GetString(file); + var value = text.IndexOf("\nendobj", text.IndexOf("\n9 0 obj\n", StringComparison.Ordinal), StringComparison.Ordinal); + var start = text.LastIndexOf(' ', value) + 1; + return Encoding.Latin1.GetBytes(text[..start] + new string(' ', value - start) + text[value..]); + } + } + + [Theory] + [InlineData("[1 2 3")] + [InlineData("<< /A 1 /B 2")] + [InlineData("<< /A 1 /B")] + [InlineData("<< /A [1 2")] + [InlineData("")] + public void An_object_whose_endobj_ends_its_containers_at_MaxObjectLength_reads_as_under_a_larger_bound(string body) + { + // Object 4 left an array or a dictionary open, or holds nothing, and MaxObjectLength's window ends right after its + // endobj: the file says no regular character follows that endobj, which ends the object as it does under a larger + // bound — the containers it left open reported as the file's —, where it was taken for the guard's cut (#174). Its + // value comes after 100 spaces, so that no other object is as long. + var padded = new string(' ', 100) + body; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R /Extra 4 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, padded), + }; + file.Table(5, rows, "/Size 5 /Root 1 0 R"); + var limit = ("4 0 obj\n" + padded + "\nendobj").Length; + + using var exact = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit } }); + using var larger = PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit + 1 } }); + + exact.GetObject(new PdfObjectId(4)).ToString().Should().Be(larger.GetObject(new PdfObjectId(4)).ToString()); + exact.Reader.IsCutAtLimit(4).Should().BeFalse(); + Of4(exact).Should().Equal(Of4(larger)); + exact.Reader.IsEndObjMissing(4, out _).Should().Be(larger.Reader.IsEndObjMissing(4, out _)); + + var throwing = () => PdfDocument.Open( + file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit }, ThrowOnLimit = true }).GetObject(new PdfObjectId(4)); + throwing.Should().NotThrow(); + + // What is reported of object 4, which starts at rows[4]. + IEnumerable<(string, long)> Of4(PdfDocument document) => + document.Diagnostics.Where(entry => entry.Position >= rows[4]).Select(entry => (entry.Code, entry.Position)); + } + + /// + /// Opens a document whose object 4 lost its >> and its endobj, its padding setting 's + /// end where a MaxObjectLength of 128 cuts it, under . + /// + private static PdfDocument CutAt(string tail, int limit) + { + const int Edge = 128; + var head = "4 0 obj\n<< /Pad ("; + var body = "<< /Pad (" + new string('x', Edge - head.Length - (")" + tail).Length) + ")" + tail + "\n<< /Foo 7 >>"; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, body), + }; + file.Table(5, rows, "/Size 5 /Root 1 0 R"); + return PdfDocument.Open(file.Bytes, new PdfReaderOptions { Limits = PdfReaderLimits.Default with { MaxObjectLength = limit } }); + } + + /// + /// A sound document whose catalog names objects 4 and 5, object 4 holding and no endobj, then + /// object 5 holding and its endobj. + /// + private static byte[] FourLostItsEnd(string value, string five = "<< /Foo 7 /Author (a) >>") => + PdfTemplate.Build(PdfTemplate.Sound + .Replace("<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Catalog /Pages 2 0 R /Test 4 0 R /Other 5 0 R >>", StringComparison.Ordinal) + .Replace("xref\n0 4\n", $"4 0 obj\n{value}\n5 0 obj\n{five}\nendobj\nxref\n0 6\n", StringComparison.Ordinal) + .Replace("{row:3}\n", "{row:3}\n{row:4}\n{row:5}\n", StringComparison.Ordinal) + .Replace("/Size 4", "/Size 6", StringComparison.Ordinal)); + [Theory] [InlineData("/ID [<41zz42> <4142>]", PdfDiagnosticCodes.SyntaxHexStringInvalid)] [InlineData("/Info#zz 3 0 R", PdfDiagnosticCodes.SyntaxNameEscapeInvalid)] diff --git a/tests/AdCodicem.Pdf.Tests/WindowEdgeTests.cs b/tests/AdCodicem.Pdf.Tests/WindowEdgeTests.cs index 69f2b3b7..f93209ac 100644 --- a/tests/AdCodicem.Pdf.Tests/WindowEdgeTests.cs +++ b/tests/AdCodicem.Pdf.Tests/WindowEdgeTests.cs @@ -4,6 +4,7 @@ using AdCodicem.Pdf.Documents; using AdCodicem.Pdf.IO; using AdCodicem.Pdf.Objects; +using AdCodicem.Pdf.Validation; namespace AdCodicem.Pdf.Tests; @@ -110,7 +111,7 @@ public void A_stream_ending_just_before_the_window_edge_is_confirmed_without_a_l { // T21's shape: the data ends 0 to 10 bytes short of the window's edge — the hospital-bed guidance's // objects 604 and 2053 end 2 bytes short and right on it, before a CR LF — and its "endstream" lies - // across or past the edge. The reader asks the file for the 13 bytes after the data instead of + // across or past the edge. The reader asks the file for a few bytes after the data instead of // parsing the object again through a window eight times larger, which the file, padded past that // size, would otherwise serve. var failures = new List(); @@ -132,7 +133,7 @@ public void A_stream_ending_just_before_the_window_edge_is_confirmed_without_a_l var read = source.BytesRead - beforeLoading; var length = stream.GetRawBytes().Length; - if (read > Window + 13 || length != Window - shortOfEdge - DataStart(Window - shortOfEdge) || + if (read > Window + PdfObjectParser.EndObjLookahead || length != Window - shortOfEdge - DataStart(Window - shortOfEdge) || document.Diagnostics.Count > 0) { failures.Add($"{shortOfEdge} bytes short: read {read} bytes, data of {length}, {document.Diagnostics.Count} diagnostics"); @@ -142,6 +143,198 @@ public void A_stream_ending_just_before_the_window_edge_is_confirmed_without_a_l failures.Should().BeEmpty($"{Describe(endOfLine)} must not cost a larger window"); } + [Theory] + [InlineData("\n", true)] + [InlineData("\r\n", true)] + [InlineData("\r", true)] + [InlineData("\n", false)] + [InlineData("\r\n", false)] + public void A_stream_reads_the_same_when_a_guard_s_window_ends_on_its_keyword(string endOfLine, bool ended) + { + // MaxObjectLength ends the window right after "stream", or after its carriage return: the dictionary is whole, and + // the end-of-line, as the data, counts toward no window. The data starts past it, as under a window one byte + // larger — never on it, a byte early, nor at the declared length past the cut with no search (#174). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object( + 4, + $"<< /Pad ({new string('x', 60)}) /Length {(ended ? 40 : 5000)} >>\nstream{endOfLine}" + new string('a', 40) + + (ended ? "\nendstream" : string.Empty)), + [6] = file.Object(6, "(six)"), + }; + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + var bytes = file.Bytes; + var header = PdfTemplate.OffsetOf(bytes, "4 0 obj"); + var keywordEnd = PdfTemplate.OffsetOf(bytes, "stream" + endOfLine) + "stream".Length; + var window = (int)(keywordEnd - header) + (endOfLine == "\r\n" ? 1 : 0); + + using var whole = PdfDocument.Open(bytes); + using var guarded = PdfDocument.Open( + bytes, + PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxObjectLength = window } }); + + var expected = whole.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray(); + guarded.GetObject(new PdfObjectId(4)).AsStream().Required().GetRawBytes().ToArray().Should().Equal(expected); + expected.Should().Equal(Encoding.ASCII.GetBytes(new string('a', 40))); + guarded.Diagnostics.Select(entry => entry.Message).Should().Equal(whole.Diagnostics.Select(entry => entry.Message)); + } + + [Theory] + [InlineData("X\nhello\nendstream")] + [InlineData("endstream")] + public void A_keyword_longer_than_stream_reads_the_same_wherever_the_window_edge_cuts_it(string tail) + { + // "streamX" is no stream keyword: the dictionary before it stands alone, its object lacking its endobj. A window that + // ends right after "stream" cannot say so, and the file is not asked where the data starts past a keyword a regular + // byte may go on: a larger window says (#174). + var failures = new List(); + + for (var shift = -2; shift <= 2; shift++) + { + var head = "4 0 obj\n<< /Length 5 /Pad () >>\nstream"; + var pad = Window + shift - head.Length; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, $"<< /Length 5 /Pad ({new string('x', pad)}) >>\nstream" + tail), + [6] = file.Object(6, "(six)"), + }; + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes); + + var value = document.GetObject(new PdfObjectId(4)); + + if (value is not PdfDictionary || !document.Reader.IsEndObjMissing(4, out _)) + { + failures.Add($"shift {shift}: {value.GetType().Name}, endobj missing {document.Reader.IsEndObjMissing(4, out _)}"); + } + } + + failures.Should().BeEmpty(); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void An_endobj_a_regular_byte_goes_on_is_missing_wherever_the_window_edge_cuts_it(bool stream) + { + // "endobjx" is no endobj: the object lacks it. One the window's edge reaches may be the start of a longer keyword, + // and is not taken whole: a stream's, as the file read on after its endstream says, and any other value's, as a larger + // window says (#174). + var failures = new List(); + + for (var shift = -2; shift <= 2; shift++) + { + // The data's length that ends the keyword "4 0 obj\n" and the body leave at the edge, shifted. + var body = Enumerable.Range(0, Window) + .Select(data => stream + ? string.Create(CultureInfo.InvariantCulture, $"<< /Length {data} >>\nstream\n") + new string('a', data) + "\nendstream\nendobj" + : "(" + new string('a', data) + ")\nendobj") + .First(text => "4 0 obj\n".Length + text.Length == Window + shift); + + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [4] = file.Object(4, body + "x"), + [6] = file.Object(6, "(six)"), + }; + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes); + + document.GetObject(new PdfObjectId(4)); + + if (!document.Reader.IsEndObjMissing(4, out _)) + { + failures.Add(string.Create(CultureInfo.InvariantCulture, $"shift {shift}: endobj taken whole")); + } + } + + failures.Should().BeEmpty(); + } + + [Fact] + public void An_endobj_the_window_s_edge_reaches_ends_the_object_where_a_source_cut_short_after_opening_gives_no_byte_more() + { + // Object 6's endobj ends right at the edge of the 8 KB window it is read through, and an "x" follows it: "endobjx" is + // no endobj, and the object lacks it. Once the document is open, the file is cut short right where the endobj ends, + // as another process truncates it: the source gives no byte past the edge, and the file says the token ends there, + // as where the file ends. The object reads whole, and nothing is reported (#174). + const string Head = "6 0 obj\n("; + const string Tail = ")\nendobj"; + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + [6] = file.Position, + }; + file.Raw(Head + new string('a', Window - Head.Length - Tail.Length) + Tail + "x\n"); + rows[7] = file.Object(7, "(seven)"); + file.Table(8, rows, "/Size 8 /Root 1 0 R"); + var bytes = file.Bytes; + + using var whole = PdfDocument.Open(bytes); + using var source = new CutShortSource(bytes); + using var document = PdfDocument.Open(source, options: null, ownsSource: false); + source.CutAt((int)rows[6] + Window); + + whole.GetObject(new PdfObjectId(6)); + whole.Reader.IsEndObjMissing(6, out _).Should().BeTrue(); + document.GetObject(new PdfObjectId(6)).Should().BeOfType().Which.Length.Should().Be(Window - Head.Length - Tail.Length); + document.Reader.IsEndObjMissing(6, out _).Should().BeFalse(); + document.Diagnostics.Should().BeEmpty(); + } + + [Fact] + public void The_file_answers_for_a_window_s_edge_outside_it_without_asking_its_source() + { + // The parser asks the file what follows a token, or a stream keyword, its window's edge reaches, which lies inside the + // file (#174, #224). Asked of an offset outside it, before its start or at its end, the file answers without asking + // a source that refuses any read outside its data: no byte there goes on a token, nor is an end-of-line there skipped + // before a stream's data, which starts where the keyword ends. + var bytes = new TestPdfBuilder() + .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") + .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") + .BuildClassic(rootNumber: 1); + using var document = PdfDocument.Open(new StrictCountingSource(bytes), options: null, ownsSource: true); + var provider = (IPdfStreamDataProvider)document.Reader; + + provider.TokenEndsAt(-1).Should().BeTrue(); + provider.TokenEndsAt(bytes.Length).Should().BeTrue(); + provider.DataStart(-1).Should().Be(-1); + provider.DataStart(bytes.Length).Should().Be(bytes.Length); + } + + [Fact] + public void A_cross_reference_stream_reads_whole_when_its_guard_s_window_ends_on_its_keyword() + { + // MaxTrailerLength ends the window right after the stream's keyword: its dictionary is whole, and its rows are read + // past it, as under a window one byte larger — not a byte early, misaligned, which rebuilt the index and judged the + // catalog missing (#174). + var bytes = new TestPdfBuilder() + .WithObject(1, "<< /Type /Catalog /Pages 2 0 R >>") + .WithObject(2, "<< /Type /Pages /Kids [] /Count 0 >>") + .BuildWithXRefStream(rootNumber: 1); + var dictionary = PdfTemplate.OffsetOf(bytes, "/Type /XRef"); + var header = Encoding.Latin1.GetString(bytes).LastIndexOf(" 0 obj", (int)dictionary, StringComparison.Ordinal) - 1; + var keywordEnd = Encoding.Latin1.GetString(bytes).IndexOf("stream", (int)dictionary, StringComparison.Ordinal) + "stream".Length; + + using var document = PdfDocument.Open( + bytes, + PdfReaderOptions.Default with { Limits = PdfReaderLimits.Default with { MaxTrailerLength = keywordEnd - header } }); + + document.Diagnostics.Should().BeEmpty(); + document.WasRepaired.Should().BeFalse(); + new PdfValidator().Validate(document).Findings.Should().BeEmpty(); + } + [Fact] public void An_anomaly_inside_an_object_longer_than_the_window_is_reported_once() { @@ -312,6 +505,136 @@ public void A_cross_reference_stream_after_more_white_space_than_the_object_wind document.Diagnostics.Should().BeEmpty(); } + [Fact] + public void An_object_that_lost_its_end_ends_at_the_next_header_wherever_the_window_edge_cuts_it() + { + // The header that ends an array which lost its ] and its endobj meets the edge at every byte, and a byte or two + // before and after: the edge never makes the header an integer and a stray keyword, nor hides it (#230). + var failures = new List(); + const string Before = "[("; + const string After = ") 1 2 3\n6 0 obj\n(six)"; + var header = Header(ObjectNumber); + + for (var shortOfEdge = -2; shortOfEdge <= 10; shortOfEdge++) + { + var filler = Window - header.Length - Before.Length - ") 1 2 3\n".Length - shortOfEdge; + using var document = PdfDocument.Open(Document(Before + new string('x', filler) + After)); + + var read = document.GetObject(new PdfObjectId(ObjectNumber)); + var reports = string.Join(" | ", document.Diagnostics.Select(entry => entry.Message)); + + if (read is not PdfArray { Count: 4 } || + reports != "An object header ended the object inside an array, which was never closed." || + !document.Reader.IsEndObjMissing(ObjectNumber, out _)) + { + failures.Add($"the header {shortOfEdge} bytes before the edge: {read}, {reports}"); + } + } + + failures.Should().BeEmpty(); + } + + [Theory] + [InlineData(-40)] + [InlineData(-20)] + [InlineData(-3)] + [InlineData(0)] + [InlineData(5)] + public void What_follows_an_endstream_the_window_holds_is_read_wherever_the_window_edge_falls(int shift) + { + // An integer follows object 4's endstream in place of its endobj, starting shift bytes past the window's edge: where + // the edge cut that integer, or the white space before it, the object's endobj was judged unseen, and its loss went + // unreported. What follows the keyword is read on in the file, as past the buffer (#174). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + var start = file.Position; + rows[4] = start; + const string Before = "4 0 obj\n<< /Length 0000 >>\nstream\n"; + const string After = "\nendstream "; + var length = Window + shift - Before.Length - After.Length; + file.Raw(Before.Replace("0000", length.ToString("D4", CultureInfo.InvariantCulture), StringComparison.Ordinal) + + new string('a', length) + After + "7777777777\n"); + rows[6] = file.Object(6, "(six)"); + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(length); + document.Reader.IsEndObjMissing(4, out _).Should().BeTrue(); + } + + [Theory] + [InlineData(-30)] + [InlineData(-21)] + [InlineData(-18)] + [InlineData(-15)] + [InlineData(-13)] + [InlineData(-11)] + [InlineData(-9)] + [InlineData(-7)] + [InlineData(-5)] + [InlineData(-3)] + [InlineData(-1)] + [InlineData(0)] + [InlineData(2)] + [InlineData(4)] + [InlineData(8)] + [InlineData(12)] + public void An_object_s_endobj_ends_data_with_no_endstream_wherever_the_reads_part_what_follows_it(int shift) + { + // Object 4 lost its endstream. Its endobj, a comment, and a header no index places, which stops the search, fall + // across the edge of the window and of the first read of the file past it: the keyword parted between two reads, the + // header started in the read before the one that shows it whole. The data ends at the endobj wherever they fall (#174). + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + rows[4] = file.Position; + const string Before = "4 0 obj\n<< /Length 3 >>\nstream\n"; + var length = Window + shift - Before.Length; + file.Raw(Before + new string('a', length) + "\nendobj\n% c\n7 0 obj\n(stray)\nendobj\n"); + rows[6] = file.Object(6, "(six)"); + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(length); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamTruncated) + .Which.Message.Should().EndWith(string.Create(CultureInfo.InvariantCulture, $"its object's endobj ends it, after {length} bytes.")); + } + + [Theory] + [InlineData(-7)] + [InlineData(-5)] + [InlineData(-3)] + [InlineData(-1)] + public void An_endobj_that_ends_a_longer_token_ends_no_data_wherever_the_reads_part_it(int shift) + { + // As above, but the keyword ends the token "aendobj", parted between the window and the first read past it: no endobj + // ends the data, and the declared length, which ends before the next object, is kept. + var file = new HandWrittenFile(); + var rows = new Dictionary + { + [1] = file.Object(1, "<< /Type /Catalog /Pages 2 0 R >>"), + [2] = file.Object(2, "<< /Type /Pages /Kids [] /Count 0 >>"), + }; + rows[4] = file.Position; + const string Before = "4 0 obj\n<< /Length 3 >>\nstream\n"; + var length = Window + shift - Before.Length; + file.Raw(Before + new string('a', length) + "endobj\n% c\n7 0 obj\n(stray)\nendobj\n"); + rows[6] = file.Object(6, "(six)"); + file.Table(7, rows, "/Size 7 /Root 1 0 R"); + using var document = PdfDocument.Open(file.Bytes); + + document.GetObject(new PdfObjectId(4)).AsStream().Required().RawLength.Should().Be(3); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid) + .Which.Message.Should().EndWith("the declared length is kept."); + } + [Fact] public void An_object_whose_header_lies_past_the_window_is_read_where_the_index_says() { @@ -376,10 +699,11 @@ public void A_stream_whose_length_the_file_cannot_hold_is_read_to_its_endstream( } [Fact] - public void A_stream_the_file_cuts_short_is_reported_without_asking_the_source_past_its_end() + public void A_stream_the_file_cuts_short_keeps_its_declared_length_without_asking_the_source_past_its_end() { // The file ends inside "endstream". The reader asks the file for the bytes after the data, and a - // source may refuse a read that runs past its end: it must be asked only for the six there are. + // source may refuse a read that runs past its end: it must be asked only for the six there are. No endstream + // nor endobj follows the declared length, which ends before the end of the file: it is kept, and reported (#174). var bytes = Encoding.ASCII.GetBytes( "%PDF-1.7\n1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj\n" + "2 0 obj << /Type /Pages /Kids [] /Count 0 >> endobj\n" + @@ -388,8 +712,9 @@ public void A_stream_the_file_cuts_short_is_reported_without_asking_the_source_p using var document = PdfDocument.Open(new StrictCountingSource(bytes), options: null, ownsSource: true); var stream = document.GetObject(new PdfObjectId(3)).AsStream().Required(); - Encoding.ASCII.GetString(stream.GetRawBytes().Span).Should().StartWith("hello"); - document.Diagnostics.Contains(PdfDiagnosticCodes.StreamTruncated).Should().BeTrue(); + Encoding.ASCII.GetString(stream.GetRawBytes().Span).Should().Be("hello"); + document.Diagnostics.Should().ContainSingle(entry => entry.Code == PdfDiagnosticCodes.StreamLengthInvalid).Which.Message.Should().Be( + "The stream declared 5 bytes, and no endstream follows them before the end of the file; the declared length is kept."); } /// diff --git a/tests/corpus/manifest.schema.json b/tests/corpus/manifest.schema.json index c12668e0..aedc28d2 100644 --- a/tests/corpus/manifest.schema.json +++ b/tests/corpus/manifest.schema.json @@ -273,6 +273,7 @@ "filter.checksum-mismatch", "filter.unsupported", "limit.decoded-stream", + "limit.stream-length", "limit.object", "limit.xref-section-length", "limit.xref-section-count", @@ -440,6 +441,12 @@ "exclusiveMinimum": 268435456, "maximum": 2147483647 }, + "maxStreamLength": { + "type": "integer", + "description": "The most of one stream's encoded data read at once, in bytes.", + "exclusiveMinimum": 268435456, + "maximum": 2147483647 + }, "maxObjectLength": { "type": "integer", "description": "The longest object, the data of a stream aside, in bytes.",