Derive item state from the ledger, and fix what the drivers found - #20
Merged
Conversation
An item is one unit of knowledge: the post, the thread parents above it, the links harvest promoted, the media, the video awaiting its transcript. Nothing about it advances to digest or wiki until every part has landed, so `status: enriched` now means the whole item landed — no unit queued, waiting, blocked, error, or manual — rather than "some markdown exists in the enrichment directory". A dead or skipped unit owes nothing and holds nothing hostage. The derivation reads the ledger; the run passes the map it already holds, and the quiet verbs read it themselves without gaining a way to die after their own write has landed. The run report then states the shape of what is missing per item — "3 of 4 units landed — 1 waiting on transcription" — so a session never infers completeness from a list of units. Digest staleness moves off file mtimes for the same reason it was undetectable on a second machine: git stamps every file at checkout, so after a clone the whole tree shares one mtime. The comparison is now the item's newest `done` ledger date against its recorded digest pass date — both committed state, day granularity, same-day enrich-then-digest not stale. (The digest file's own `date:` is the item's SHARE date per state-formats.md, so it cannot serve as the digest's date.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A video with no usable captions returned `waiting` with no body, so the description yt-dlp had just handed over was thrown away — and a video deleted or made private during a transcription backlog took it for good. The podcast driver in the same position writes its show notes first; the two kinds now behave alike. The park writes the description as its `## Description` section and the transcribe drain appends the transcript to that same file instead of recomposing from a fresh probe, so a re-drain supersedes the transcript without duplicating or losing anything around it. That requires one body shape per kind: the transcript is now always its own `## Transcript` section, description or not, because the drain splits the stored body on that heading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
§9 states three routes; only two existed. The third could not be built out of URL patterns — bare `/feed` and `/rss` suffixes and an RSS `<link rel>` are blog vocabulary, so `matches()` guessing at them would steal ordinary posts from the web driver. Registry order supplies the answer instead. Nothing claims an indie episode page, so the catch-all fetches it; a page advertising its own audio — an `og:audio` pointer or an `<audio>` element holding the file — re-detects to `podcast`, and the podcast driver resolves it: the feed its `<link rel>` names first (richer show notes), the page's own enclosure when that feed is unreachable or does not list the episode. That fallback is required, not a convenience: the driver was routed here by a re-detection, and bouncing back would park the unit as a loop. A bare link to an mp3 stays web work — a post linking one is still a post, and the extraction is worth more than the guess. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The blob round trip is about to have two callers: the github driver fetches a blob to fence it as source, and the file driver is about to re-fetch that same blob when its bytes turn out to be a document to extract. Left where it was, the second caller would have to import the first — a driver reaching into another driver, the coupling already recorded against `pipeline.transcribe` importing `drivers.youtube`. So `drivers/gh.py` now owns the whole route: the `gh` invocation, the JSON parse and its failure classification, the blob reference, the ref guess-and-resolve, the contents endpoint, the base64 decode, and the oversize park. `drivers/github.py` is a consumer of it and keeps only what a github URL's shape means and what to do with the bytes that come back. It sits beside `drivers/transport.py` for that module's reason: a seam several drivers share is not itself a driver, and like transport it reaches only for pipeline vocabulary, so the dependency arrow still runs drivers → pipeline and never back. `BlobRef` carries the ref/path tail unsplit, because which segments are the ref is not knowable from the URL, and `fetch_blob` answers with the path the winning split left — that name is what the bytes have to be sniffed under. No behaviour changes. The ref-boundary tests move to the seam they belong to, where the file driver's own blob fetches will inherit them rather than regressing against a driver-local copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A `github.com/<o>/<r>/blob/<ref>/<path>.pdf` parked manual naming its format, because no driver could re-fetch it: the blob URL serves an HTML viewer, so a file driver pointed at it over plain HTTP would find HTML and re-detect straight back — a loop park on a public repo, and `dead` on a private one, where the unauthenticated fetch 404s. The file driver now reads blob bytes through `drivers/gh.py`, the same authenticated route the github driver reads them through, which makes the round trip real. It inherits the ref guess-and-resolve with it, so a slashed branch or a `refs/heads/` permalink resolves on the way back in too — without that, re-detecting a blob would have handed the file driver the very 404-into-`dead` the boundary work exists to stop. With the reason for the refusal gone, the github driver's document blobs emit `Result.redetect` to file/<format> instead of parking manual, and the extraction runs on the committed file itself. Sniffing stays BY NAME, which is what catches the two extractable shapes with no byte signature. That includes CSV, which had been parked for capture on the argument that a fenced CSV and a fenced LFS pointer are indistinguishable. The argument was against the FENCE, and re-detection is not one: an extractor now reads the committed bytes into a real table where capture asked for a manual step, and an LFS pointer that sniffs `.csv` by name meets the file driver's own pointer guard, which parks it before any extractor is handed 130 bytes of stand-in text. One rule for every extractable document, and the pointer still never stands in for what it points at. A non-document binary still parks manual naming what it is, and a blob over the contents API's 1MB inline ceiling still parks manual, never dead. HTML arriving from the contents API is a committed HTML file, not a viewer page, so blob work never takes the file driver's re-route back to `web`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Twenty hops was doing the wrong job. A thread is ONE piece of content — one work unit, one enrichment file — and the walk is a linear chain of cheap calls with no fan-out, unlike the 12-URL harvest cap, which bounds how much a single item drags in. Truncating at 20 protected nothing; it stored three quarters of one coherent argument, and 30-post technical threads are ordinary. The bound stays, because a cycle or a self-referencing parent would otherwise walk forever — now sized as a sanity bound rather than an editorial one, with the marker and the incomplete-chain recording untouched so a thread past even 100 is still recorded honestly. Added the cycle test the bound actually exists for. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mutation-testing the reconciliation found the segment-wise ref match unguarded. `_longest_ref_match` documents that `automation/bors` must not claim a URL whose branch is `automation/bors-next`, but the test set the case up in the direction that cannot tell the two apart: with both refs pushed, a string-prefix match picks the same winner a segment-wise one does. Swapping to a string prefix passed the whole suite. The discriminating shape is the sibling the repo has NOT pushed: the URL names `automation/bors-next`, only `automation/bors` comes back, and a string prefix would split there and ask for `-next/README.md`. Segment-wise nothing matches, so the guess's 404 stands. The both-pushed case stays as a second test — it is still worth stating that the longest real ref wins. The file driver's by-name sniff was likewise only caught in passing, by a test about extractor ordering. A committed CSV re-detected out of the github driver has no signature to find, so the name that came back with the bytes is the whole routing decision; it now says so directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The web driver read the podcast driver's own helper to decide whether a fetched page was an episode page, which is the pairing gh.py's docstring forbids: drivers are dumb and isolated, and a seam two of them share is a lib beside transport.py and gh.py, not one driver importing the other. The rule is now checked over the source rather than trusted to review. paper -> web is named as the one blessed exception: that is the whole web driver composed as a fetch strategy (design §2), not a shared helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An <audio> element was the whole "this is a podcast episode" signal, so a read-aloud text-to-speech widget or an embedded media sample handed an ordinary article to the podcast driver: the widget resolved as the enclosure, the unit parked waiting: transcribe with an empty body, and the article was never extracted nor its links harvested. Live: an AWS blog post parked against an Amazon Polly mp3 while trafilatura would have yielded 11,626 chars, and en.wikipedia.org/wiki/Podcast discarded 67,542. Two markups now say "the audio is this page's subject": og:audio, the publisher naming the audio as the page's own object, and an <audio> element on a page with no substantial extractable body, where the player is all there is. Ordering carries the rule — the article's own body is tried before the player is believed. The asymmetry sets the bias: a false positive costs the whole article, a false negative costs a podcast page keeping its show notes instead of a transcript. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
digest_orphans never consulted item status, so an item held raw by a unit that is manual, error on an unchanged engine, blocked past its attempts, or waiting with no provider was reported as needing a digest on every single run — while the ingest procedure forbids digesting a raw item. The owner already carries 26 manual and 29 error units that qualify. The backstop now skips items that still owe a unit, on the ledger it already loads for the staleness dates. An unreadable ledger holds nothing back, exactly as it calls nothing stale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drain read a park's stored description back by splitting the body on "## Transcript", so a YouTube description (or show notes) containing that literal line was truncated there — and the drain wrote the truncation back, losing everything below it from disk permanently. The frontmatter answers the question the heading cannot: the transcriber stamps via, neither park does, so a park's body is notes end to end however many headings the author wrote. Once a body does hold a transcript, the split takes the last such section — the one the drain appended. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_walk kept no record of the ids it had walked, so a post naming itself (or an ancestor) as its parent spent the entire 100-hop bound on 100 back-to-back requests to a free community API for one unit — while the driver otherwise paces at 4s per unit. A seen set ends the walk where the chain repeats, recorded like any short chain rather than presented as complete. Hops now sleep 1s apart through an injected seam. The 4s politeness is spent between units and a walk is many requests inside one; a second a hop keeps an ordinary 30-post thread under a minute while making the walk a paced sequence instead of a burst. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
references/schema.md still said the pipeline "flips status to enriched", the disk-derived rule ingest-item.md was moved off. The status is derived from the ledger: enriched only once every unit the item owns has landed, raw while any is still owed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both halves of "the transcript is always its own labelled section" — the driver's captions path and the drain's composer — returned a bare transcript when there was no description, and no test noticed. A body with no heading reads back as show notes, so a re-drain stacks the stored transcript under the fresh one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four claims the branch made and nothing held: that staleness compares the NEWEST landing against the NEWEST digest pass (both were free to read the oldest), that a digest with no pass record at all is exempt rather than stale, and that queued work keeps an item raw — reachable through run(limit=N) or a cap-deferred rerun, where under the mutant an item flips enriched with work still owed. test_an_unreadable_ledger_leaves_the_derived_status_alone was vacuous: it asserted raw on a fixture that was already raw. The item is now enriched before the ledger goes bad, so the name is what the test checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The transcribe drain is the second place a unit writes its own output, and it never dropped the earlier kind's file: a web -> podcast correction that landed its transcript through `run_transcribe` left the stale `web-*.md` on disk, and the item's frontmatter listed both files for one work unit forever. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both negative cases asserted `"incomplete" not in report`, which only holds the behaviour while the section's heading happens to spell that same word: rename the heading and the two tests go quiet without a single behaviour changing. They now assert the run omits the `incomplete` key altogether, and read the report for the sentence the section states rather than a bare word. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An item whose harvest overran the URL cap read "15 of 16 units landed" on the run report: the skipped cap-fire markers were counted into both halves of the shape, though they record work the run refused rather than a unit it admitted, and no user surface reports a harvest-time cap fire at all. The count is now the admitted cohort — 11 of 12. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`http.client` ascii-encodes the request line, so an accented Wikipedia path, an IDN host or a CJK slug raised UnicodeEncodeError before a byte left the machine. Being a ValueError it walked past every caller's connection guard into the bad-seed containment, and the item parked manual — unfetched — with a raw codec message as its stated reason. The seam now encodes what it is handed the way a browser does: punycode for the host, percent-encoding for the path and query, idempotent so an already-encoded URL passes through untouched. Only a host DNS genuinely cannot carry still fails, and it fails with a stated reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`http.client` rejects `[\x00-\x20\x7f]` in the request line as well as ascii-encoding it, so an unencoded space in an href — everyday in a corpus — failed with InvalidURL AFTER the encoder had returned at its `isascii()` guard deciding there was nothing to do. The outcome was `blocked`, so the run spent five attempts and five wayback lookups on a condition no retry could change, while the same path with one accent added fetched fine: `quote` had always mapped the space to `%20`. The guard now asks whether the URL can go on the wire as it stands, not whether it happens to be ascii. Encoding stays idempotent — a URL with nothing forbidden in it still returns byte-for-byte as asked — and the userinfo guard in `_ascii_netloc`, which had the same shape, asks the same question. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both inputs of the already-encoded-passthrough test were pure ascii by the time `_ascii_url` saw them, so they returned at the guard and never reached `quote` at all: drop `%` from `_PATH_SAFE` and the whole suite stayed green while `%C3%A9` became `%25C3%25A9` — the exact double encoding the safe set exists to prevent, shipping green. The test now leads with a URL that is both already percent-encoded and non-ascii, so it reaches the encoder and the surviving `%C3%A9` is the assertion. Dropping `%` from `_PATH_SAFE` now fails it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
leeovery
changed the title
leeovery
marked this pull request as ready for review
August 23, 2026 21:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Semantics fixes across the pipeline and drivers.
drivers/gh.py(github + file) anddrivers/audio.py(podcast + youtube); drivers never import each other🤖 Generated with Claude Code