Ask the corpus who owns a unit, at every reader and every writer - #24
Open
leeovery wants to merge 18 commits into
Open
Ask the corpus who owns a unit, at every reader and every writer#24leeovery wants to merge 18 commits into
leeovery wants to merge 18 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olds Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drain writes three output shapes and the seed-time repair sweep knew one of them, so every media download and extraction asset read as "output is lost" and requeued — and a `via: extract-asset` unit's work key is a repo path, not a URL, so the fetch it was queued for could only ever raise. The unit sat in `error`, holding its item `raw` out of digest and wiki, with the asset on disk under the new name the whole time. The line already records the name its writer chose, and a rename moves the directory, never the file inside it — so the basename is read back rather than derived a second time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`is_drainable` retries an `error` entry once per newer engine, and the seed-time repair sweep runs before the drain builds its queue in the same run — so re-recording through `stamp` bumped `engine` and spent that one retry without ever running it. A renamed item's error line then waited for the NEXT release, forever. Sharpest after a sync: migration 1 stamps old lines `0.0.1`, so the run meant to rescue them was the run that silently burned their shot. `date` went the same way, and the digest staleness backstop reads it. `item` is the one field a pure re-attribution rewrites. `at` is stamped either way — it orders the write, and this is a write. A requeue is the exception and stamps in full: "the output is gone, fetch it again" is a verdict this engine reached today. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An interrupted rename — corpus file moved, enrichment directory left behind — leaves a `done` unit whose output is on disk under the old id. The item's `enrichment:` listing is the markdown in ITS directory, so it derives `raw` with an empty listing forever, and a `done` unit is never drainable again. Lint named this by its ghost row; the seed-time sweep now heals the line's item id, correctly, and the row went with it. Moving files is not the drain's business, and withholding a correct re-attribution to preserve a lint signal would be worse — it would leave the dead id for the drain to carry forward on any rerun, the exact fault the sweep exists to fix. So lint asks the output path twice instead: is it there, and is it where its item can see it. Different states, different repairs, both judgment and neither an exit-1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reverting `derive_no_source_items`, `_incomplete_items`, `_last_enriched` and `mark`'s refresh loop to filter on `entry.item` left the whole suite green — a read site no test distinguishes from the old filter is not actually routed through the corpus. Each gets the case that separates them: two LIVE twins sharing a URL (the excluded-twin tests cannot pin it, because there the sweep has already rewritten the line's item), the shape row for an item whose second unit is ledgered under its twin, the staleness backstop's landing date for a renamed item, and a `mark` heal on a renamed item's unit. Mutating all nine routed reads now fails at least one test each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seeding dedupes by work hash, so a URL shared into two captures is ledgered once and enriched under the first item only. `_derived_status` returned `raw` on an empty `enrichment/<id>/` before it ever asked the ledger, so the second item sat `raw` permanently, on no surface, with nothing that could move it: `fetch` refuses a URL another item enriches, `mark` heals the unit — which was never the thing that was wrong — and a digest pass cannot lift a status it does not derive. Owing is the whole question and the ownership map answers it. The directory listing still answers for one item: no units AND no files is a text-only capture, which owes its description and its digest. `digest_orphans` walked enrichment directories, so the co-claimant — now `enriched`, and therefore digestible — had nothing to be found by. Its candidate set gains the live items the ledger records a landing for. The `enrichment:` listing stays empty for the co-claimant, deliberately: the field is the item's own directory, its entries are bare filenames every reader resolves against `enrichment/<id>/`, and one file for one unit is named by one item. `enrich status <item>` is where the shared unit shows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The design doc and the ingest-item reference both still read `enriched` as "holds enrichment AND owes nothing", which is now wrong in both halves. The status is derived from the ownership map alone: an item's own directory answers where its files are, which is a different question from what it owes. The one exception is the capture that seeded nothing at all — no units and no files owes its description and its digest. The consequence worth writing down is `enrichment: []` on an `enriched` item. Seeding dedupes by work hash, so a URL two captures share is fetched once and lands under whichever hit first; the second item is enriched on that landing and its listing is legitimately empty, because the field names its own directory. Read as a fault, it invites exactly the hand-repair the listing rule exists to prevent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`stamp` grew a provenance-carrying mode and the doc never said so. `date` and `engine` answer when, and by which engine, the work a line records was done, so a pure re-attribution — the sweep moving a line off a dead id onto the item whose corpus file claims that work — carries them and rewrites only `item`, stamping `at` because ordering a write is what `at` is for. Both fields are load-bearing, which is the part a reader needs: re-stamping `engine` spends the retry-on-new-engine shot the same section documents, without ever running it. A requeue is the exception and stamps in full. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The health report's third referential row — a done entry whose output sits under another item's directory — was landing on guidance written for the two rows above it, whose repair is "the item is gone, decide whether its entries follow". Nothing is missing in the third case: the output exists, filed where its own item cannot see it, so that item shows an empty `enrichment:` listing and stays raw while the unit is done and never drainable again. The repair is finishing the interrupted rename or re-fetching, and it is judgment either way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…luded `exclude` keeps a ledger line a surviving co-claimant shares, and deletes `enrichment/<purged>/` with the item that produced it. The survivor's URL is then `done` with nothing on disk, and seeding's already-a-unit short-circuit means no run ever fetches it again: the item owes nothing, derives `enriched`, and the digest and query layers have no file to read. Recorded here, where the deletion is a fact this command knows, rather than inferred by a later run from a path that is missing for innocent reasons too — a rename moves the enrichment directory as well, and re-fetching on that puts a `via: extract-asset` unit's repo path into the fetch queue, where no transport can take it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The readers already resolve ownership off the corpus; only the drain read the line's stored string, carrying it into the outcome line it records and into the `enrichment/<item>/` path it writes. One unrouted write path forced a seed-time sweep that REWROTE persisted lines to heal the string, and that sweep reconstructed an output naming convention it got wrong for two of three shapes, silently spent an `error` line's one retry-on-newer- engine, and could not heal a `via: extract-asset` unit at all. So the write path asks the same question instead: the outcome line, the enrichment file, a media download's directory and slot, the transcribe drain's park file, a redetection's superseding line, and the `WorkUnit` a driver is handed. Where nothing live claims the unit the stored string stands, exactly as the read side falls back. Nothing rewrites a persisted line, so `ledger.stamp`'s provenance carve-out goes with the sweep: every write records work, and every write stamps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both resolve a path under `enrichment/<id>/`, and both were asking the line's stored `item`. With nothing rewriting that string, a rename carried all the way through now leaves a recorded path naming a directory that is gone while the file sits under the new id, under the same name — which read as an output missing from disk, a finding with nothing behind it. So each check looks in the owning item's directory as well: a completed rename answers neither, and the interrupted one — corpus file moved, enrichment directory left behind — answers the misfiled row, named after the live item, the one that cannot see its own output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The design described a seed-time re-attribution and a provenance carve-out for the write it made, both of which are gone. What is true instead: the write path resolves the owner the same way every reader does, so a stale stored `item` never needs healing — stated in §4 alongside the reader list it belongs with, and §5's stamping rule loses its exception. And the lint skill had no reading at all for the `renamed` ghost row it emits, which is the row a session will meet most often and the one that needs nothing from it. It says so now, and points at the misfiled-output row for the half of an interrupted rename that IS work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
leeovery
changed the title
leeovery
marked this pull request as ready for review
August 23, 2026 21:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ownership resolved from the corpus, so a renamed item's stale ledger lines become harmless history.
pipeline.ownership.corpus_claims; a hash claimed by two live items is owed by bothowner_of), so no persisted line ever needs healing — the earlier re-attribution sweep is deleted along with its provenance carve-outexcludere-queues a landing whose enrichment it deleted while a surviving co-claimant still lists the URL — the survivor's URL gets fetched againenrich status(id, URL, reason), and the empty-run report no longer claims instance-wide quiet🤖 Generated with Claude Code