Order the ledger by write time, and make every state write honest - #21
Merged
Conversation
Git's union driver emits ours-then-theirs, so the last line of a hash after a merge is whichever side git appended, not whichever machine wrote later: a scheduled run's 09:00 blocked could supersede the owner's 10:00 manual, the unit re-drained against the owner's decision, and compact then deleted the newer line permanently. Every line now carries `at`, the UTC write instant with sub-second precision, stamped through ledger.stamp from an injected clock (the two migration seams gain the same injection). load resolves each hash by (at, file position) with absent treated as oldest, and compact keeps whatever load resolves to. No backfill: every writer after this release stamps one, so an unstamped line necessarily predates every stamped one. Migration 1's tolerated-keys list gains `at` and its translation carries the value through — the exact re-apply exposure that list already had with `capped`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The quarantine parked lines in state/enrichment-ledger.unmigrated.jsonl and three surfaces told the session to re-add them with `enrich mark` — a verb that heals a unit, and so structurally cannot re-add a line with no attributable item. A migration is fully automated and leaves no residue. Attribution now tries harder before giving up: an old line that recorded no item and no path may still have left its output on disk, and outputs are named enrichment/<item>/<kind>-<hash[:6]>.md, so the tree names the owner. A prefix landing under two items is dropped rather than guessed. What survives none of the four attributions is dropped, counted, and named in the report (first five, then the remainder counted). Safe by what the ledger is: the corpus is the source of truth and the ledger is derived work state, so seeding re-raises anything that still matters through the front door; and the pre-migration ledger is in git history. Verified against a copy of the real dex-engineering state: all 22 formerly quarantined lines drop, no residue file is created, and a second apply reports nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`bin/dex exclude` deletes a corpus item and its enrichment permanently and on the record, but left the item's ledger lines behind. Seeding builds work from corpus items' URLs, so those lines name work nothing can ever raise again — they linger forever and read as state. The verb now drops them in one atomic rewrite after the exclusions record lands, and states the count. Unparseable lines are kept: a line the code cannot read is not provably the excluded item's, and a purge must not become incidental data loss. Git history keeps what goes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`at` breaks ties between concurrent writes; it is not load-bearing state like item, kind, status or date. Refusing it was defensible while untranslatable lines went to a quarantine file a human could inspect — with quarantine gone, that same strictness deletes a line's whole work history to protect a cosmetic field. Migration 1 now drops an unreadable `at` (unparseable, or naive and so uncomparable across machines), names it in the report, and keeps the line, which then orders by file position exactly as every line did before the field existed. `ledger.load` keeps its strictness untouched: a malformed field in a live ledger still refuses to load. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`exclude` now takes an item's ledger entries with it, but purges made before that left theirs behind, and so did items removed by hand. On the owner's dex-engineering state that is 10 entries — 8 from purges, 2 removed by hand. Migration 4 is the one-time historical cleanup, run automatically so no session has to notice. Same disposal argument as migration 1's dropped lines: the corpus is the source of truth, seeding can never raise this work again, and git history holds the entries. Same reporting shape: a count and a capped named list, telling excluded-on-record apart from removed-by-hand. Two distinctions the sweep is careful about. The ITEM is the test, never the output file: an entry whose path is missing from disk is a different finding with a different repair, since the item still exists and the fix is to re-fetch. And the unit swept is the hash, decided on the line `load` resolves to, so a hash whose live line names a real item is kept whole even where an older line named a purged one. `ledger.resolution_key` becomes public so the migration resolves hashes by the same rule `load` does instead of reimplementing it — the migration reads the file tolerantly, because one hand-tampered line must not abort a sweep, and a sweep must never remove what it cannot read. The design doc's migration-2 rationale still claimed a purged item's ledger history stays on file; corrected, and pointed at the sweep. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migration 1 read the line's stored `path` before the enrichment tree, so an item renamed since the line was written — same shortid, new slug — was attributed to a directory that no longer exists. That hands the ghost-item sweep an item with no corpus file and the work reads as a purge. The tree answers first now, and where it disagrees with the recorded path the path moves onto the attributed item too, but only where the file is demonstrably there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`drop_items` matched per line on the stored `item`, but a work unit is keyed by URL hash: two corpus items sharing a URL share one entry, which names only one of them. Excluding that one deleted the unit's whole history while the other item was still claiming it. dex-engineering has 80 hashes claimed by more than one live corpus item (24 queued, 51 done, 5 dead). It now judges the hash on the line `load` resolves to and asks the corpus which work survives, exactly as migration 4 does — scanning after the deletions, so a purged item cannot claim its own work. The summary states both counts; `exclude` runs in bulk from the scope-filter pass, so that line is the owner's only signal. Lint's docstring said `exclude` "leaves the ledger history standing, deliberately" — false since this branch started sweeping it. It now names what an id with no corpus file actually means. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migration 4 inferred "this item was purged" from "its corpus file is missing". That inference is ambiguous by construction: a rename, a partial checkout and a hand deletion are indistinguishable to it, and each guard added afterwards was a patch over the first guess. It shipped to every instance forever to tidy a handful of lines in one, and it deleted a real 17-minute transcription on the way. The residue it existed to clean is created by migration 1 attributing a line to an item that no longer exists, so the repair belongs there. The shipping set is 1, 2, 3; every count expectation moves with it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An excluded item loses its corpus file and its enrichment directory, but a done line's recorded path still spells the id — so reading the id back out of that string attributed finished work to something that no longer exists. That residue is what the deleted migration 4 was built to sweep; it is created here, so it is fixed here. Every derived attribution is now checked against the item's corpus file before it is accepted, and a line left with none is dropped, counted and named exactly like a line that never named anything. Rescue effort stays with items that are still there: a renamed item resolves through the enrichment tree or the corpus URLs and keeps its work, path repoint included. On a copy of real dex-engineering this drops the 8 entries that previously survived migration 1 to be swept later, leaving a ledger identical to the four-migration chain's — and the 17-minute transcription under the renamed item survives, repointed, as it must. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`at` orders a hash's lines, and it is another machine's wall clock: a jumped one stamped a line 2099 and owned the hash outright. `mark` reported success, the item's frontmatter was rewritten from the drain and read healed, `load` went on resolving the stale line, and `compact` deleted every correct write behind it. So `at` is the writer's own clock or nothing — `ledger.stamp` is the one place it is set, migration 2's seeds included — and a stored value more than FUTURE_SKEW_ALLOWANCE ahead of the reader is read as unstamped: it sorts oldest, loses to every real write, and `compact` drops it rather than them. A bad clock costs one line's ordering, never the hash. Resolution now reads a clock, which stamping still must not: the default is the wall clock, injectable so the ceiling is testable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migration 2 read last-per-hash off file position while `load` reads it off `at`. Harmless on a first sync, where pre-release lines carry no `at` and both rules fall back to position — but the framework promises re-application against already-migrated state, and there a union merge leaving the older line last makes position name a superseded `done` and requeue work the owner has since ruled on. `resolution_key` was made public for exactly this and had no migration caller. It has one now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rule is categorical: a malformed tie-breaker is never grounds for dropping a line. It held only for a string that fails to parse — `"at": 1755772800` and `"at": null` went through `_expect_str`, raised untranslatable, and destroyed the whole work history of a line naming a live corpus item and a real output file. Any unreadable `at`, whatever its JSON type, now costs the value and is named in the report; the line keeps its history and orders by file position, as it did before the field existed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`shutil.rmtree(enrichment_dir / item_id)` took the id straight from the exclusions JSON. `Path / "/abs/path"` discards the left operand and `Path / "../../x"` climbs, so an id shaped like a path deleted a real directory outside the instance, reported "removed 0 items (1 already gone)", exited 0, and recorded itself in exclusions.tsv permanently. The file is LLM-authored and this runs in bulk, so a wrong id is a realistic input, not a hypothetical. An id now names one file in the corpus tree by shape, and both paths it removes are resolved under the root through `resolve_repo_path` — the same containment check the `file:` work key already uses, which also catches an enrichment directory symlinked out of the instance. A refused id reaches neither the disk nor the TSV. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`exclude` indexed and `.split()` the JSON values with no type check, so a
batch of three whose middle entry was `{"id": 999}` died on a TypeError
`main` does not catch. Entry 1's exclusion record was already in
exclusions.tsv — permanently, in a file `normalize` consults, so its
clusters are never regenerated — while the ledger purge that runs after
the loop never ran, and entry 3 was lost without a word.
Every entry's types are now checked before anything is written or
deleted, and a bad one refuses the batch naming the entry. A refused
batch leaves no trace.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The claimed-hash veto is asked of the corpus, and `corpus_owners` skips an item it cannot parse. That silence fails OPEN toward deletion: exclude item A while live item B shares one of A's URLs and B's frontmatter is unparseable, and the shared hash's whole history goes, reported as a clean drop with no mention that a file could not be read. On one real instance 80 hashes are claimed by more than one live item. An unreadable corpus file now holds the whole purge — the design already keeps an unreadable LEDGER line rather than purge what it cannot judge, and a file it cannot read is the same call — and the summary names the count and the repair. The item and its enrichment still go; the re-run purges once the file parses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`resolve_repo_path` is no longer only the `file:` work key's: exclude resolves the paths it deletes through it too. Docstring only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The future ceiling bounds a clock that ran ahead. Nothing bounds one that ran behind — a dead RTC, a container with no NTP, a restored snapshot — and in that direction `mark` stamps the owner's correction in the past, it loses to the line it was written to supersede, and `compact` then deletes it as superseded with no audit trail. Strictly worse than the forward case, which at least kept the correction, and the verb reported success throughout: `mark` printed "marked … manual", `status` still read done, and the heal was gone by the next compaction. A symmetric floor is not available (there is no reference point for "too old"), so the verb verifies its own write instead. `mark` re-reads the ledger, and unless the hash now resolves to the line it just wrote it fails naming the URL, what the ledger resolves that unit to instead, and this machine's clock as the likely cause. It fails BEFORE dropping the superseded outputs and refreshing the item's frontmatter, so nothing acts on a correction the ledger did not take; the appended line stays as the audit trail of the attempt. The same hole one layer up: the enrich-report states outcomes on the strength of having written them, so `run`, `fetch` and `transcribe` read their own writes back at `_finish` and name the units whose lines did not land rather than reporting them as work that landed. The resolution rule is untouched — the forward ceiling stays exactly as it is. This is only about a verb telling the truth about its own write. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`existing` was read once before the loop and never updated inside it, so the same id twice in a batch appended two identical rows to `state/exclusions.tsv` — a permanent record — and the second pass found the file the first had just deleted, reporting "1 already gone" for an item nothing had removed. The exclusions file is LLM-authored and runs in bulk, so the same id twice is a realistic input, and the summary line is the owner's only signal about what happened. A duplicate is not an error: excluding an item is idempotent by construction — the TSV record is exactly what makes a re-run safe — so the second copy asks for what the first already did, and refusing the whole batch over a harmless redundancy would send a bulk file back for hand-editing. The copies collapse to one entry, first occurrence winning, and the summary states how many it collapsed so the count is never silently short. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An output is named `<kind>-<hash6>.md`, so the identity re-key left every moved unit's file on disk named for a hash nothing computes any more. Nothing in the engine could reach it again: `_drop_superseded_outputs` builds its candidate names from the NEW hash, so the rerun the migration seeds landed BESIDE the stale file and the item carried two views of one unit — the pre-rewrite one (link-stripped, non-thread) and the new one — both listed in engine-owned `enrichment:` frontmatter, which is derived from the directory and therefore ADDS the orphan. Every pre-rewrite `x` entry on a real instance triggers it: twitter.com and x.com both re-key to x.com/i/status/<id>. The second leg is silent data loss. A `nocaptions` youtube park on a /live/, /shorts/, /embed/ or youtu.be URL becomes waiting + transcribe under migration 1, is re-keyed here, and the transcribe drain then reads `youtube-<newhash6>.md` — which does not exist — dropping the description the park had stored. Nothing detects or repairs it afterwards: lint's only enrichment read is the thread-marker scan, and the engine's two unlink sites are the superseded-output drop (wrong hash) and `exclude` (the whole directory). So the file moves with the hash, following migration 1's precedent for the outputs its own rewrite renamed — and the entry's stored `path` follows the file, keyed by the whole path so a superseded line naming it follows too. Every kind prefix is tried, since a redetected unit's output sits under the older kind, and each candidate must prove it belongs to this unit by the URL it records: six hex digits collide, and moving a neighbour's file would strand THAT unit. A target that already exists — both old spellings of one post landing on one identity — is refused and reported rather than overwritten, and an unreadable file is skipped with why rather than aborting the chain. `tests/migrations/test_migration_2.py` never created an `enrichment/` directory, which is why the suite was green; the new cases build one and cover both legs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`exclude` unlinked the corpus item, rmtree'd `enrichment/<id>/` and purged the item's ledger entries, and left `state/digests/<id>.md` on disk. Nothing else ever deletes one: the drain writes digests, the superseded-output drop only knows enrichment names, and lint's digest check is shape-only, so the file was never named by anything afterwards. The item's permanent fact index therefore outlived its permanent exclusion, still feeding query and wiki, undetected. The trigger is ordinary: excluding an item that has already been digested is exactly what re-judging scope at a health check does. The digest path is resolved through `resolve_repo_path` like the other two, so a digest symlinked out of the instance refuses the batch rather than deleting outside it, and the summary counts what went — a line that read "removed 1 items" while a file it did not mention stayed behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The design doc still described the state before this branch on three points, so a reader reconciling it against the code would read each of them as drift. Migration 2's section never mentioned the output file, while migration 1's carries the hazard note for its own half of the name. Migration 2 now renames `<kind>-<hash6>.md` with the hash and repoints the entry's `path`, proving ownership by the URL the file records because six hex digits collide, and refusing a pre-existing target rather than clobbering it. The `at` section stopped at the forward skew ceiling, which is the only direction it bounds. A clock set backwards had `mark` reporting success while the heal was inert and `compact` deleting the correction, so `mark` now reads the ledger back and raises, and the drain's exit path reports the same check as a note because a run does real work. And `exclude` now collapses a duplicate id inside one batch instead of writing the permanent record twice, with the count on the summary line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
leeovery
changed the title
leeovery
marked this pull request as ready for review
August 23, 2026 21:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ledger's union-merge story, and exclude hardened.
atwrite timestamp: last-per-hash resolves by write time, not git merge order; a forward skew ceiling (5 min) stops a future-dated line owning its hash; legacy no-atlines resolve exactly as beforemarkverifies its own write — re-reads the ledger and raises naming the clock if the heal did not land (backwards-clock protection); the drain reports unlanded writes as a noteexclude: path containment before any delete (absolute/traversal/symlink ids refused loudly), whole-batch validation with no half-written state, duplicate ids collapsed and counted, the item's digest removed with it; the claimed-hash veto fails closed when a corpus file will not parseat(value dropped, line kept); migration 2 resolves last-per-hash by the ledger's own rule and renames<kind>-<hash6>.mdoutputs when it re-keys identities (ownership proven via the file's recordedurl:)🤖 Generated with Claude Code