Skip to content

Pipeline: media wedge, unhealable units, premature deletion - #17

Open
leeovery wants to merge 6 commits into
fix2/driversfrom
fix2/pipeline
Open

Pipeline: media wedge, unhealable units, premature deletion#17
leeovery wants to merge 6 commits into
fix2/driversfrom
fix2/pipeline

Conversation

@leeovery

@leeovery leeovery commented Aug 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Eleven confirmed pipeline defects from the sweep, including two blockers.

  • A relative og:image bricked enrich run permanently. The media redrain sat outside the pipeline's single broad except, so an unfetchable media URL charged its failure to the parent (superseding a good done and filing a spurious issue), and every subsequent run died before doing any work. Media URLs are now absolutised and validated before they can enter the queue, and the dispatch runs inside the same protection as everything else.
  • mark could not heal any raw-keyed unit. Bad seeds and every via: media line are keyed on the URL verbatim, while mark canonicalised first, so the hashes could never meet and the contract forbids hand-editing. mark now resolves canonical-first, then by the exact stored key.
  • Re-detection deleted the old output before the corrected kind produced one, so a corrected fetch that parked left the item empty and its digest orphaned. The stale file now goes only when its replacement lands.
  • A cross-item URL aborted a fetch batch mid-way; it now reports and continues.
  • A cap refusal rendered as a bare skipped, hiding the --force affordance the design says the owner is owed.
  • .env was read outside the CLI's error wrapper (raw traceback on an unreadable file) and kept literal quotes around values.
  • The enrichment url: line bypassed YAML quoting, so a work key containing ": " made the whole frontmatter unparseable.
  • _refresh_item_frontmatter missed UnicodeDecodeError, so mark/pass exited 1 after their write had landed.
  • Seeding's media read is guarded, LedgerEntry rejects multi-line URLs, and a capture still carrying a staged asset is refused loudly instead of silently losing the binary reference.

Gates: 1082 tests, ruff and ty clean.

🤖 Generated with Claude Code

@leeovery leeovery changed the title fix2/pipeline Pipeline: media wedge, unhealable units, premature deletion Aug 22, 2026
leeovery and others added 6 commits August 22, 2026 15:13
@leeovery @claude
Keep one bad media URL from bricking every future run
8992702 
A page-relative og:image reached the media stage verbatim, where the
transport refuses it as a ValueError — outside the connection-failure
lifecycle. The inline download sat outside the drain's single broad
except, so the failure superseded the PARENT's done line and the next run
died unhandled, permanently.

Three layers, all of them:

- the web driver absolutizes og:image against the page, unescapes
  entities, and emits nothing that is not http(s); its regex stops at a
  line break so a wrapped content attribute cannot yield a multi-line URL;
- the media stage validates every URL before the birth line — anything
  unfetchable parks manual as its own media unit — and runs the download
  through _process, so any exception class is charged to the media unit;
- the via:media dispatch moves inside the one broad except, restoring the
  exactly-one-except contract.

The media slot comes from the unit's position among the item's media
units in ledger order rather than a disk scan: the crash window between
the file write and the outcome line now overwrites media-0 instead of
writing media-1 beside an orphan.

Also: LedgerEntry rejects a multi-line url (the item-status surface
renders it verbatim), and seeding's media read — the one unguarded read
left — notes an unreadable file instead of dying.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@leeovery @claude
Make raw-keyed units healable and never delete unreplaced content
dbe13fa 
Three heals the pipeline could not perform:

- `mark` canonicalized before looking a unit up, while bad seeds and every
  via:media line are keyed on the URL verbatim — the two hashes could
  never meet, so 100% of that class was unhealable and the contract
  forbids hand-editing the ledger. It now resolves canonically first, then
  by the exact stored key, and the healed line keeps the entry's own
  hash/url.
- A redetection unlinked the old kind's output at correction time. When
  the corrected fetch then parked, the item's enrichment directory was
  empty, its status fell back to raw and its digest was orphaned with
  nothing left to re-derive from. The stale file now leaves on the
  success that replaces it.
- `_refresh_item_frontmatter` missed UnicodeDecodeError (a ValueError, not
  an OSError), so one non-UTF-8 corpus item made mark and pass exit 1
  after their write had already landed — and the retry duplicated it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@leeovery @claude
Answer every refusal instead of aborting or staying silent
97c5ffb 
Five surfaces that swallowed what the owner needed to see:

- a cross-item URL raised out of `enrich fetch` mid-batch, aborting URLs
  already ledgered and losing the report with them; it is reported now,
  naming the owning item, and the batch continues;
- a capped fetch refusal rendered as a bare `skipped 1`, leaving the
  --force route unreachable — it appends a report note carrying the stored
  reason, and a repeat refusal no longer appends a byte-identical line;
- `_load_env` sat one line outside enrich.main's try, so an unreadable
  .env (non-UTF-8, or a directory) came back as a raw traceback; it also
  kept surrounding quotes on values, and `KEY="sk-…"` 401s every call with
  a baffling waiting reason;
- the enrichment `url:` line bypassed `_yaml_value`, so a work key like
  `file:media/plan: v2.pdf` made the whole frontmatter unparseable;
- `enrich item new` silently turned a capture still carrying `asset:` /
  `name:` frontmatter into a text item, losing the binary's provenance at
  exit 0. It refuses loudly and points at `dex inbox`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@leeovery @claude
Drop only the superseded output the unit itself wrote
7f084ae 
A redetection's stale output left the disk by pattern: every
`*-<hash6>.md` in the item directory that was not the file just written.
`hash6` is six hex digits, so two units under one item collide often
enough that a real pair exists (`web-6968e3.md` beside `file-6968e3.md`)
— and whichever landed second deleted the other's enrichment, leaving a
`done` ledger line pointing at nothing.

Candidates are now the closed `<kind>-<hash6>.md` set, and each must
prove it belongs to this unit by the `url:` it records before it is
unlinked. Nothing is dropped for a replacement that is not itself on
disk.

The drop also runs on the mark route. The prescribed recovery for a
web→file correction whose corrected fetch parks — a scanned PDF, no
extractor — is a hand-written enrichment closed with `enrich mark <url>
done --path …`, and that output is one of the unit's own. Without the
drop the item kept two files for one unit, both listed in frontmatter,
and served the stale pre-correction view to the digest and query layers
permanently.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@leeovery @claude
Cap media on the files that exist, never on ledger positions
dae9772 
The media slot became the unit's position among the item's media units,
which fixed the crash-window duplicate — and silently made that position
decide the cap too. A unit's parked, dead and skipped siblings then spent
cap they had put no file on disk for. An X thread pooling six photos
whose first two 404 ended as `p2 done, p3 done, p4 skipped "media cap (4
files) reached", p5 skipped` with two files in the item: `skipped` is
terminal, so p4 and p5 were lost forever, under a reason that was false.

The position still names the file — reruns overwrite, never duplicate —
but the cap decision counts media-family files that exist, as
`_media_file_count` does and as the docstring already claimed. An index
past the cap is ordinary: that thread now lands its four survivors at
`media-2` through `media-5`.

The "this slot already holds my own file" shortcut returned before the
cap check on a glob that matched `media-<n>.md` — the session's written
description of a media capture, which `_media_file_count` deliberately
excludes. One `_is_media_file` predicate now answers both questions, so
an item can no longer end with five media-family files and a downloaded
`media-0.png` beside a `media-0.md` describing something else.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@leeovery @claude
Emit no og:image at all for a wrapped content attribute
1c4f123 
The capture stopped at the line break, so a wrapped `content=` yielded
the truncated head of the URL — `https://cdn.example.test/` — which is a
perfectly well-formed request for a resource that does not exist. That
became a real media unit and a guaranteed junk fetch, and the test
pinned it as the intended outcome rather than a defensible one.

The value must now run from the opening quote to the closing one: a URL
this driver cannot read whole is not a URL it hands on. The page still
enriches; it just carries no media.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@leeovery
leeovery marked this pull request as ready for review August 22, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@leeovery