Drivers: private-repo blobs, Apple episodes, channel URLs, X articles - #16
Open
leeovery wants to merge 9 commits into
Open
Drivers: private-repo blobs, Apple episodes, channel URLs, X articles#16leeovery wants to merge 9 commits into
leeovery wants to merge 9 commits into
Conversation
raw.githubusercontent.com is unauthenticated: it 404s every private-repo blob however the machine is signed in, and classify_http(404) condemned that live content as dead. Blobs now ride the contents API through gh like every other route, base64 body decoded from the JSON payload. A blob over the API's inline ceiling parks manual rather than dying, and a path that genuinely does not exist still classifies dead through gh's own 404. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The iTunes lookup API resolves show ids only: handed the ?i= episode id the driver was sending, it answers resultCount 0 for every episode that exists, so the Apple route could never resolve anything. The show id comes from the /idNNNN path segment, the lookup asks for the show's episode window, and the episode is matched by trackId against ?i=. An episode older than the window parks manual saying so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A /@handle, /user/, /c/, /channel/ or /results URL fell through to the probe, which answers it by enumerating the whole channel (87s and 63 videos on a live run) — and the transcribe drain would then download all 63 audio files over one filename. These shapes are collections like a playlist and park manual the same way, ahead of any probe. A channel's /live path addresses a single video and still fetches; the canonical form of every collection shape is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fxtwitter returns text: "" for long-form articles, the prose under article.title / article.preview_text, and raw_text.text holding only the shortlink — so the raw_text fallback ledgered a fifth of the wild x corpus done on a ~74-char body of URL. Articles now render title, preview text and the link, and a body that is nothing but a t.co shortlink counts as no content at all, taking the existing "no text or media" manual park. The new fixture carries raw_text, which the old ones omitted — that omission is why the tests could not see this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/features, /topics, /sponsors, /orgs, /collections, /marketplace, /trending, /about, /pricing, /settings and their siblings are neither user nor repo: the API 404s them while a browser renders them fine, so the driver was condemning live pages as dead. matches() now screens reserved first segments and registry ordering hands them to the catch-all, which extracts them like any page. Only the first segment is screened — acme/topics stays an ordinary repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A PDF or image committed to a repo was decoded with errors="replace" and ledgered done as 40k characters of replacement-character soup. Blob bytes are now sniffed first: a recognized document format parks manual naming the format, and anything else that is not UTF-8 text parks manual saying so. Neither re-detects to file work — a GitHub blob URL serves an HTML viewer rather than the bytes, so the file driver would fetch that page, find HTML and re-detect straight back; the rescue route is capturing the file itself, which is already file work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
leeovery
changed the title
raw.githubusercontent.com resolved the ref/path boundary server-side. The contents API takes the two halves apart, so splitting a blob URL at the first segment sent a wrong ?ref= with a wrong path: every slashed branch name and every blob/refs/heads/ permalink 404'd, and a 404 on contents is terminally dead. Guess the shortest ref the URL's shape allows — free, and right for nearly every link — and only when that 404s ask git/matching-refs which of the repo's refs the path really starts with. Matching is segment-wise so automation/bors cannot claim a URL whose branch is automation/bors-next, and the longest match wins. When nothing re-splits, or the lookup itself fails, the original classification stands: a missing path is still dead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The guard read /@handle, /user, /c and /channel, and missed the oldest channel address of all: the bare vanity name at the root. /veritasium and /veritasium/videos reached the probe, which enumerates the whole channel, and the transcribe drain would then pull every video's audio over one filename. Percent-encoded handles (/%40jamesbriggs) and /hashtag/<tag> slipped through the same way. Nothing in a bare root segment says channel, so name youtube.com's own functional first segments — /watch, /playlist, /results, /feed, the video prefixes, the product surfaces — and read everything else at the root as a channel. The list errs toward parking: a missing functional segment costs one recoverable ledger line, a missed channel costs a full enumeration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sniff_format ran unnamed while file_path sat two lines above it. An unsmudged Git-LFS pointer is honest UTF-8 with no byte signature, so 130 bytes of `version https://git-lfs.github.com/spec/v1 / oid sha256:…` decoded cleanly, fenced, and ledgered done as though they were the document they stand for. detect.py already documents the extension fallback as existing for exactly this. Naming the file also routes a committed CSV by extension, so it parks manual rather than fencing. Deliberate: a CSV has no signature either, and letting text-shaped documents fence is what let the LFS pointer for a .csv through. One rule — an extractable document parks for capture — and csv-builtin then renders a real table instead of a fence truncated at 40k characters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
leeovery
marked this pull request as ready for review
August 22, 2026 14:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six confirmed driver defects from the nine-lens engine sweep.
dead. Blobs fetchedraw.githubusercontent.comunauthenticated, which 404s for private repos regardless of gh auth; 36% of the real blob population sits in private repos. Now fetched through the authenticatedghcontents API like every other GitHub route.resultCount: 0, verified live). Now looks up by show id and matches ontrackId. The test fixture encoded the same wrong assumption and has been corrected./@handleprobe enumerates the whole channel (87s, 63 videos, verified live) and the transcribe drain would have downloaded 63 files over one name. They now parkmanualbefore any probe, like playlists.donewith a bare t.co link as their body (20% of the real X corpus). Now rendered from the article payload; a body that is only a shortlink is not content./features,/topics,/sponsors, …) were 404'd intodead; they now decline to the web driver. Four candidate names were dropped from the reserved set after checking the live API, because real accounts own them.done; they now park with the format named rather than storing 40k characters of replacement characters.Gates: 1121 tests, ruff and ty clean.
🤖 Generated with Claude Code