Claude Fable 5 and 5.1 on annapolis, via a second serving path (#156) - #157
Open
jonfroehlich wants to merge 6 commits into
Open
Claude Fable 5 and 5.1 on annapolis, via a second serving path (#156)#157jonfroehlich wants to merge 6 commits into
jonfroehlich wants to merge 6 commits into
Conversation
…aks (#156) Item A of #156 cannot run until the project enables data sharing for the `anthropic` publisher, but three things that do not depend on that decision were missing, and two of them would have cost a paid run to discover. The probe that produced #156's 403/404/200 table existed only in a session transcript. It is now a script, with the deliberately-misspelled `claude-fable-5.1` kept as a CONTROL rather than dropped as a typo: a 404 on a spelling that does not exist is what proves the 403s resolved real model ids behind a per-model gate, instead of a blanket block on the publisher. `claude-opus-5` rides along as the positive control, without which a wall of 403s is indistinguishable from broken credentials. Re-run 2026-09-04: unchanged, gate still closed. Two API differences in the Fable family would each have surfaced as a confusing failure mid-leg: - Forced tool use is gone -- `tool_choice` of `tool`/`any` returns a 400. The detector defaults to `auto` so a default run is fine, but `--claude-tool-choice forced` would have 400'd on every one of ~750 views with an error that names neither the flag nor the family. Refused at construction instead. - Thinking cannot be disabled there, so the near-zero-thinking floor that makes our effort=low Opus leg cheap does not exist, and thinking bills against `max_tokens`. That was hardcoded at 4096; it is now `--claude-max-tokens`, and the truncation error says which of the two knobs actually helps on Fable. `max_tokens` enters the detection signature only when it DEVIATES from the as-run 4096, the same rule image_format and temperature follow -- recording an unchanged default would rewrite the key of every paid detection already in the cache. test_claude_cache_key_is_frozen still passes on all four published legs. Not done here, and not blocked on the same thing: pricing.py rows still need the Vertex Anthropic rate card read by a human (the page does not render through WebFetch, and this repo does not take a price from memory or an aggregator). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjWxvSJybx4xJUaVTvGnR7
Vertex gates the Fable family behind a publisher data-sharing setting that is not ours to flip, so #156 item A now has a second candidate serving path: Anthropic's own API. Same weights, different credential, different rate card. Before any of that reaches the detector, the question the probe already answers for Vertex has to be answerable for first-party too -- *is this id real, and is it enabled for us?* -- because the alternative is discovering the answer partway through a paid leg. Generalized rather than duplicated. The probe's actual work -- call each id with a 16-token budget, print the status verbatim, run a deliberate misspelling as a 404 control -- is identical on both paths; only the client construction differs. So `--serving-path {vertex,anthropic}` selects it and the file loses the `_vertex_` in its name, which was already the wrong name for what it does. Nothing referenced it, so the rename costs nothing. Vertex stays the default: it is what the four published Claude legs ran on. Three things the first-party path needs that Vertex did not: - Its status codes mean different things, and the docstring now says which. 401 is a credential problem and nothing else; 400 is a request that was understood and refused, which here is either an empty credit balance or the Fable family's data-retention requirement -- both name themselves, hence printing the provider's message verbatim. - The 404 control earns its keep twice over. On Vertex it separates "wrong spelling" from "not entitled"; on first-party it separates "not a model" from "not enabled for us", which otherwise share a status code. - The key lives in the gitignored repo-root .env, so the probe now loads it the way vertex_usage.py does. That also fixes a latent gap on the Vertex side: GOOGLE_CLOUD_PROJECT in .env was previously invisible to this script, which read os.environ only. A 200 now prints stop_reason and the token counts alongside it. That is not decoration: Fable cannot disable thinking, so a 16-token budget may be spent before any text is emitted. The call still proves reachability, and calling that a failure would be wrong -- but a reader needs to see which of the two happened. `--max-tokens` is a flag rather than an edit so bracketing Fable's thinking floor, when someone gets to it, needs no change to this file. Costs nothing to have run: every non-200 is free, and a 200 is a fraction of a cent. No detection cache and no usage-log rows are touched -- this is a probe, not a leg. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjWxvSJybx4xJUaVTvGnR7
…156) Vertex gates the Fable family behind a publisher data-sharing setting that is not ours to flip. Anthropic's first-party API serves both ids today (probed 2026-09-05, e0ca0df), so `--claude-serving-path {vertex,anthropic}` picks the account. `vertex` stays the default: it is what all four published Claude legs ran on, and nothing about them changes here. THE SERVING PATH IS NOT IN THE CACHE KEY, and that is the load-bearing decision. It names which account answered, not what was asked -- the same model id on the same views should return the same detections whoever bills for them. Putting it in `signature()` would fragment the cache along an axis that does not change the answer, and would orphan $28.82 of paid annapolis detections the moment any leg moved. test_claude_cache_key_is_frozen still passes on all four published legs, and a new test asserts the two paths hash identically. If the paths ever DO diverge in output, that is a finding to measure and write down, not a cache key to add quietly. So provenance goes to the usage log instead, as a top-level `serving_path` field beside `model_versions` -- same reasoning, it cannot be reconstructed later. `model_versions` cannot stand in: both paths report the bare model id. Without this field nothing distinguishes a Vertex leg from a first-party one afterwards. Three behaviours are Vertex-specific and now say so: - The transient-404 retry. That is entitlement propagating unevenly across Vertex serving backends; on first-party a 404 is permanent, and retrying it turns a mistyped --claude-model into 30 seconds of backoff before the same error. - The regional-pricing warning. GOOGLE_CLOUD_LOCATION is shared with the Gemini legs, so it can be set to a region for reasons unrelated to Claude. On Vertex that really does make cost figures ~9% low; first-party has no endpoint dimension, and warning anyway trains the reader to ignore a real alarm. - The credential error. Each path names what IT needs, and the first-party one names the prepaid trap: a valid key with no balance authenticates and then fails every call with a 400. pricing.py gains both Fable ids, read off claude.com/pricing on 2026-09-05 -- the page that governs THIS billing path, which is also why the rows that were blocked on an unrenderable Vertex rate card are no longer blocked. Both are $10/$50, exactly 2x claude-opus-5. The cache rows are why reading rather than deriving matters: Fable 5 reads cache at the standard 0.1x of input ($1.00), but Fable 5.1 reads at $0.25/MTok, which is no multiple of its input rate. The usual multipliers would have overstated 5.1's cache reads 4x. The two Fable legs are deliberately NOT registered in roster.py yet. A row there is a leg that has RUN and published detections -- three tests enforce that, and they caught me adding the rows in advance. They land with the annapolis results. Recorded next to the gap: `published_as` will have to name the serving path as well as the effort, because claude_serving_path is the first pin that is not a signature key and every pin's value must appear in the published name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjWxvSJybx4xJUaVTvGnR7
…legs (#156) Five annapolis panos on each Fable id, first-party path, effort low. Purpose was to replace the $19-$90 cost band in this issue with a measured number before committing to 750 calls. Answer: the low end. claude-fable-5-1 2,033 in / 136 out per call $0.0272/call -> $20.40 claude-fable-5 2,035 in / 98 out per call $0.0253/call -> $18.98 Input per call lands within 4 tokens of the claude-opus-5 measurement (2,031), which is the cross-check that the rig is sending identical pixels down a different serving path. THIS REFUTES A PREMISE OF THIS ISSUE. #156 reasoned that because the Fable family cannot disable thinking, the near-zero-thinking floor that makes our effort=low Opus leg cheap does not exist. It does. claude-fable-5 spent 1,618 thinking tokens across 48 calls (~34/call) and claude-fable-5-1 reported zero across 54; both sit near Opus's 110 output tokens/call. Always-on is adaptive, not a floor. The earlier probe hinted at this on a toy prompt; this is the same finding on real 1024x1024 views, which is what makes it load-bearing. Accuracy is NOT reported as a result -- n=5, and the confidence intervals cover most of the unit interval. It is recorded only because the direction is worth knowing before the full run: identical recall (4/5 both), but 1 false positive for 5.1 against 6 for 5. If that survives 125 panos it decides which leg is the one to publish; at n=5 it decides nothing. Cost of the calibration itself: $2.68 over 102 calls, against a ~$1.55 estimate for 60. The overrun is bookkeeping, not the models. A first attempt was launched with `set -x` in the wrapper, which would have traced the API key into a log file; it was killed for that reason, but it had already completed both models (60 calls, $1.58) -- killing a task does not un-spend it. The re-run then re-called 42 of 60 because most of the first run's detections were not in the cache when it started, a gap this commit does not explain and does not paper over. The cache is verified intact now: 5 panos per model resolve at the keys the detector computes today, so the full leg pays for 120 panos, not 125. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjWxvSJybx4xJUaVTvGnR7
Both clear the 0.567 gate and both beat claude-opus-5 at effort low (0.588), making this the first time a general-purpose model has displaced Opus on annapolis. The two are also indistinguishable from each other -- 0.610 vs 0.611 -- while sitting at visibly different operating points, which is the same shape as this benchmark's effort finding (#123): the knob moves precision against recall, not the ceiling. claude-fable-5-1 P 0.637 R 0.585 F1 0.610 172/98/122/15 $19.86 claude-fable-5 P 0.579 R 0.646 F1 0.611 190/138/104/12 $18.47 Cost landed within 1.5% of the calibration projection on both legs, and thinking held at the calibrated rate: 254 tokens over 720 calls for 5.1, 23,699 (~33/call) for 5. The always-on-thinking cost premise this issue assumed is now refuted at full scale on both ids. Ledger rows committed before any write-up: token counts are the one artifact a re-run cannot reproduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjWxvSJybx4xJUaVTvGnR7
Annapolis, first-party path, effort low, 125/125 panos each: claude-fable-5 P 0.579 R 0.646 F1 0.611 190/138/104 2.72 boxes/pano claude-fable-5-1 P 0.637 R 0.585 F1 0.610 172/98/122 2.28 boxes/pano Both displace claude-opus-5 (0.588), which had displaced gemini-3.1-pro (0.567). First time a general-purpose model has beaten Opus on this split. RampNet still leads by 0.228. The two are separated by 0.001 F1 -- nothing -- while sitting at visibly different operating points: 5.1 trades 0.061 recall for 0.058 precision and emits 0.44 fewer boxes/pano. Within this family the version is an operating-point dial, the same shape as effort (#123) and the Qwen 8B->32B inversion. Under this project's recall-first framing that argues for the OLDER id, which is worth saying out loud because the reflex is the opposite. Both are standing=False at 1/8 splits, and both clear the pre-registered 0.567 expansion gate -- so that expansion is now a live decision. It has NOT been taken and is priced in the doc (~$155/id for manual_gold alone). Publishing this took four guards to get right, and each caught something real: - The exporter wrote the BARE model id, exactly the failure its own docstring warns about: leg_for() could not match a leg pinned on claude_serving_path because the exporter had no such flag. Added, wired into overrides, and documented as NOT a signature field -- it plays no part in finding the cache, only in resolving the leg's published name from the registry. - test_each_published_file_names_the_leg_it_says_it_is assumed every pin is a signature field. That assumption is what claude_serving_path breaks by design, so the published envelope now carries `pins` and the test looks there for pins the signature does not hold. Legacy files lack the field and never reach that branch, since their pins are all signature fields. - test_every_registered_leg_has_published_detections had already refused these rows once, when they were added before the run. They land now, with the measured density they could not have had then. - PUBLISHED_LEGS keys grew a serving_path element. Six rows come from Vertex and two from a different account; a fixture that could not say which would be hiding the one caveat these numbers most need to carry. Also fixed a stale reference: docs pointed at tests/test_claude_annapolis_leg.py, renamed to test_claude_published_legs.py when it grew past annapolis. The reproduction instructions named a file that does not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjWxvSJybx4xJUaVTvGnR7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes item A of #156. Adds a first-party Anthropic serving path to
ClaudeDetector, runs both Fable ids on annapolis, and publishes/registers/documents the results.Why a second serving path
Vertex gates the whole Fable family behind a project-level publisher data-sharing setting we do not control. Probed 2026-09-04 and again 2026-09-05: both Fable ids return 403 there,
claude-opus-5returns 200, and a deliberately misspelledclaude-fable-5.1returns 404 — which is what proves the 403s resolved real ids behind a per-model gate rather than a blanket publisher block.Anthropic's first-party API serves both ids today, so these legs ran there: a different account and a different rate card for the same weights.
Results
Annapolis,
--claude-effort low, 125/125 panos each.claude-fable-5claude-fable-5-1claude-opus-5(low)Both displace
claude-opus-5, which had displacedgemini-3.1-pro-preview(0.567) — the first time a general-purpose model has beaten Opus on this split. RampNet still leads by 0.228.The two ids are the same detector at different operating points. 0.610 against 0.611 is nothing; 5.1 trades 0.061 recall for 0.058 precision and emits 0.44 fewer boxes/pano. Same shape as the effort finding in #123 and the Qwen 8B→32B inversion — the knob moves along the P/R curve, the ceiling does not move. Under this project's recall-first framing that argues for
claude-fable-5, the older id.#156's always-on-thinking cost premise is refuted. It predicted a band "wider than a flat 2x" because Fable cannot disable thinking. It is a flat 2x: ~33 thinking tokens/call on 5, ~0.35 on 5.1, against Opus-low's ~0.7. A 5-pano calibration predicted full-leg cost to within 1.5% on both ids.
The load-bearing design decision
The serving path is deliberately NOT in the detection cache key. It names which account answered, not what was asked. Putting it in
signature()would fragment the cache along an axis that does not change the answer, and would orphan $28.82 of already-paid annapolis detections the moment any leg moved.test_claude_cache_key_is_frozenstill passes on all four published Vertex legs, and a new test asserts the two paths hash identically.Provenance goes to the usage log and the published file's new
pinsfield instead —model_versionscannot stand in, because both paths report the bare model id.This makes
claude_serving_paththe first roster pin that is not a signature field, which is documented wherepinsis defined. If the two paths ever do diverge in output, that is a finding to measure, not a cache key to add quietly.Guards that caught real problems
publication_name's docstring warns about.leg_for()could not match a leg pinned on a knob the exporter had no flag for. Added and wired through.test_each_published_file_names_the_leg_it_says_it_isassumed every pin is a signature field. The envelope now carriespinsand the test looks there for pins the signature does not hold; legacy files never reach that branch.test_every_registered_leg_has_published_detectionsrefused the roster rows when I added them before the run. They land here with measured density.PUBLISHED_LEGSkeys grew a serving-path element: six rows come from Vertex and two do not, and a fixture that could not say which would hide the caveat these numbers most need to carry.Also fixes a stale pointer — the docs told readers to run
tests/test_claude_annapolis_leg.py, renamed totest_claude_published_legs.pysome time ago.Caveats that travel with the numbers
docs/model_comparison.md. Whether the two paths return identical detections for one model id is untested; nothing here depends on it, since no model was run on both.scripts/analysis/vertex_usage.pyrecovers server-side spend for Vertex and has no first-party equivalent, so for these legs the committed ledger is the only record.standing=False, excluded from pooled tables.Not done, deliberately
The expansion is not taken. Both legs clear the pre-registered 0.567 gate, so it is now a live decision rather than a hypothetical — ~$155/id for
manual_goldalone, ~$390/id for all remaining splits. Priced in the doc, awaiting a call. Item B (GPT-6 Astra) is untouched and independent.Verification
pytest -q— 1,495 passed, 1 skipped. Both new rows recompute from committed detections with no API key, no GPU and no.model_cache.export_model_cache.py --verifyreports both legs score identically to the cache.Total spend: $41.01, all recorded in
analysis_out/usage_log.jsonl.🤖 Generated with Claude Code (claude-opus-5[1m])