Repository navigation
feat(billing): price cache-write (cache-creation) tokens #432
Description
Summary
cache_write_input_per_m exists in the model config: it is deserialized, validated, documented and present in the example config. Nothing prices it. This issue proposes wiring it into the billing path, so a model that configures a cache-write price actually charges for cache-creation tokens.
CONTRIBUTING.md asks that a large change be discussed in an issue before it is built, which is why this is a proposal rather than a pull request.
What the repository says today
config/config.example.toml section 4 (model catalogue) states that the field is expressed but not billed. Translated from the Chinese original:
Cache-write price (rant 2026-10-16T14:08:14): optional
cache_write_input_per_m(per million tokens), default 0 means cache writes are not billed, as today. This field currently only expresses, it does not bill: there is no path in the upstream response to obtain the cache-write token count (gatewayparse_usage/ sse do not produce it), so wiring it into billing needs another usage parse and is a later change.
src/config.rs:472-479 carries the same note on the field itself, and src/config.rs:541-548 rejects a negative value.
One correction to that note
The note's conclusion is right; its second clause is too strong. The count already arrives and is already deserialized — what is missing is a billing path, not a parse path:
src/sse.rs:164—StreamUsagealready declarescache_creation_input_tokens: Option<u32>.src/sse.rs:585—build_anthropic_usage_jsonreads it.src/protocol.rs:1217— the Chat/Responses to Anthropic conversion reads it out of the JSON body.
Both conversion sites use it for shape only: each subtracts it from prompt_tokens with saturating_sub, so an Anthropic-shaped usage does not report cache writes as ordinary input, and then writes cache_creation_input_tokens back into the converted payload. Nothing downstream of those two functions reads it.
sse::extract_cache_read_tokens extracts the read side only, so "sse does not produce it" holds for the extract_* helpers but not for StreamUsage itself.
Stated precisely, the gap is: the cache-creation token count reaches the process and is discarded before settlement.
Where it is dropped
src/gateway.rs:58-120—parse_usagereturns a 3-tuple(input, cached, output); its Anthropic branch readsinput_tokens,cache_read_input_tokensandoutput_tokens, but notcache_creation_input_tokens.src/gateway.rs:839—parse_anthropic_usage, used for the streamingmessage_startevent, likewise returns(input, cached).src/gateway.rs:736—UsageCapturekeeps the same three-way split.src/gateway.rs:161—settle_usage(input_tokens, cached_tokens, output_tokens)writestransactions,usage_recordsandkeys.usedfrom those three numbers.src/billing.rs:44—raw_cost(input_tokens, cached_tokens, output_tokens, input_per_m, cache_hit_input_per_m, output_per_m)has no cache-write operand.calc_points(src/billing.rs:58) wraps it, and the only other caller issrc/gateway.rs:218, insidesettle_usage.
Storage has no room for it either
- The
modelstable (src/db.rs) holds onlyprovider,model,currency,input_per_m,output_per_m,updated_at. Every other price column —cache_hit_input_per_m, thepeak_*triple,tiers_json— arrives additively throughensure_column(src/db.rs:469), anddao::get_model_price(src/dao.rs:202) selects them explicitly. Socache_write_input_per_mis absent from the database even where config sets it: theseed_modelsINSERT list (src/db.rs:633) does not carry it. usage_recordsandtransactionseach storetokens,cached_tokens,output_tokens, and no cache-write column, so even a correct charge would not be auditable per row.
Proposed scope
- Storage —
ensure_column("models", "cache_write_input_per_m", ...), extend theseed_modelsINSERT and itsON CONFLICT DO UPDATElist, and add the column todao::get_model_price's SELECT. Add a cache-write token column tousage_recordsandtransactionsso the charge can be audited. - Carry the count — extend
parse_usage,parse_anthropic_usage,UsageCaptureandsettle_usagewith a fourth count.sse.rsalready holds the field; the OpenAI-family spellings underprompt_tokens_detailsneed the same treatmentcached_tokensalready gets. - Price it — extend
raw_costandcalc_pointswith a cache-write operand, priced atcache_write_input_per_mfor the tier selected bybilling::select_tier. - Tests — a
raw_costgold case with a non-zero cache-write price; asettle_usagedatabase test asserting both the charge and the stored column; an idempotency test alongside the existing settlement ones.cargo test,cargo fmt --checkandcargo clippy --all-targetsgreen.
Questions for a maintainer
- Tiers.
ModelTier(src/config.rs) carriesinput_per_m,cache_hit_input_per_mandoutput_per_m. Should a tier also be able to carry a cache-write price, or is the model-level value enough for the four models that currently have tiers (gemini-3.1-pro,grok-4.6,grok-4.7,minimax-m3)? - Fallback when unset. The default stays 0, meaning free — today's behaviour, so no existing bill moves. Is that the intent, or should an unset cache-write price fall back to
input_per_mon the grounds that some providers charge cache writes above the input rate? - Schema version. Is one column on
modelsplus two on the usage tables worth aSCHEMA_VERSIONbump, or does the additiveensure_columnpath stay unbumped, as the earlier price columns did?
I am glad to implement this along the lines above, but I would rather not change the settlement path on a guess. A word on those three questions and I will open the PR. If the scope is too large for a single change, I will split it into three — storage, then carrying the count, then pricing — as separate PRs.
Activity
argszero commentedon Oct 23, 2026
I'd like to work on this. I will wait for a word on the three questions above before touching the settlement path; if the answers are not in yet, I will start with step 1 (storage), which is purely additive and moves no existing bill.
argszero commentedon Oct 23, 2026
Answering the three questions myself, since the proposal's author and the implementer are the same account here. The short version: the scope is the Anthropic path only, and the change must not move a single existing bill.
1. Tiers — no, model-level only. The four models that carry
[[models.tiers]]today aregemini-3.1-pro,grok-4.6,grok-4.7andminimax-m3. None of them is the path that emitscache_creation_input_tokens, so a per-tier cache-write price would be config that nothing can ever charge — a field with no writer, which is the exact defect this issue exists to remove.ModelTierstays untouched. A per-tier cache-write price becomes real work only if a tiered provider starts billing cache writes; until then the field would be decorative.2. Unset means free — the default stays 0, with no fallback. Falling back to
input_per_mwould silently reprice every existing deployment that never opted in: config that says nothing would start charging more than it does today. 0 keeps the invariant this change is judged by — no existing bill moves — and a deployer who wants the charge sets the price. Note this is not merely the conservative choice: some providers charge cache writes above the plain input rate, so a fallback would also be wrong in the cases it appeared to help.3. No
SCHEMA_VERSIONbump. The earlier price columns (cache_hit_input_per_m, thepeak_*triple,tiers_json) all arrived as additiveensure_columncalls without a bump, and the new columns are the same shape: one onmodels, plus the token count onusage_recordsandtransactions. The version bump has been reserved for shapes an additive column cannot express.Implementation follows in one PR rather than three. The reason is testability, not convenience: a column with no writer cannot be tested, and a charge with no column cannot be audited, so splitting the wire would leave two of the three pieces unverifiable. Scope is the Anthropic path (
usage.cache_creation_input_tokens) on both the non-streaming and streaming parsers. The OpenAI-family spellings are left alone: if a provider does not send a field, the count is 0 rather than a guess.test(billing): pin both sides of the settlement guard
feat(billing): bill cache-write (cache-creation) tokens (#433)
argszero commentedon Oct 23, 2026
Implemented in #433, merged as
357ab01. Recording what is now billed and what is deliberately still free, so this does not have to be rediscovered later.Now billed. Anthropic cache creation.
usage.cache_creation_input_tokensreaches settlement from both parsers (parse_usagefor non-streaming,UsageCapture/parse_anthropic_usagefor streaming) and is priced at the model'scache_write_input_per_m, which now exists as a column onmodelsalongside the other price columns. The count is stored onusage_recordsandtransactionsso a charge can be audited per row. Three boundaries are structural, not omissions:ModelTiercarries no cache-write price, so tier selection cannot change it;peak_*has no cache-write variant, so peak hours cannot either; and the default price stays 0 with no fallback toinput_per_m, so a deployment that never sets the field bills exactly what it billed before.Still free, and why that is the end of it. Nothing in the OpenAI-family shape we parse carries a separate cache-write count.
prompt_tokens_detailsmodelscached_tokensand nothing else insse::PromptTokensDetails, andprompt_tokens_details.cache_creation_tokensis not a field any of them emits. The one catalogue provider that does distinguish cache writes is DeepSeek, and its spelling needs no new field:prompt_tokens = prompt_cache_hit_tokens + prompt_cache_miss_tokens, so the miss segment is the write, andparse_usagealready bills it —prompt_tokens - cachedis priced atinput_per_m. That segment is not free today; it is billed at the input rate, which is the rate DeepSeek charges for it.So there is no OpenAI-family follow-up to file. If a provider starts emitting a distinct cache-write count, it belongs on this thread and the fix is the same shape as this one: carry the count, price it, store it.
Two things this change leaves open, named rather than hidden.
openai_chat, outboundanthropic) records usage throughsse::UsageSlot, which isArc<Mutex<Option<(f64, f64, f64)>>>— three counts by construction — so cache writes on that path bill at 0. Settlement never parses a converted body (it reads the raw upstream bytes beforetransform_response, or the slot), so this cannot double count; it simply does not count yet.src/routes/admin_models.rscannot set the new price, so a model created through the admin API prices cache writes at zero. Config is the declared single source for the catalogue andseed_modelsoverwrites runtime edits on restart, so this matches howpeak_*already behaves — but it is a real gap in the admin surface, and it is the next thing worth doing rather than a docs line.An unrelated defect found while writing the tests, not fixed here:
UsageCapture::finishapplies the OpenAI-shaped(input - cached)correction to Anthropic too, while Anthropic'sinput_tokensalready excludes cache reads. Streaming Anthropic therefore under-counts uncached input — 0 where the non-streaming parser reports 100 on the same payload. Fixing it raises existing bills, so a test pins the current behaviour and it needs its own decision.