Skip to content

LOU-V5: usage and cost accounting on every run - #62

Merged
LinuxDevil merged 11 commits into
mainfrom
lou-v5-usage-and-cost
Oct 1, 2026
Merged

LinuxDevil merged 11 commits into
mainfrom
lou-v5-usage-and-cost

Conversation

@LinuxDevil

Copy link
Copy Markdown
Owner

What

Every ExecutionResult carries usage: RunUsage (inputTokens, outputTokens, totalTokens, costUsd | undefined, modelCalls, estimated, byModel, optional delegated, optional cache/reasoning totals) and stepUsage: StepUsage[]. New public Usage, RunUsage, StepUsage, CallUsage, normalizeUsage, formatUsage (from src/models), ProviderUsage (from providers). Docs: "Usage and cost of a run" under "Models, tokens and cost" in docs/api-overview.md, plus a README feature bullet.

Why

The audit found no token/cost accounting on results; LOU-V6 (budgets) builds on these totals.

How

  • src/models/usage.ts: types, normalizeUsage, formatUsage. src/execution/runUsage.ts: per-call measurement (reported, else estimateTokens flagged estimated), accumulation, delegation roll-up (AsyncLocalStorage sink, so no tool-result plumbing), checkpoint restore.
  • GenerateResult.usage is now optional; aiSdkProvider returns undefined for the SDK's NaN counts and reads cache/reasoning tokens from providerMetadata when documented keys are present.
  • costUsd is undefined if any contributing model is unpriced (no partial sum); per-model costs are recomputed from tokens so they stay consistent after merges.
  • Resume: checkpoint stores the full running usage (old checkpoints: counts attributed to an unknown model, cost undefined). Approval pause/resume also carries usage (ExecutionSnapshot.usage, ExecuteOptions.initialUsage).
  • finish/abort events carry the RunUsage; text-complete has stepUsage; onLLMResponse gets the call usage as a 3rd arg; the chat span reads the same numbers (loushy.usage.estimated when estimated).
  • Edits in AgentExecutor.ts: the step loop moved verbatim into runSteps (so it can run inside the usage sink), plus a recordStep call where the step result is recorded.

How verified

  • npx tsc --noEmit: clean. npm run lint: 0 errors (433 warnings, none new in touched code).
  • npm run build and the create-loushy-agent build: success.
  • Coverage run (merged with main 995f649): 117 files passed, 1648 tests passed, 4 skipped; npm run fallow exit 0. npm run test:types: no errors. docs:llms:check ok; docs:verify-snippets: all 59 snippets type-check, 6 run.
  • Caveat: cloudflare.test.ts > wrangler dev serves the worker fails in my Windows environment (needs the wrangler CLI; unrelated), so the coverage run for fallow excluded that one file. guardrails.test.ts timeout test failed once under load and passed on rerun.

Notes for reviewer

  • Behaviour change: mockModel with no usage now reports none (it reported zeros), so executor runs estimate; mockModel.test.ts expectation updated accordingly. Streaming types still require usage, so a usage-less mock turn streams zeros.
  • RunUsage keeps promptTokens/completionTokens as deprecated aliases so existing consumers (evals budget, flows) work unchanged.
  • The delegate tool still returns the legacy { promptTokens, completionTokens, totalTokens } to the model, not the breakdown.
  • The streaming path (being added elsewhere) needs to call recordStep and run inside collectDelegatedUsage to get the same accounting.

🤖 Generated with Claude Code

LinuxDevil and others added 11 commits October 1, 2026 10:16
Every ExecutionResult now carries usage: RunUsage (input/output/total tokens, costUsd, modelCalls, estimated flag, byModel breakdown, delegated roll-up) plus per-call stepUsage. A provider that reports no usage yields undefined (the ai SDK's NaN counts included) and the executor falls back to estimateTokens, flagging the run estimated. costUsd is undefined if any model used is unpriced, never a partial sum.

Delegated child runs roll up into the parent totals and byModel and are also shown as usage.delegated. Checkpoints and approval snapshots store the running usage so resumed runs continue from it; older checkpoints load with their token counts and unknown cost. The finish events, onLLMResponse and the chat span read the same normalized numbers. formatUsage() gives a one-line summary.

mockModel now reports no usage when a turn omits it (it used to report zeros). promptTokens/completionTokens stay on RunUsage as deprecated aliases.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…syncLocalStorage

node:async_hooks is not available in workerd, so importing it on the AgentExecutor path broke the Cloudflare Worker bundle (wrangler dev: No such module async_hooks). The executor now hands the delegate tool an onDelegatedUsage callback via the tool execute options.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…done and run.done

Streams that report no usage now resolve to undefined (StreamResult.usage, StreamChunk.usage) so the executor estimates instead of counting zeros. step.done and run.done carry inputTokens, outputTokens, estimated and costUsd next to the existing promptTokens/completionTokens.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
LOU-V5 made GenerateResult.usage optional; cassettes now record and replay
'no usage' faithfully instead of assuming it is always present.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@LinuxDevil
LinuxDevil merged commit a298274 into main Oct 1, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant