Repository navigation
LOU-V5: usage and cost accounting on every run - #62
Merged
Merged
Conversation
Every ExecutionResult now carries usage: RunUsage (input/output/total tokens, costUsd, modelCalls, estimated flag, byModel breakdown, delegated roll-up) plus per-call stepUsage. A provider that reports no usage yields undefined (the ai SDK's NaN counts included) and the executor falls back to estimateTokens, flagging the run estimated. costUsd is undefined if any model used is unpriced, never a partial sum. Delegated child runs roll up into the parent totals and byModel and are also shown as usage.delegated. Checkpoints and approval snapshots store the running usage so resumed runs continue from it; older checkpoints load with their token counts and unknown cost. The finish events, onLLMResponse and the chat span read the same normalized numbers. formatUsage() gives a one-line summary. mockModel now reports no usage when a turn omits it (it used to report zeros). promptTokens/completionTokens stay on RunUsage as deprecated aliases. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…syncLocalStorage node:async_hooks is not available in workerd, so importing it on the AgentExecutor path broke the Cloudflare Worker bundle (wrangler dev: No such module async_hooks). The executor now hands the delegate tool an onDelegatedUsage callback via the tool execute options. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…done and run.done Streams that report no usage now resolve to undefined (StreamResult.usage, StreamChunk.usage) so the executor estimates instead of counting zeros. step.done and run.done carry inputTokens, outputTokens, estimated and costUsd next to the existing promptTokens/completionTokens. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
LOU-V5 made GenerateResult.usage optional; cassettes now record and replay 'no usage' faithfully instead of assuming it is always present. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every
ExecutionResultcarriesusage: RunUsage(inputTokens,outputTokens,totalTokens,costUsd | undefined,modelCalls,estimated,byModel, optionaldelegated, optional cache/reasoning totals) andstepUsage: StepUsage[]. New publicUsage,RunUsage,StepUsage,CallUsage,normalizeUsage,formatUsage(fromsrc/models),ProviderUsage(from providers). Docs: "Usage and cost of a run" under "Models, tokens and cost" indocs/api-overview.md, plus a README feature bullet.Why
The audit found no token/cost accounting on results; LOU-V6 (budgets) builds on these totals.
How
src/models/usage.ts: types,normalizeUsage,formatUsage.src/execution/runUsage.ts: per-call measurement (reported, elseestimateTokensflaggedestimated), accumulation, delegation roll-up (AsyncLocalStorage sink, so no tool-result plumbing), checkpoint restore.GenerateResult.usageis now optional;aiSdkProviderreturnsundefinedfor the SDK's NaN counts and reads cache/reasoning tokens fromproviderMetadatawhen documented keys are present.costUsdisundefinedif any contributing model is unpriced (no partial sum); per-model costs are recomputed from tokens so they stay consistent after merges.unknownmodel, cost undefined). Approval pause/resume also carries usage (ExecutionSnapshot.usage,ExecuteOptions.initialUsage).finish/abortevents carry theRunUsage;text-completehasstepUsage;onLLMResponsegets the call usage as a 3rd arg; thechatspan reads the same numbers (loushy.usage.estimatedwhen estimated).AgentExecutor.ts: the step loop moved verbatim intorunSteps(so it can run inside the usage sink), plus arecordStepcall where the step result is recorded.How verified
npx tsc --noEmit: clean.npm run lint: 0 errors (433 warnings, none new in touched code).npm run buildand the create-loushy-agent build: success.995f649): 117 files passed, 1648 tests passed, 4 skipped;npm run fallowexit 0.npm run test:types: no errors.docs:llms:checkok;docs:verify-snippets: all 59 snippets type-check, 6 run.cloudflare.test.ts > wrangler dev serves the workerfails in my Windows environment (needs the wrangler CLI; unrelated), so the coverage run for fallow excluded that one file.guardrails.test.tstimeout test failed once under load and passed on rerun.Notes for reviewer
mockModelwith nousagenow reports none (it reported zeros), so executor runs estimate;mockModel.test.tsexpectation updated accordingly. Streaming types still require usage, so a usage-less mock turn streams zeros.RunUsagekeepspromptTokens/completionTokensas deprecated aliases so existing consumers (evalsbudget, flows) work unchanged.{ promptTokens, completionTokens, totalTokens }to the model, not the breakdown.recordStepand run insidecollectDelegatedUsageto get the same accounting.🤖 Generated with Claude Code