Repository navigation
LOU-W1: token estimation and a model registry (context window + pricing) - #45
Merged
Merged
Conversation
Adds src/models/ (exported from src/index.ts): estimateTokens (dependency-free heuristic: ~chars/4 for Latin, higher ratios for CJK/other scripts and emoji, counted by code point, plus per-message overhead and tool-call JSON), setTokenEstimator / options.estimator for plugging in a real tokenizer, getModelInfo / registerModel (exact id, provider/ prefix, then dated-snapshot prefix match), and estimateCost (undefined when unknown). The built-in table only contains models whose context window and prices were read from OpenAI, Anthropic and Ollama pages; retrieval date and sources are recorded in modelData.ts. Not wired into the executor yet. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New
src/models/module, exported fromsrc/index.ts:estimateTokens,setTokenEstimator/options.estimator,getModelInfo,registerModel,estimateCost. Docs: "Models, tokens and cost" section indocs/api-overview.md. Not wired into the executor.Why
Foundation for context compaction, budgets and cost accounting: small, dependency-free, overridable.
How verified
npx tsc --noEmit: cleannpm run lint: 0 errors (432 pre-existing warnings, none insrc/models)npm run build+ create-loushy-agent build: successnpm run test:coverage: 95 files passed, 1163 tests passed, 3 skippednpm run fallow: exit 0npm run docs:verify-snippets: all 30 snippets type-check, 5 run cleanlyorigin/mainat 13a6131 before verification (already up to date).Notes for reviewer
src/models/modelData.ts) only includes models read from OpenAI model/pricing pages, Anthropic's models overview and Ollama library pages. Ollama entries have no prices (local), soestimateCostreturnsundefined. Qwen2.5 was omitted because its context window was ambiguous. OpenAI pages were read via a summarizing fetch tool, so a quick spot-check of numbers is worthwhile.-2024-07-18,-latest,:tag,@date, soo3-minidoes not resolve too3andgpt-5.4does not resolve togpt-5.Message.contentis typedstring; non-string content is JSON-stringified defensively.modeloption is only forwarded to custom estimators; the built-in heuristic is model-independent.🤖 Generated with Claude Code