Repository navigation
feat: prompt caching support and fix per-key usage tracking - #1
Merged
Merged
Conversation
semidark
commented
Apr 23, 2026
semidark
commented
Owner
- Add docs/PROMPT_CACHING.md: comprehensive guide with validated test results (93-97% cache hit rate on gpt-4.1), best practices, KQL monitoring queries, and cost impact analysis
- Add infra/test-prompt-caching.sh: automated validation script that verifies prompt_cache_key and prompt_cache_retention work through the proxy
- Fix infra/usage_callback.py: usage was being logged under the Azure provider API key hash instead of the client Bearer token hash. Switch to litellm_params.metadata.user_api_key (LiteLLM's internal hash of the client key) as the primary key identifier; retain _hash_key(kwargs[api_key]) only as last-resort fallback
- Update scripts/usage-report.py: add Cached and Cache % columns to both key and model views; add --debug/-v flag to make KQL query output optional; remove duplicate code block; fix column index alignment; fix cost field key (CostNorm -> Cost)
- Update README, docs/ARCHITECTURE.md, docs/DEPLOYMENT_SUMMARY.md, docs/USAGE_ANALYSIS.md with prompt caching guidance and cache hit rate KQL queries
added 2 commits
April 23, 2026 12:55
- Add docs/PROMPT_CACHING.md: comprehensive guide with validated test results (93-97% cache hit rate on gpt-4.1), best practices, KQL monitoring queries, and cost impact analysis - Add infra/test-prompt-caching.sh: automated validation script that verifies prompt_cache_key and prompt_cache_retention work through the proxy - Fix infra/usage_callback.py: usage was being logged under the Azure provider API key hash instead of the client Bearer token hash. Switch to litellm_params.metadata.user_api_key (LiteLLM's internal hash of the client key) as the primary key identifier; retain _hash_key(kwargs[api_key]) only as last-resort fallback - Update scripts/usage-report.py: add Cached and Cache % columns to both key and model views; add --debug/-v flag to make KQL query output optional; remove duplicate code block; fix column index alignment; fix cost field key (CostNorm -> Cost) - Update README, docs/ARCHITECTURE.md, docs/DEPLOYMENT_SUMMARY.md, docs/USAGE_ANALYSIS.md with prompt caching guidance and cache hit rate KQL queries
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.