Add with_prompt_cache_key for provider-side prompt caching - #53
Merged
Merged
Conversation
OpenAI's `prompt_cache_key` routes requests sharing a key to the same cache, which is what makes long-prefix cache hits reliable. On GPT-5.6+ it is required for dependable matching — without it a shared system prompt re-tokenizes at full price on most requests. Excluded from the local response-cache key, like service_tier: it is a routing hint that cannot change response content, so setting or renaming a key never invalidates cached responses. Both the current and legacy cache-key functions strip it, so existing caches keep hitting. Motivated by yap's clean-nlp-data, where every Japanese request shares a ~10k token tokenization policy as its system prompt. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FgmaWYGPXGHGermM2JKUvQ
anchpop
added a commit
to yaptown/yap
that referenced
this pull request
Jul 31, 2026
The previous commit carried a [patch] pointing tysm at ../tysm, which built here and nowhere else. The change it needed — with_prompt_cache_key — is now pushed as anchpop/tysm#53, so the dependency pins that rev directly and the patch section is gone. Drop the `rev` once the PR merges. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FgmaWYGPXGHGermM2JKUvQ
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSrT63cKE2XK6JMt9YCFjj
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSrT63cKE2XK6JMt9YCFjj
anchpop
added a commit
to yaptown/yap
that referenced
this pull request
Aug 1, 2026
with_prompt_cache_key landed on tysm main as anchpop/tysm#53 and shipped in 0.20.0, so the pin is no longer needed. The rev we were on diffs empty against tysm's main, so this is a no-op in terms of code. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FgmaWYGPXGHGermM2JKUvQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ChatClient::with_prompt_cache_key, exposing OpenAI'sprompt_cache_key.Requests sharing a key are routed to the same cache, which is what makes long
prefix hits reliable. On GPT-5.6+ setting it is required for dependable
matching — without it a shared system prompt re-tokenizes at full price on most
requests.
Cache behaviour: the key is stripped from the local response-cache key, the
same way
service_tieralready is. It's a routing hint that can't changeresponse content, so setting or renaming a key never invalidates cached
responses. Both
cache_keyandlegacy_cache_keystrip it, so existing cacheskeep hitting.
Motivation: yap's
clean-nlp-datasends a ~10k-token tokenization policy asthe system prompt on every Japanese request; without a stable key the prefix
cache misses often enough to matter.
Verified by using it to generate data —
cargo checkis clean and the 11offline tests pass. The 5 failing tests are pre-existing and environmental
(
OpenAiApiKeyError(NotPresent)— they make live API calls).🤖 Generated with Claude Code
https://claude.ai/code/session_01FgmaWYGPXGHGermM2JKUvQ