Repository navigation
Cache design: content-addressed local SQLite store #128
Description
Activity
Thanks for the issue! I think this is a nice and reasonable fit.
Does
input_hashhashed from the entire input text to the judgesystem prompt + user prompt + two annotations? Also sinceannotationsandcompletionsare, on a high level, quite similar perhaps we can do something likeCREATE TABLE responses ( input_hash TEXT PRIMARY KEY, input_text TEXT NOT NULL, completion TEXT NOT NULL, top_logprobs TEXT, -- JSON created_at TEXT NOT NULL, created_by TEXT NOT NULL ); CREATE TABLE completion_metadata ( input_hash TEXT PRIMARY KEY REFERENCES responses(input_hash), benchmark TEXT NOT NULL, instruction_id TEXT NOT NULL, model TEXT NOT NULL, run_id TEXT NOT NULL ); CREATE TABLE judgement_metadata ( input_hash TEXT PRIMARY KEY REFERENCES responses(input_hash), benchmark TEXT NOT NULL, instruction_id TEXT NOT NULL, model_a TEXT NOT NULL, model_b TEXT NOT NULL, judge TEXT NOT NULL, orientation TEXT, run_id TEXT NOT NULL );
but we should also be sure that our
judgementis not specific to pairwise (in my opinion). It should include both multi-turn conversation, should be suitable forpointwisejudgements (not necessary but may be, do we really needmodel_bnot the be NULL.@kargibora Thanks!
Does
input_hashhashed from the entire input text to the judgesystem prompt + user prompt + two annotations?Yes,
input_hashis the SHA-256 of the rendered input we send to the backend. For chat judges that is the message list (system + user), so the instruction and both completions are included. For text backends it is the flattened prompt.I think we should keep completions and judgements in separate folders, so we can inspect, delete, or sync one role without touching the other. If we do that, a shared
responsestable would not actually be shared, each folder would still have its own db. Splittingresponsesfrom the metadata tables would help if one cached output could have several associations (e.g. severalrun_ids), however I don't think that is realistic to happen often in practice. So I'm not sure if separting the metadata table and sharing the responses table would buy us much.On pairwise, I agree making
model_bnullable is a small change, we can do that now so pointwise can usemodel_aonly. Multi-turn should already work, since we hash the full rendered conversation.What do you think?
Reacted by Bora Kargı
Context
Follow-up to #36. The langchain cache does not work with vLLM and gives little control, and the earlier drafts mixed identity keys with content keys in different ways. There is now a lean draft stack on top of #118 (#121–#124, #127) that tries to settle the local cache design. I would like to walk through the current approach and get feedback.
What we do
We cache both completions and judgements in separate SQLite trees under an optional
--run.store_root. The cache boundary isdo_inference. The model is prepared lazily, so a full hit does not load the backend.Rows are keyed by a hash of the rendered input, and live under a folder keyed by the validated model descriptor (sampling, max tokens, thinking settings, chat template, output-affecting engine settings). Completions and judgements stay in separate role roots so the same model string as battle model and as judge cannot collide and thus they have separate metadata files. Typed columns (
benchmark,instruction_id,model/model_a/model_b,orientation, …) are kept for inspection and targeted deletion, but we don't use them as key anymore. Prompt-construction options whose effect already appears in the rendered messages are captured by the input hash. Parsing and aggregation stay outside the raw-output cache.We additionally add a Hugging Face fetch/push as a separate CLI in #127.
Layout
metadata.jsonstores the descriptor.Schema
top_logprobsis there so cached judgements can restore the first-token logprobs used by the #118 parsers.orientationis direct/reversed relative to the source model order when that applies.model_a/model_bmatch the positional slots in the rendered judge prompt.Why this shape
Identity keys alone miss cases where the same battle is judged with a different rendered prompt or different model settings. Content keys alone don't provide easy filtering of the rows. The current shape keeps content-addressed hits, and the other fields for filtering and targeted deletion of the cache. VLLM execution-only settings (
tensor_parallel_size,gpu_memory_utilization,enforce_eager) stay out of the descriptor.Any thoughts on this? cc: @kargibora