Skip to content

fix: allow k,v combo values for draft_cache_mode - #483

Merged
turboderp merged 1 commit into
theroyallab:mainfrom
dripston:fix/482-draft-cache-mode-combo
Sep 20, 2026
Merged

turboderp merged 1 commit into
theroyallab:mainfrom
dripston:fix/482-draft-cache-mode-combo

Conversation

@dripston

Copy link
Copy Markdown
Contributor

Problem

draft_cache_mode: 4,3 in config.yml fails to start the server, even though the field's own generated documentation (config_sample.yml) says the k_bits,v_bits combo form is supported — the same form cache_mode already accepts.

Root cause

common/config_models.py:

CACHE_SIZES = Literal["FP16", "Q8", "Q6", "Q4"]
CACHE_TYPE = Union[CACHE_SIZES, constr(pattern=r"^[2-8]\s*,\s*[2-8]$")]

cache_mode is typed Optional[CACHE_TYPE] (accepts both the legacy literals and the "k,v" combo). draft_cache_mode was typed Optional[CACHE_SIZES] — only the legacy literals — so Pydantic validation rejects "4,3" before the value ever reaches DraftModel.create_cache() in backends/exllamav3/model.py, which already parses the combo form via the same create_cache() helper used for the main cache.

Fix

Change draft_cache_mode's annotation from CACHE_SIZES to CACHE_TYPE, matching cache_mode, and update its description to mention the combo form (mirroring cache_mode's existing description).

Testing

  • python3 -c "import ast; ast.parse(open('common/config_models.py').read())" — syntax check passes.
  • Confirmed config_sample.yml's existing (already-committed) description text for draft_cache_mode already documented the k_bits,v_bits form, and traced backends/exllamav3/model.py's create_cache() to confirm it parses "k,v" generically for both the main and draft cache — so no runtime changes are needed, only the validation type.
  • No local exllamav3/CUDA environment available to run the server end-to-end; this is a type-annotation-only change with no behavioral code path changes beyond widening what Pydantic accepts.

Fixes #482

🤖 Generated with Claude Code

draft_cache_mode was typed as Optional[CACHE_SIZES], which only
accepts the legacy literals (FP16/Q8/Q6/Q4). CACHE_TYPE (already used
by the non-draft cache_mode field) additionally accepts the
"k_bits,v_bits" combo form, e.g. "4,3".

config_sample.yml's description for draft_cache_mode already
documented the combo form, and DraftModel.create_cache() in
backends/exllamav3/model.py already parses it via the same
create_cache() used for the main cache -- only the Pydantic
validation on draft_cache_mode rejected it before that code ever ran.

Fixes theroyallab#482

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@turboderp
turboderp merged commit 4f7f7e9 into theroyallab:main Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Not all quants possible with draft_cache_mode [BUG]

2 participants