fix: allow k,v combo values for draft_cache_mode - #483
Merged
turboderp merged 1 commit intoSep 20, 2026
Merged
Conversation
draft_cache_mode was typed as Optional[CACHE_SIZES], which only accepts the legacy literals (FP16/Q8/Q6/Q4). CACHE_TYPE (already used by the non-draft cache_mode field) additionally accepts the "k_bits,v_bits" combo form, e.g. "4,3". config_sample.yml's description for draft_cache_mode already documented the combo form, and DraftModel.create_cache() in backends/exllamav3/model.py already parses it via the same create_cache() used for the main cache -- only the Pydantic validation on draft_cache_mode rejected it before that code ever ran. Fixes theroyallab#482 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
draft_cache_mode: 4,3inconfig.ymlfails to start the server, even though the field's own generated documentation (config_sample.yml) says thek_bits,v_bitscombo form is supported — the same formcache_modealready accepts.Root cause
common/config_models.py:cache_modeis typedOptional[CACHE_TYPE](accepts both the legacy literals and the"k,v"combo).draft_cache_modewas typedOptional[CACHE_SIZES]— only the legacy literals — so Pydantic validation rejects"4,3"before the value ever reachesDraftModel.create_cache()inbackends/exllamav3/model.py, which already parses the combo form via the samecreate_cache()helper used for the main cache.Fix
Change
draft_cache_mode's annotation fromCACHE_SIZEStoCACHE_TYPE, matchingcache_mode, and update its description to mention the combo form (mirroringcache_mode's existing description).Testing
python3 -c "import ast; ast.parse(open('common/config_models.py').read())"— syntax check passes.config_sample.yml's existing (already-committed) description text fordraft_cache_modealready documented thek_bits,v_bitsform, and tracedbackends/exllamav3/model.py'screate_cache()to confirm it parses"k,v"generically for both the main and draft cache — so no runtime changes are needed, only the validation type.Fixes #482
🤖 Generated with Claude Code