Describe the bug
minference_patch_kv_cache_cpu (patch.py) monkeypatches transformers.cache_utils.DynamicCache.update/.get with cpu_cache_update/cpu_cache_get. Both read and write self.key_cache, self.value_cache and self._seen_tokens/self.seen_tokens directly. Current transformers moved DynamicCache's per-layer state onto self.layers and no longer defines those attributes, so the first cache update after kv_cache_cpu=True raises AttributeError.
Same root cause as #207 (fixed for DynamicCacheWithRepeat in #216), but a different call site that fix didn't touch. kv_cache_cpu is a documented MInferenceConfig option used across the example/benchmark scripts, not a corner case.
Steps to reproduce
No vllm or model load needed, this extracts the two patched functions and runs them against a real DynamicCache:
import torch
from transformers.cache_utils import DynamicCache
# cpu_cache_update copied verbatim from minference/patch.py
cache = DynamicCache()
hasattr(cache, "key_cache") # False
cpu_cache_update(cache, torch.randn(1, 2, 3, 4), torch.randn(1, 2, 3, 4), layer_idx=0)
Expected Behavior
The cache update should succeed (or the patch should adapt to the cache layout transformers actually ships).
Logs
AttributeError: 'DynamicCache' object has no attribute 'seen_tokens'
Same AttributeError from cpu_cache_get.
Additional Information
Confirmed on current main (29ef197) against transformers==5.17.0. Happy to send a PR if useful.
Describe the bug
minference_patch_kv_cache_cpu(patch.py) monkeypatchestransformers.cache_utils.DynamicCache.update/.getwithcpu_cache_update/cpu_cache_get. Both read and writeself.key_cache,self.value_cacheandself._seen_tokens/self.seen_tokensdirectly. CurrenttransformersmovedDynamicCache's per-layer state ontoself.layersand no longer defines those attributes, so the first cache update afterkv_cache_cpu=TrueraisesAttributeError.Same root cause as #207 (fixed for
DynamicCacheWithRepeatin #216), but a different call site that fix didn't touch.kv_cache_cpuis a documentedMInferenceConfigoption used across the example/benchmark scripts, not a corner case.Steps to reproduce
No vllm or model load needed, this extracts the two patched functions and runs them against a real
DynamicCache:Expected Behavior
The cache update should succeed (or the patch should adapt to the cache layout
transformersactually ships).Logs
Same
AttributeErrorfromcpu_cache_get.Additional Information
Confirmed on current main (29ef197) against transformers==5.17.0. Happy to send a PR if useful.