Skip to content

[Bug]: kv_cache_cpu patch reads retired DynamicCache attributes, AttributeError on current transformers #217

Description

@AmirF194

Describe the bug

minference_patch_kv_cache_cpu (patch.py) monkeypatches transformers.cache_utils.DynamicCache.update/.get with cpu_cache_update/cpu_cache_get. Both read and write self.key_cache, self.value_cache and self._seen_tokens/self.seen_tokens directly. Current transformers moved DynamicCache's per-layer state onto self.layers and no longer defines those attributes, so the first cache update after kv_cache_cpu=True raises AttributeError.

Same root cause as #207 (fixed for DynamicCacheWithRepeat in #216), but a different call site that fix didn't touch. kv_cache_cpu is a documented MInferenceConfig option used across the example/benchmark scripts, not a corner case.

Steps to reproduce

No vllm or model load needed, this extracts the two patched functions and runs them against a real DynamicCache:

import torch
from transformers.cache_utils import DynamicCache
# cpu_cache_update copied verbatim from minference/patch.py
cache = DynamicCache()
hasattr(cache, "key_cache")  # False
cpu_cache_update(cache, torch.randn(1, 2, 3, 4), torch.randn(1, 2, 3, 4), layer_idx=0)

Expected Behavior

The cache update should succeed (or the patch should adapt to the cache layout transformers actually ships).

Logs

AttributeError: 'DynamicCache' object has no attribute 'seen_tokens'

Same AttributeError from cpu_cache_get.

Additional Information

Confirmed on current main (29ef197) against transformers==5.17.0. Happy to send a PR if useful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions