Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

fix: update key cache access for compatibility with newer DynamicCache implementation (#2348)

chen, suyuef331bb024551
diff --git a/auto_round/experimental/kv_cache.py b/auto_round/experimental/kv_cache.py
index ba35c408e..5bee3e8c0 100644
--- a/auto_round/experimental/kv_cache.py
+++ b/auto_round/experimental/kv_cache.py
@@ -141,7 +141,9 @@ class QuantizedKVParameterCache(DynamicCache):
Returns the sequence length of the cached states.
A layer index can be optionally passed.
"""
- if len(self.key_cache) <= layer_idx:
+ # Transformers' newer DynamicCache stores cache state in `layers`, not `key_cache`.
+ layer_idx = 0 if layer_idx is None else layer_idx
+ if len(getattr(self, "key_cache", ())) <= layer_idx:
return 0
# since we cannot get the seq_length of each layer directly and
# rely on `_seen_tokens` which is updated every "layer_idx" == 0,