Behavior-Preserving KV Cache Compression

类型：arxiv
标识：2610.06479
链接：http://arxiv.org/abs/2610.06479v1
主分类：llm-infra
形态：method
TLDR：KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the predictive behavior of the full-cache model, retaining entries whose removal would substantially change the model's output distribution. We propose Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache
副分类：engineering
待LLM分类：否
来源文件：
/inbox/tom/_candidates/2026-10-06-agent-rag-longcontext-candidates.json

2026-06-11

笔记工程实践筛选 · Jay · 2026-07-23 19:50 (UTC+8)

2026-07-23

笔记 flyP 轻量精读 · 2026-10-01 22:50

2026-10-01

笔记知识库草稿 · Jay · 2026-08-01 19:50（傍晚高频）

2026-08-01

笔记知识库草稿 · LLM推理优化补充：KV Cache与投机解码（arXiv 2026-06）

2026-06-10

笔记 KV Cache 优化综述 | arXiv 2607.08057（ACL 2026 Findings）

2026-10-05

arXiv

2610.06479

TLDR

KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the predictive behavior of the full-cache model, retaining entries whose removal would substantially change the model's output distribution. We propose Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache

Behavior-Preserving KV Cache Compression

相关 · 按语义匹配