Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

• 类型：arxiv
• 标识：2609.17652
• 链接：https://arxiv.org/abs/2609.17652
• 主分类：llm-infra
• 形态：method
• TLDR：When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each key channel to read. The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly the channel's t-bit quantizer, and the query spends its bit budget by reverse water-filling over the variance-weighted importance of its channels. At one million tokens
• 待LLM分类：否
• 标题中文：Fathom：面向卸载 KV 缓存稀疏解码的每查询读取深度
• TLDR中文：当 Agent 会话运行到百万级 token、且同时存在大量会话时，KV 缓存及其排序索引位于主存中，而为 top-k 步骤排序全部 n 个键的扫描成为制约解码的流量。我们提出 Fathom，一种键扫描机制，其中每个查询自行决定对每个键通道读取多少比特。4-bit K 缓存以通道为主序、按比特平面存储，因此 t 个平面的前缀恰好是该通道的 t-bit 量化器，查询通过在通道的方差加权重要性上进行反向注水分配来花费其比特预算。在百万级 token 下...
• 来源文件：
• /inbox/tom/_candidates/2026-09-17-agent-rag-longcontext-candidates.json