研究库 论文知识库

内容库 / 论文

arXiv:2609.13141 · 工程化

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

类型：arxiv
标识：2609.13141
链接：https://arxiv.org/abs/2609.13141
主分类：engineering
形态：method
TLDR：Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods usually use a lightweight selector to score context units, followed by hard Top-K selection that blocks gradients from the language modeling loss. Consequently, these methods commonly distill layer-wise dense attention distributions. Although this encourages the selector to rank context units by dense attention weights in the original model, the ranking is not directly aligned wi
待LLM分类：否
来源文件：
/inbox/tom/_candidates/2026-09-14-agent-rag-longcontext-candidates.json

相关 · 按语义匹配

笔记 Cool Papers · cs.IR (papers.cool) · RSS 摘要

笔记 Tom 文献雷达 · AI Agent / RAG / 长上下文 · 2026-08-03 晚间

笔记 Tom 文献雷达 · AI Agent / RAG / 长上下文 · 2026-07-11 下午版

文档 Stephen 自测 · R62 · 2026-09-01

攻略 Mamba-3：SSM 与线性注意力混合最强线性模型 · 干货攻略

仓库 griver/Q-RAG

arXiv

2609.13141

TLDR

Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods usually use a lightweight selector to score context units, followed by hard Top-K selection that blocks gradients from the language modeling loss. Consequently, these methods commonly distill layer-wise dense attention distributions. Although this encourages the selector to rank context units by dense attention weights in the original model, the ranking is not directly aligned wi

相关解读

arXiv ↗ 中文全文