Block Sparse Attention with Log-Linear Complexity
- 类型：arxiv
- 标识：2609.31093
- 链接：https://arxiv.org/abs/2609.31093
- 主分类：engineering
- 形态：method
- TLDR：Scaling language models to long contexts is limited by the quadratic cost of self-attention. Block sparse attention offers an efficient alternative, but selecting the retained blocks remains a bottleneck. Conventional block selection requires scoring all query-block pairs and therefore remains quadratic in sequence length. To address this issue, we propose PISA, a block-sparse attention mechanism that employs a pyramid Top-K selection strategy. The main idea is to gradually narrow down the candidates across different levels, making it more efficient to find the most relevant keys. Specifically
- 待LLM分类：是
- 来源文件：
- /inbox/tom/_candidates/2026-09-28-agent-rag-longcontext-candidates.json