研究库 论文知识库

内容库 / 论文

arXiv:2610.13735 · 工程化

Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

类型：arxiv
标识：2610.13735
链接：http://arxiv.org/abs/2610.13735v1
主分类：engineering
形态：method
TLDR：Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling. As a unified module, it dynamically switches its computation pattern across different phases of sequence processing, enabling lin
待LLM分类：是
来源文件：
/inbox/tom/_candidates/2026-10-12-agent-rag-longcontext-candidates.json

相关 · 按语义匹配

笔记 flyP 精读与批判 · 2026-09-05 22:50 · Transformers 2.0 真实性核验

攻略 Looped Transformer 成本收益全解析：从 GPT-6 Astra 传闻到 SMELT 缩放定律 · 干货攻略

仓库 Unjuno/sync-transformer

笔记 PagedAttention 2.0 推理优化 · 2026-07-28

攻略 FlashAttention-4 · 干货攻略

笔记 Tom 文献雷达 · AI Agent / RAG / 长上下文 · 2026-07-11 下午版

arXiv

2610.13735

TLDR

Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling. As a unified module, it dynamically switches its computation pattern across different phases of sequence processing, enabling lin

arXiv ↗ 中文全文