Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

  • 类型：arxiv
  • 标识：2607.07953
  • 链接：https://arxiv.org/abs/2607.07953
  • 主分类：llm-infra
  • 形态：method
  • 被引：0
  • 被引来源：Semantic Scholar + OpenAlex
  • S2被引：0
  • OpenAlex被引：0
  • 影响力被引：0
  • TLDR：A comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 is presented, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity.
  • OpenAlex ID：W7167909665
  • OpenAlex DOI：10.48550/arxiv.2607.07953
  • DOI：10.48550/arxiv.2607.07953
  • DOI来源：OpenAlex
  • 开放获取：green
  • 开放获取链接：https://doi.org/10.48550/arxiv.2607.07953
  • OpenAlex更新：2026-07-19
  • 副分类：engineering
  • 待LLM分类：否
  • 标题中文：线性注意力架构：机制、权衡与跨层路由
  • TLDR中文：本文对 softmax 注意力与四种近期的循环线性注意力架构（DeltaNet、Gated DeltaNet、Kimi Delta Attention 与 Gated DeltaNet-2）进行对比研究，明确阐述它们在表达能力、记忆衰减、擦写控制、训练吞吐量与实现复杂度上的差异。
  • 来源文件：
  • /inbox/tom/_candidates/2026-07-10-agent-rag-longcontext-candidates.json
  • [S2 enrich]
  • /inbox/tom/_candidates/2026-07-11-agent-rag-longcontext-candidates.json
  • /inbox/tom/_candidates/2026-07-12-agent-rag-longcontext-candidates.json
  • /inbox/tom/_candidates/2026-07-13-agent-rag-longcontext-candidates.json
  • /inbox/tom/_candidates/2026-07-13-agent-memory-tool-use-candidates.json
  • [OpenAlex backfill]