SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

类型：arxiv
标识：2609.14320
链接：https://arxiv.org/abs/2609.14320
主分类：rag
形态：method
TLDR：Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linear attention state dynamics. In this work, we study long-context extension of Gated DeltaNet (GDN) from a spectral perspective of transition matrix and identify two essential factors governing long-range information retrieval: (1) a sufficiently broad slow spectral band aligned with the target depend
待LLM分类：否
来源文件：
/inbox/tom/_candidates/2026-09-17-agent-rag-longcontext-candidates.json

2026-07-11

仓库 horvitzs/Interactive_Segmentation_Models

2018-05-18

笔记 Tom 文献雷达 · Agent RAG LongContext · 2026-08-12 20:40

2026-08-12

笔记 VaLR: Vision-aligned Latent Reasoning for Multi-modal LLM

2026-06-16

笔记 MIOH: Fine-Grained Multi-Image Object Hallucination Benchmark（CVPR 2026）· v2 重写覆盖原 7-08 15:50 v1

2026-07-08

仓库 iron-XiangruChen/reservoir-image-review-reproducibility

2026-08-27

arXiv

2609.14320

TLDR

Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linear attention state dynamics. In this work, we study long-context extension of Gated DeltaNet (GDN) from a spectral perspective of transition matrix and identify two essential factors governing long-range information retrieval: (1) a sufficiently broad slow spectral band aligned with the target depend

SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

相关 · 按语义匹配