研究库 论文知识库

内容库 / 论文

arXiv:2609.14306 · 工程化

Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training

长上下文 Mixture-of-Experts 训练中每个内存峰值的平整化

Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training

类型：arxiv
标识：2609.14306
链接：https://arxiv.org/abs/2609.14306
主分类：engineering
形态：method
TLDR：Training a Mixture-of-Experts (MoE) model at long context or large batch size fails as soon as any one component's peak allocation exceeds device memory, so the target is every peak at once, not the average footprint. Four are left unbounded by the parallelism plans in common use, and each grows differently: expert dispatch with the routing matrix, the vocabulary projection with tokens times vocabulary, gradient checkpoint boundaries with depth times sequence length, and optimizer state with parameter count. Which one runs out first changes with the model, the context length, and the device co
待LLM分类：否
标题中文：长上下文 Mixture-of-Experts 训练中每个内存峰值的平整化
TLDR中文：在长上下文或大批量尺寸下训练 Mixture-of-Experts（MoE）模型时，只要任一组件的峰值分配超过设备内存即会失败，因此目标是同时平整每个峰值，而非平均占用。常用并行方案对其中四类不设上限，且各自增长规律不同：随路由矩阵增长的专家分派、随 token×词表增长的词表投影、随深度×序列长度增长的梯度检查点边界、以及随参数量增长的优化器状态。哪个先耗尽取决于模型、上下文长度与设备协...
来源文件：
/inbox/tom/_candidates/2026-09-17-agent-rag-longcontext-candidates.json

相关 · 按语义匹配

笔记 Substack · α-signal · RAG and Long Context Aren't Enough for Agent Memory. δ-mem Is a Third Option

笔记知识库草稿 · Jay · 2026-08-01 19:50（傍晚高频）

笔记 MaP-WAM · Memory-as-Plans World-Action Modeling 精读与批判（2026-09-13 flyP 下午棒）

笔记 ReMemR1: Look Back to Reason Forward (Revisitable Memory for Long-Context LLM Agents)

文档 Stephen 自测 · R74 · 2026-09-13

仓库 djasdh/interest-memory

arXiv

2609.14306

TLDR

在长上下文或大批量尺寸下训练 Mixture-of-Experts（MoE）模型时，只要任一组件的峰值分配超过设备内存即会失败，因此目标是同时平整每个峰值，而非平均占用。常用并行方案对其中四类不设上限，且各自增长规律不同：随路由矩阵增长的专家分派、随 token×词表增长的词表投影、随深度×序列长度增长的梯度检查点边界、以及随参数量增长的优化器状态。哪个先耗尽取决于模型、上下文长度与设备协...Training a Mixture-of-Experts (MoE) model at long context or large batch size fails as soon as any one component's peak allocation exceeds device memory, so the target is every peak at once, not the average footprint. Four are left unbounded by the parallelism plans in common use, and each grows differently: expert dispatch with the routing matrix, the vocabulary projection with tokens times vocabulary, gradient checkpoint boundaries with depth times sequence length, and optimizer state with parameter count. Which one runs out first changes with the model, the context length, and the device co

相关解读

arXiv ↗ 中文全文