Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

  • 类型：arxiv
  • 标识：2607.19058
  • 链接：https://arxiv.org/abs/2607.19058
  • 主分类：engineering
  • 形态：method
  • 被引：0
  • 被引来源：Semantic Scholar + OpenAlex
  • S2被引：0
  • OpenAlex被引：0
  • 影响力被引：0
  • TLDR：SkewAdam, an optimizer built on the observation that the three parameter populations of an MoE differ enough in size and gradient statistics that they should not receive the same state, is studied, suggesting where optimizer state lives matters at least as much as how much of it there is.
  • OpenAlex ID：W7170057390
  • OpenAlex DOI：10.48550/arxiv.2607.19058
  • DOI：10.48550/arxiv.2607.19058
  • DOI来源：OpenAlex
  • 开放获取：green
  • 开放获取链接：https://doi.org/10.48550/arxiv.2607.19058
  • OpenAlex更新：2026-08-24
  • 待LLM分类：否
  • 标题中文：标题中文：优化器状态应放在哪里？面向内存高效混合专家训练的分层状态分配
  • TLDR中文：本文研究 SkewAdam——一种基于以下观察构建的优化器：MoE 的三类参数群体在规模与梯度统计上差异足够大，不应共享相同的状态；研究表明优化器状态的存放位置至少与状态容量同等重要。
  • 来源文件：
  • /inbox/tom/_candidates/2026-07-22-agent-rag-longcontext-candidates.json
  • /inbox/tom/_candidates/2026-07-23-agent-rag-longcontext-candidates.json
  • [S2 enrich]
  • [OpenAlex backfill]