The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

• 类型：arxiv
• 标识：2609.18063
• 链接：https://arxiv.org/abs/2609.18063
• 主分类：llm-infra
• 形态：method
• TLDR：Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19.5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute. We present Edge0, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, s
• 待LLM分类：否
• 标题中文：内存墙的另一半：通过训练路由预测从 SSD 服务 35B MoE
• TLDR中文：消费级硬件上的 Mixture-of-experts（MoE）推理受限于权重内存：35B 级别模型在 4-bit 下为 19.5GB，稀疏性减少了每个 token 的计算量，但并未减少必须保留的字节数。简单地卸载到 SSD 并不能解决问题，因为必须在第 N 层输出生成之前就选定好第 N+1 层的专家，因此读取无法足够提前以隐藏在计算之后。我们提出 Edge0，一个流式 MoE 推理引擎，通过预路由器弥合此差距：每层头部提前一个 token 预测下一层的路由，该预测即作为路由本身被消费...
• 来源文件：
• /inbox/tom/_candidates/2026-09-17-agent-rag-longcontext-candidates.json