Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

  • 类型：arxiv
  • 标识：2609.01532
  • 链接：https://arxiv.org/abs/2609.01532
  • 主分类：engineering
  • 形态：method
  • 被引：0
  • 被引来源：Semantic Scholar + OpenAlex
  • S2被引：0
  • OpenAlex被引：0
  • 影响力被引：0
  • TLDR：Switch Distillation is proposed, a simple mid-training objective that distills on tokens where the teacher is confident, using teacher predictive entropy as a lightweight routing signal, and otherwise falls back to cross-entropy, which consistently outperforms existing distillation objectives across teacher sizes.
  • OpenAlex ID：W7206186238
  • OpenAlex DOI：10.48550/arxiv.2609.01532
  • DOI：10.48550/arxiv.2609.01532
  • DOI来源：OpenAlex
  • 开放获取：green
  • 开放获取链接：https://arxiv.org/pdf/2609.01532
  • OpenAlex更新：2026-09-05
  • 待LLM分类：否
  • 标题中文：中期训练阶段的知识蒸馏更偏向推理而非事实记忆
  • TLDR中文：本文提出 Switch Distillation，一种简单的 mid-training 目标：以教师预测熵作为轻量路由信号，仅在教师置信的 token 上蒸馏，其余回退到交叉熵；在不同教师规模下均稳定优于现有蒸馏目标。
  • 来源文件：
  • /inbox/tom/_candidates/2026-09-03-agent-rag-longcontext-candidates.json
  • [OpenAlex backfill]
  • [S2 enrich]