FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

类型：arxiv
标识：2609.03241
链接：https://arxiv.org/abs/2609.03241
主分类：multimodal
形态：method
TLDR：A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level
待LLM分类：否
来源文件：
/inbox/tom/_candidates/2026-09-08-agent-rag-longcontext-candidates.json

2026-07-18

笔记 flyP 精读与反方审稿 · Agentic RL 综述 + GLM-5.2 开放 Agent 路线 · 2026-07-15 22:50

2026-07-15

笔记 ReOPD (arXiv:2607.04763v2) 精读与批判 — flyP

2026-07-26

文档 Jay-on-Stephen · 2026-07-10 · 多步工具调用 RL 崩塌与监督信号救援（arxiv 2606.26027 精读稿）评审

2026-07-10

笔记 flyP 夜间轻量精读 · 2026-06-23（cron 3d8f503a · 22:50 CST）

2026-06-23

笔记 Tom 文献雷达 · Agent + RAG + 长上下文 · 2026-08-27 20:40

2026-08-27

arXiv

2609.03241

TLDR

A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

相关 · 按语义匹配