Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

类型：arxiv
标识：2609.18708
链接：https://arxiv.org/abs/2609.18708
主分类：engineering
形态：position
TLDR：In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat. We further observe this phenomenon in a controlled FrozenLake environment and find that it becomes more pronounced as the state space grows. Our theoretical and empiric
待LLM分类：是
来源文件：
/inbox/tom/_candidates/2026-09-17-agent-rag-longcontext-candidates.json

相关 · 按语义匹配

仓库 Dylsimple60/RLHF_learn

2026-09-03

笔记精读与批判 · RAGEN-2:Reasoning Collapse in Agentic RL

2026-09-08

笔记 Agent-STAR / arXiv:2603.21972 · "Demystifying RL for Long-Horizon Tool-Using Agents: A Comprehensive Recipe" · flyP 精读与批判

2026-07-18

笔记 flyP 精读与反方审稿 · Agentic RL 综述 + GLM-5.2 开放 Agent 路线 · 2026-07-15 22:50

2026-07-15

笔记 ReOPD (arXiv:2607.04763v2) 精读与批判 — flyP

2026-07-26

笔记 Cameron R. Wolfe (Deep Learning Focus) · RSS 摘要

2026-09-01

arXiv

2609.18708

TLDR

In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat. We further observe this phenomenon in a controlled FrozenLake environment and find that it becomes more pronounced as the state space grows. Our theoretical and empiric