Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
- 类型：arxiv
- 标识：2609.20715
- 链接：https://arxiv.org/abs/2609.20715
- 主分类：agent
- 形态：application
- 被引：0
- 被引来源：Semantic Scholar + OpenAlex
- S2被引：0
- OpenAlex被引：0
- 影响力被引：0
- TLDR：This work introduces ActObs, which also supervises the observation tokens already present in each trajectory, and traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model.
- OpenAlex ID：W7213581419
- OpenAlex DOI：10.48550/arxiv.2609.20715
- DOI：10.48550/arxiv.2609.20715
- DOI来源：OpenAlex
- 开放获取：green
- 开放获取链接：https://doi.org/10.48550/arxiv.2609.20715
- OpenAlex更新：2026-09-20
- 副分类：multimodal
- 待LLM分类：否
- 标题中文：不要遮蔽环境：观测监督会改变 RL 下智能体的探索行为
- TLDR中文：本文提出 ActObs，对每条轨迹中已有的观测 token 也进行监督，并将这一差异归因于 SFT：动作与观测梯度迅速趋于正交，而仅训练动作会留下较大的残留观测梯度，并将环境预测能力拉低至基座模型之下。
- 来源文件：
- /inbox/tom/_candidates/2026-09-18-agent-rag-longcontext-candidates.json
- [S2 enrich]
- /inbox/tom/_candidates/2026-09-19-agent-rag-longcontext-candidates.json
- [OpenAlex backfill]
- /inbox/tom/_candidates/2026-09-20-agent-rag-longcontext-candidates.json
- /inbox/tom/_candidates/2026-09-21-agent-rag-longcontext-candidates.json
- /inbox/tom/_candidates/2026-09-21-agent-memory-tool-use-candidates.json