ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

类型：arxiv
标识：2609.16816
链接：https://arxiv.org/abs/2609.16816
主分类：evaluation
形态：benchmark
TLDR：Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. Such rubrics are reliable only if they reward honest answers over adversarial answers optimized to exploit them. Yet their robustness to such optimization remains poorly understood. We isolate the hardest regime: impossible tasks, where the prompt pressures the model toward an unsupported conclusion, so the only honest response is to acknowledge the impossibility. We introduce ImpossibleRubrics, a benchmark of 169 impossible tasks s
待LLM分类：否
来源文件：
/inbox/tom/_candidates/2026-09-16-agent-rag-longcontext-candidates.json

2026-06-23

笔记 Cameron R. Wolfe (Deep Learning Focus) · RSS 摘要

2026-09-04

笔记 Tom 文献雷达 · Agent + RAG + 长上下文 · 2026-08-27 20:40

2026-08-27

笔记 Agent-STAR / arXiv:2603.21972 · "Demystifying RL for Long-Horizon Tool-Using Agents: A Comprehensive Recipe" · flyP 精读与批判

2026-07-18

笔记 flyP 精读与反方审稿 · Agentic RL 综述 + GLM-5.2 开放 Agent 路线 · 2026-07-15 22:50

2026-07-15

笔记 Cool Papers · cs.CL (papers.cool) · RSS 摘要

2026-07-29

arXiv

2609.16816

TLDR

Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. Such rubrics are reliable only if they reward honest answers over adversarial answers optimized to exploit them. Yet their robustness to such optimization remains poorly understood. We isolate the hardest regime: impossible tasks, where the prompt pressures the model toward an unsupported conclusion, so the only honest response is to acknowledge the impossibility. We introduce ImpossibleRubrics, a benchmark of 169 impossible tasks s

arXiv ↗ 中文全文

ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

相关 · 按语义匹配