QuoteBench: How Matched Scores Can Hide Command-Path Failures
- 类型：arxiv
- 标识：2608.13547
- 链接：https://arxiv.org/abs/2608.13547
- 主分类：agent
- 形态：method
- TLDR：LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must com
- 待LLM分类：否
- 标题中文：QuoteBench：匹配得分如何掩盖命令路径失败
- TLDR中文：LLM 编程 Agent 通过接口下发 Bash 命令，这些接口可能对模型输出进行序列化、包装与再解析。仅凭匹配的执行得分无法区分命令生成阶段的错误与生成后引入的失败。QuoteBench 围绕一道刻意未转义的额外解析器，将生成合约与执行传输交叉，在 14 个源自真实事件的家族、共 56 个一次性任务上以精确最终状态校验来度量该边界。在插值点进行转义可复现每条回放回复的原始路径结果，因此任何在已声明边界下取得的恢复成效都必须……
- 来源文件：
- /inbox/tom/_candidates/2026-08-22-agent-rag-longcontext-candidates.json
- /inbox/tom/_candidates/2026-08-23-agent-rag-longcontext-candidates.json