evaluation · 知识库活文档

  • 更新:R56 ReliabilityBench+HORIZON 立基础锚+AgentDebug 17×5错误分类 P0反方+OmniAssistBench 行为对齐新锚+EnvHarness 258▲新高

0. 范围与定调

本活文档面向 LLM / Agent / RAG / 多模态系统评测 主题,覆盖 benchmark 设计、指标体系、judge / harness 工程、可信评测方法学、跨模态评测、EDD 等。与 agent.md、rag.md、multimodal.md、llm-infra.md 平行。时间锚定 2026 年中。

四十六重范式跃迁(R56 终态,简版):R0-R37 早期范式(见 archive);R37 评测诊断栈分层;R38-R44 评测方法学合流期;R45 立标信号 vs 立标等级双维度评估方法论首次立标;R46 BDH-CQ跌出+OpenART/Latent-to-4D/StateFlow/Mechanist五件;R47 LLMRouter/DarwinX/PlayWorld/AutoDesign四件+立标池6向并存;R48 Cameron Wolfe+PostTrainBench++AI Agents Stack 2026三件社区内容;R49 立标池6→7向并存扩展+AgentRx邻接级候选;R50 立标池7→6收缩+OpenART反弹锚断裂+arXiv:2608.13417第80维立标级+5件归口化+MMLongBench/MMLongEmbed/MerchantBench/Skill²-Bench §2.207 13→17元组+立标池饱和度v44-v52九日连续;R51 HarnessEval-W(111▲)+Gathered Not Admitted+Accuracy&Order Sensitivity三件eval专项net-new+立标池双新锚+ClawGym II/信息满足度/HarnessRisk四件邻接;R52 立标池第7日结构性切换(StateM 374▲登顶+ASI-Bench 51▲新晋+LLMRouter/DarwinX/Mechanist三件同日跌出)+HarnessRisk(6阶段128场景)+PTXBench(GPU kernel)+Six Degrees(几何签名)+harness生态三角StateM/HarnessEval-W/HarnessRisk首次完整形成+SWE-bench Pro量化实证(同模型不同harness pass@1 23%→52%,rank correlation -0.05)+§2.207评测方法学17→19元组+失败模式反方体系37→38层+跨主题共识30→35+立标池饱和度v44-v53十日连续;R53 Zetta ζ(arXiv:2608.16590 paper_cards 1027 HF Daily 8-21 #3 130▲)闭环具身harness自进化+SoftVTBench(arXiv:2608.18701 paper_cards 1025)形变感知视触觉操作benchmark+Evaluating Music Context Preservation/MuseCP(arXiv:2512.14629 paper_cards 1036)音乐context preservation benchmark 三件eval专项net-new立基础锚★★★+SemComp-Bench(arXiv:2608.17426 paper_cards 1026)HF Daily 8-21 #2 153▲视频生成语义完成度评测新晋锚+SemaPLC(arXiv:2608.18565)HF Daily 8-21 #4 111▲工业控制PLC代码生成验证门控harness邻接+ASI-Bench(arXiv:2608.17271)HF Daily 8-21 #7 55▲连续第二日攀升+SWE-bench Science(arXiv:2608.19799)Tom radar高价值科学软件Agent可靠性评测+MissDiag(arXiv:2608.18489 paper_cards 1034 work-queue Top15 #5)KGQA/KG-RAG不完整知识诊断式评估+ARFT/AutoResearch(arXiv:2608.14905)metacognitive loop缺失诊断式评估框架flyp critical-read 8-20 六件eval邻接+harness生态三角→四角(StateM/HarnessEval-W/HarnessRisk/Zetta ζ)+§2.207评测方法学19→22元组候选+§2.2评测对象80→84维候选+§2.1评测方法学32→34轴候选+§2.6失败模式反方体系38→39层+评测方法学社区内容补全三联→四联立标(R48 Cameron Wolfe+R50 flyp 7要点+R52 SWE-bench Pro+R53 ARFT/StateM反方立基础候选双锚)+立标池饱和度v44-v54十一日连续;R54 HSI/Hierarchical Self-Improvement(arXiv:2608.08466 paper_cards 1057 R54新建 evaluation主分类 agent副分类 application work-queue Top15 #1)三层可热替换harness(task harness+evolver+meta-evolver,thinking-on/off 隔离性,feedback-fidelity bound+backbone limits)eval专项net-new立基础锚★★★+EnvHarness(arXiv:2608.19880 paper_card未建 HF Daily 8-22 #1 235▲)静态世界唤醒支持Agent学习立标级新锚登顶(Google Cloud AI Research,EnvHarness=可编程插件层Setup/Rule/Link+EnvRigger=LLM设计师agent,reset/step接口不动+原始verifier保留语义不动,5 benchmark×4域held-out +9.0分/-9.8%步数,RL co-evolution政策-环境共同进化)+FACET(arXiv:2608.18580 paper_card未建 HF Daily 8-22 #5 107▲)终端任务合成源代码意图与可执行状态保留邻接+MemTrapBench(arXiv:2608.20202 paper_card未建 HF Daily 8-22 #13 29▲)memory-induced reasoning distortion填补memory评测第5维邻接+QuoteBench(arXiv:2608.13547 paper_cards 1056 R54新建 agent主分类 eval副分类 work-queue Top15 #2)命令执行边界量化邻接+HELIX(arXiv:2608.13951 R54 web_search意外发现)model-harness co-evolution邻接预备+Co-RL(arXiv:2608.17253 HF Daily 8-22 #6 90▲)多智能体强化学习协同推理邻接+SPADE(arXiv:2608.19197 HF Daily 8-22 #10 44▲)自适应合成可执行环境自博弈邻接+SkillEvo(arXiv:2608.13120 HF Daily 8-22 #14 27▲)多轮交互反馈自我更新演化梯度邻接+立标池双向锚第9日结构性切换(EnvHarness 235▲登顶超越SemComp-Bench 155▲/Zetta ζ 140▲/SemaPLC 114▲三件延续+ASI-Bench 8-22跌出top15连续两日攀升后衰减确认+SWE-bench Science 56▲新晋邻接)+评测方法学22→24元组候选(HSI第23+EnvHarness第24)+harness生态四角→五角预备(HSI第5节点:任务特定可热替换harness)+评测方法学延革第12例反方立基础候选预备(EnvHarness:环境侧harness化,与HarnessEval-W评估方agent化垂直互补)+§2.2评测对象84→85维候选(MemTrapBench memory reasoning distortion)+§2.6失败模式反方体系39→41层(HSI反馈保真度上界+EnvHarness EnvRigger黑盒可靠性)+评测方法学社区内容补全四联→五联立标(R48 Cameron Wolfe+R50 flyp 7要点+R52 SWE-bench Pro+R53 ARFT/StateM+R54 HSI/EnvHarness HELIX三件隐含发现)+立标池饱和度v44-v55十二日连续;R55 HarnessEval-W(arXiv:2608.16859 paper_card 992 R55新建 cs.CV视觉世界agentified evaluation pipeline paper_card 992 8-23)web_search验证视觉世界评测harness范式迁移(MirroS Lab)+Large Discovery Models(arXiv:2608.15669 paper_card 998 R55新建 cs.LG经验驱动开放式搜索评测 paper_card 998 8-23)web_search验证贝叶斯非参数奖励+生成模型耦合不确定性感知价值信号引导候选生成/选择 两件eval专项net-new立基础锚★★★+Zetta ζ(arXiv:2608.16590 paper_card 1027)+SemaPLC(arXiv:2608.18565 paper_card未建)双锚flyp 8-23 09:50 critical-read落盘"真实执行是最终裁判"叙事 vs Harness-Evolution-Eval-Rethink(arXiv:2607.12227 Allen Institute/UW)flyp 8-22 22:50 critical-read落盘 controlled budget protocol否定harness evolution在Terminal-Bench上稳定超越test-time scaling 实验环境分歧构成评测方法学质疑对位临界+The Embedder's Dilemma(arXiv:2608.12875 paper_card 1055 R53/55复用 rag主分类 eval邻接)Tom radar 8-23高价值 LLM vs embedding评测范式各有所长37任务10LLM×26embedding 最佳LLM 77.6 vs 最佳embedding 77.2仅差0.4分+ClawVM(arXiv:2604.10352 EuroMLSys '26)Jay substack 8-23 1450 stateful tool-use虚拟内存管理harness 引用Long-MemEval benchmark+From Prompts to Contracts(arXiv:2607.08028)Jay substack 8-23 企业agent可审计性harness+Task-CoEvolve(arXiv:2608.20169)Cool Papers 8-23 harness高效优化自适应验证任务选择+PV-SST/Peer-Voted Stress Tests(arXiv:2608.20438 HF Daily 8-25)448 trials 4主题4种子4模型群体行为feed诱导词汇收敛+FlavourBench(arXiv:2608.20574 HF Daily 8-25)版本化烹饪系统executable ground truth 534任务27 frontier端点14,418 model-task cells+立标池双向锚第10日结构性切换(EnvHarness 8-23 246▲ +11▲续立+8-24 254▲ +8▲续立极显著 vs SemComp-Bench 8-23 155▲ +0▲持平 vs 8-24 155▲持平=Zetta 8-23 141▲+1▲ vs 8-24 142▲+1▲=SemaPLC 8-23 115▲+1▲ vs 8-24 115▲持平=FACET 8-23 112▲+5▲ vs 8-24 114▲+2▲=Co-RL 8-23 91▲+1▲ vs 8-24 92▲+1▲=MemTrapBench 8-23 31▲+2▲ vs 8-24 31▲持平=SWE-bench Science 8-23 58▲ vs 8-24 61▲+3▲=SPADE 8-23 47▲ vs 8-24 48▲+1▲=SkillEvo 8-23 29▲ vs 8-24 30▲+1▲=ForgeWM 8-23 22▲ vs 8-24 22▲持平)十三日连续+ASI-Bench 8-23/8-24 跌出确认(连续三日跌出)+harness生态五角→六角预备(HarnessEval-W cs.CV视觉世界harness第6节点,与HSI任务特定可热替换垂直互补)+§2.1评测方法学34→35轴候选(LDM开放式发现/探索-利用评测轴第35)+§2.2评测对象85→86维候选(Embedder's Dilemma embedding vs LLM评测范式比较维)+§2.4 Judge&Harness 86→90件套+Judge校准34→35+Rethink/Zetta+SemaPLC双锚归口+§2.5反方体系41→44层(R55+3:§2.5.42 Rethink "harness evolution budget-confounded"质疑+"verifier-rich setting下harness evolution算力等价不优于parallel sampling/sequential refinement"+§2.5.43 Zetta "11.1×推理加速基线未明+harness演化算力对齐未量化+Aha Moments可复现性"+§2.5.44 SemaPLC "65任务样本量小+私有数据集风险+verifier-rich局限")+§2.6失败模式反方体系41→43层(R55+2:§2.6.44 Rethink "单基准依赖Terminal-Bench"+§2.6.45 HarnessEval-W "agentified pipeline引入sub-agent失败传播风险未量化")+§2.7评测对象86→88维+36→39邻接维+135→137子领域(R55 +2维 Embedder's Dilemma+PV-SST +3邻接 Embedder's Dilemma/ClawVM/From Prompts to Contracts)+§3.5 harness生态六角预备(HarnessEval-W cs.CV视觉世界harness第6节点:与HSI/HarnessEval-W执行层/EnvHarness环境层/HarnessEval-W评估方agent化/Zetta ζ具身/Zetta ζ evaluation方法学质疑对位)+§6.71-§6.76三节扩展(HarnessEval-W §6.74 + LDM §6.75 + Rethink §6.76)+§7.1共识39→41(R55+2:HarnessEval-W+LDM立基础锚+Rethink/Zetta+SemaPLC方法学质疑对位)+§7.2争议95→98(R55+3)+§7.3开放127→132(R55+5)+§8趋势123→126(R55+3:Embedder's Dilemma评测范式比较+多智能体压力测试+可执行ground truth评测)+§9评测平台32维持+评测harness/benchmark套件227→233件(R55+6:HarnessEval-W/LDM/Rethink/Embedder's Dilemma/ClawVM/From Prompts to Contracts/Task-CoEvolve)+评测方法学延革第13例预备(双锚 Zetta+SemaPLC "真实执行判定" 与Rethink "Terminal-Bench算力等价质疑"构成第13例对位)+立标池饱和度v44-v56十三日连续;R56 ReliabilityBench(arXiv:2603.29231 flyp 8-25 0507 critical-read落盘★★★P0反方立基础锚)+HORIZON(arXiv:2604.11978 COLM 2026 flyp 8-25 critical-read落盘★★★长视Agent元评测trajectory-grounded LLM-judge)+AgentDebug(GitHub ulab-uiuc/AgentDebug Jay 8-25 1450工程筛选★★★17种错误×5模块memory/reflection/planning/action/system两阶段调试pipeline MIT license)+41种Agent失败模式分类学(arXiv:2607.28802 jay 8-25 X硬核干货雷达+Omar Sarheed★★评测方法学延革第18例实测预备+反方立基础延革第2例预备+Cohen's kappa 0.76 substantial agreement +5-30×成本波动根因来自harness而非模型)+OmniAssistBench(arXiv:2608.21360 HF Daily 8-25 #12 27▲ 新晋锚★助手风格交互评测)+Beyond Correctness(arXiv:2608.12781 HF Daily 8-25 #14 19▲ 新晋锚★行为对齐评测)+Datadog State of AI Engineering 2026(Jay 8-25 工程筛选★★★5% LLM调用报错+60% rate limit +840万次/月)+TraceCoder ICSE 2026多Agent调试框架(Jay 8-25 Instrumentation→Analysis HLLM→Repair rollback Pass@1 +34.43%)+八件8-25新候选(EnSI-RAG/PV-SST/FlavourBench/Hydra-0/PhysCaP/SparsePR/Human-Centric Survey/UniSpace)新增立标池双向锚第14日信号(EnvHarness 8-25 258▲ #1 创历史新高 +4▲续立极显著 vs FACET 8-25 115▲ #2 持平 vs SWE-bench Science 8-25 61▲ #4 +3▲ vs MemTrapBench 8-25 31▲ #9 持平 vs SkillEvo 8-25 30▲ #10 +1▲ vs OmniAssistBench 8-25 27▲ #12 新晋 vs ForgeWM 8-25 23▲ #13 +1▲ vs Beyond Correctness 8-25 19▲ #14 新晋 vs ASI-Bench 8-25 跌出确认连续四日)+评测对象88→90维+39→41邻接维+137→140子领域(R56+2维 OmniAssistBench助手风格交互评测+Beyond Correctness行为对齐评测+2邻接 OmniAssistBench/AgentDebug+TraceCoder)+§2.1评测方法学35轴并行+§2.207评测方法学22→24元组候选+评测元方法学97→103件套(R56+3:ReliabilityBench/HORIZON/41种失败模式)+Judge&Harness 90→92件套(R56+2)+Judge校准35→36(R56+1:ReliabilityBench RDC/VAF/GDS/MOP四指标+AgentDebug 17×5错误分类)+评测基础套件83→87(R56+4)+§2.5反方+2条(R56四十六层:§2.5.45 ReliabilityBench memory scaffold普遍伤害 + §2.5.46 HORIZON inter-annotator κ=0.61归因类目校准)+§2.6失败模式反方体系43→46层(R56+3五阶:§2.6.46 ReliabilityBench单模型≈2,300样本VAF统计力 + §2.6.47 HORIZON跨域harness异质性 + §2.6.48 AgentDebug 17种错误分类未公开Cohen's kappa)+§3.5 harness生态六角延伸(R56+1节点:AgentDebug错误归因harness层第7节点候选)+§6.77-§6.80四节新增(ReliabilityBench §6.77 + HORIZON §6.78 + AgentDebug §6.79 + 41种失败模式分类学 §6.80)+§7.1共识41→43(R56+2:ReliabilityBench memory scaffold反方+评测协议失真类延展到meta-evaluation)+§7.2争议98→101(R56+3)+§7.3开放132→138(R56+6)+§8趋势126→131(R56+5:ReliabilityBench memory scaffold反直觉+HORIZON跨域诊断+AgentDebug错误归因工程化+41种失败模式分类学+Datadog生产错误率量化)+评测方法学延革第14-18例(R56新增5例)+评测方法学社区内容补全五联→七联立标(R48 Cameron Wolfe+R50 flyp 7要点+R52 SWE-bench Pro+R53 ARFT/StateM+R54 HSI/EnvHarness HELIX三件+R55 HarnessEval-W/LDM/Rethink三件+R56 ReliabilityBench/HORIZON/AgentDebug三件七联)+立标池双向锚稳态化趋势v33以来首次机制化第5例+立标池饱和度v44-v57十四日连续。

LLM 评测进入 多维/过程化/可验证/可审计/可复现/训练-评测一体化/Harness自演进/Judge Reliability Harness工业落地/评测协议可操作化/成本-成功率第三维度/Agent-as-a-Judge/行为轨迹诊断/行为隐私/效率-质量权衡/AI-to-AI治理/3D空间推理/测试充分性/法律垂直/多Agent评判/评估混淆量化/物理定律video评测/答案接口不匹配/Compounding Error静默失败量化/harness-benchmark可比性/程序化评判器/Agent记忆盲点/原子视觉感知失败分类/多模态memory/Agent Memory五维纵向评估/Eval-first Development/Token级credit assignment/Repo-level代码推理/用户对齐评测/模拟用户准确性/节点级backward credit assignment/多Agent心理安全/工业API可靠性评测/场景专化评测/长程可靠性科学/长程多Agent/MLLM多模态日常安全/Agent失败归因分类法/Harness工程化方法论/长程任务状态管理/代码编辑行为度量/记忆价值量化/Agent个性化安全/金融推理可信度/自主Agent长视野Harness/CUA轨迹VLM Judge/元评测方法论/长程失败结构迁移规律/VLM视频级全局空间感知/国家标准规则密集型/评测环境作为攻击面/Evaluation-as-Infrastructure/Harness披露-测量-Loop三元组/Harness演进能力/群体用户模拟/视频段落描述/有机多模态推理/latent vs verbalized reasoning/翻译基准污染探测/多语言代码Agent/logit-space鲁棒性/评测环境作为攻击面二元实证/生活化世界主动性与持久性/立标信号vs立标等级/立标池双向锚/harness生态三角(StateM/HarnessEval-W/HarnessRisk)/SWE-bench Pro量化实证/ASI-Bench超对齐评测/闭环具身harness自进化/形变感知视触觉操作评测/音乐context preservation评测/视频生成语义完成度评测/工业控制PLC代码生成验证门控harness/科学软件Agent可靠性评测/KGQA/KG-RAG不完整知识诊断式评估/metacognitive loop缺失诊断式评估/任务特定可热替换harness/静态世界唤醒支持Agent学习/终端任务合成源代码意图与可执行状态保留/memory-induced reasoning distortion评测/命令执行边界量化/model-harness共同进化/视觉世界agentified evaluation pipeline/经验驱动开放式搜索评测/评测协议质疑对位/真实执行最终裁判/embedding vs LLM评测范式比较/ReliabilityBench可靠性科学框架/HORIZON跨域诊断trajectory-grounded LLM-judge/AgentDebug 17种错误×5模块两阶段调试/41种Agent失败模式分类学/OmniAssistBench助手风格交互评测/Beyond Correctness行为对齐评测 等一百五十+项特性的工程体系。

1. 现状全景

2026年中评测不再是"哪个模型在MMLU上分数高",而是 多维/过程化/可验证/可审计/可复现/训练-评测一体化 等一百五十+项特性的工程体系。

R56 评测对象 88→90维+39→41邻接维+137→140子领域+评测元方法学97→103件套+评测方法学35轴并行+Judge校准35→36+评测基础套件83→87+失败模式反方体系43→46层(R56+3五阶:§2.6.46 ReliabilityBench单模型≈2,300样本VAF统计力 + §2.6.47 HORIZON跨域harness异质性 + §2.6.48 AgentDebug 17种错误分类未公开Cohen's kappa)+评测方法学反方体系44→46层(R56+2:§2.5.45 ReliabilityBench memory scaffold普遍伤害 + §2.5.46 HORIZON inter-annotator κ=0.61归因类目校准)+跨主题共识41→43(R56+2)。

R37-R56 增补五十七件(详见§2全文)。

2. 关键工作脉络

2.1 评测方法学综述与范式分类

R56 评测元方法学 97→103件套:R55 97件套+R56 +3件(ReliabilityBench/HORIZON/41种失败模式分类学);R56 +6邻接(AgentDebug/TraceCoder/Datadog/OmniAssistBench/Beyond Correctness/Eval-as-Infrastructure沿用)= 109件。评测方法学35轴并行维持。

R56 §2.1 评测方法学综述新增:

Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents(arXiv:2603.29231, Aaditya Khanal 等, 23 页 4 图, 2026-03-31) flyp 8-25 0507 critical-read落盘 ★★★ P0 反方立基础锚 - 核心方法:把 capability(pass@1)与 reliability(跨时长/跨重复的稳定性)区分开;四指标设计:Reliability Decay Curve (RDC)= 成功率随时长衰减曲线;Variance Amplification Factor (VAF)= 跨重复方差放大倍数;Graceful Degradation Score (GDS)= 随时长性能软着陆程度(0=悬崖式,1=平稳下降);Meltdown Onset Point (MOP)= Meltdown 概率陡升临界时长 - 实验规模:396 tasks × 4 duration buckets × 3 domains = 33 task per cell × 10 开源模型 × 2 scaffolds(ReAct vs memory-augmented)× k=3 重复 = 23,392 episodes - 5 个关键发现:(1) reliability decay 是 domain-stratified:SE GDS 0.90→0.44,文档处理 0.74→0.71;(2) VAF 按 capability 分层:高 VAF 是能力签名而非不稳定信号;(3) capability 与 reliability 排序显著发散,长视下出现多档排名倒置;(4) 前沿模型 meltdown 率最高(up to 19%),策略野心大反而螺旋;(5) memory-augmented scaffold 普遍伤害长视性能(10 模型无一例外) — 与流行叙事相反 - 关键反方:memory scaffold 普遍伤害是对当下 memory-augmented agent 热潮的强证伪信号;single model ≈2,300 samples对 VAF 这种方差指标统计力可能不足;没有公开 leaderboard(不像 HORIZON),复现门槛偏高 - 归入:§2.5反方第 45 层(memory scaffold 普遍伤害);§2.6失败模式反方体系第 46 层(VAF 统计力);§2.207评测方法学第 28 元组候选(Reliability Decay Curve / VAF / GDS / MOP 四指标);评测方法学延革第 14 例实测;评测方法学社区内容补全七联立标预备第 7 件(ReliabilityBench);Judge 校准 35→36

HORIZON: The Long-Horizon Task Mirage(arXiv:2604.11978, Haoyue Bai 等, COLM 2026) flyp 8-25 critical-read落盘 ★★★ 长视 Agent 元评测 - 核心方法:跨域诊断 benchmark 框架;trajectory-grounded LLM-as-a-Judge pipeline;与人工标注对照 inter-annotator κ=0.61, human-judge κ=0.84 - 评测范围:4 个代表性 agentic domains × 700+ tasks(WebArena 61 / AgentBench 27 / MAC-SQL 43 / Isaac Sim 18 + 自有);收集 3,100+ trajectories,GPT-5 与 Claude-4 variants - 关键贡献:把 failure attribution 当成评测对象,公开 LLM-judge 校验流程;跨域比较:哪些失败模式是 domain-specific,哪些是 horizon-specific;公开 leaderboard(xwang2775.github.io/horizon-leaderboard) - 关键反方:inter-annotator κ=0.61 偏低,failure attribution 类目本身需要再校准;评测模型仅 GPT-5 + Claude-4 variants,没有 open-source 队列 — 跟 2603.29231 形成强互补;跨域比较受各 domain harness 异质性干扰(WebArena / AgentBench / MAC-SQL harness 差异大) - 归入:§2.5反方第 46 层(HORIZON inter-annotator κ=0.61 归因类目校准);§2.6失败模式反方体系第 47 层(跨域 harness 异质性);§2.207评测方法学第 29 元组候选(跨域诊断 + trajectory-grounded LLM-judge);评测方法学延革第 15 例实测;评测方法学社区内容补全七联立标预备第 7 件(HORIZON)

41 种 Agent 失败模式分类学(arXiv:2607.28802, Omar Sarheed @omarsar0 jay 8-25 X 硬核干货雷达)★★评测方法学延革第 18 例实测预备 - 核心:41 种失败模式 × 六节点归因 model/harness/user/tools/memory/environment;Cohen's kappa 0.76 达到 substantial agreement(0.61-0.80 区间)= harness 故障可被自动检测和归因 - 关键洞察:harness bug vs model bug 边界首次系统性定义 = 当前 agent 工程最模糊也最贵的边界;5-30× 成本波动根因来自 harness 设计缺陷而非模型能力不足 - 关键反方:Cohen's kappa 0.76 substantial agreement 区间(0.61-0.80)但非 perfect,实际应用中需根据具体场景调整;5-30× 成本波动根因来自 harness 缺陷而非模型能力 = 与 ReliabilityBench memory scaffold 普遍伤害形成"双重反方立基础延展预备" - 归入:§2.207评测方法学第 30 元组候选(41 种失败模式六节点归因 + Cohen's kappa 0.76);评测方法学延革第 18 例实测预备;反方立基础延革第 2 例预备(5-30× 成本波动根因来自 harness 而非模型);评测方法学社区内容补全七联立标预备第 7 件(41 种失败模式分类学)

AgentDebug(GitHub ulab-uiuc/AgentDebug, UIUC NLP/SE Lab, MIT license) Jay 8-25 1450 工程筛选 ★★★ Agent 错误分类法 - 核心:AgentErrorTaxonomy:17 种错误类型 × 5 模块(memory/reflection/planning/action/system);AgentErrorBench:ALFWorld、GAIA、WebShop 失败轨迹标注数据集;两阶段调试 pipeline:隔离根因 + 纠正反馈 - Action 模块错误:misalignment、invalid_action、format_error、parameter_error;System 模块错误:step_limit、tool_execution_error、llm_limit、environment_error - 关键贡献:首次提供系统性 Agent 错误分类法;为"eval 驱动的 debug"提供可执行方法论;5 模块覆盖 Agent 架构全栈 - 归入:§2.207评测方法学第 31 元组候选(AgentErrorTaxonomy 17×5 + AgentErrorBench);§6.79 §2.6失败模式反方体系第 48 层候选(AgentDebug 17 种错误分类未公开 Cohen's kappa);§3.5 harness 生态六角延伸第 7 节点候选(错误归因 harness 层);评测方法学延革第 16 例实测;评测方法学社区内容补全七联立标预备第 7 件(AgentDebug)

Datadog State of AI Engineering 2026(datadoghq.com/state-of-ai-engineering, Datadog Research) Jay 8-25 ★★★ 第一方生产 trace 数据 - 核心数据:2026 年 2 月:LLM 调用 span 中 5% 报错,其中 60% 是 rate limit 错误;2026 年 3 月:约 840 万次 rate limit 错误;容量天花板是生产环境中 LLM 调用失败的首要原因;纯基础设施监控(uptime)无法反映输出质量 - 关键贡献:生产环境 LLM 错误率量化 + 错误类型分布 = eval 驱动的容量规划数据;填补了"学术 benchmark vs 生产错误率"鸿沟 - 归入:§2.1评测方法学邻接(生产环境错误率量化轴候选);§9 评测平台对比邻接(生产 trace 数据接入);评测方法学延革第 17 例实测(从"学术评测"扩展到"生产落地量化失败率"再扩展到"生产错误类型分布")

TraceCoder ICSE 2026(Jay 8-25 1450 awesome-harness-engineering 引用)★★多 Agent 调试框架 - 核心:Instrumentation Agent → Analysis Agent(HLLM 历史经验)→ Repair Agent(含 rollback);Pass@1 提升 34.43% - 归入:§2.207评测方法学第 32 元组候选(多 Agent 调试框架 + HLLM 历史经验 + Repair rollback);评测方法学延革第 17 例实测邻接

评测方法学社区内容补全七联立标:R48 Cameron Wolfe 7要点+R50 flyp 7要点二次精读+R52 SWE-bench Pro量化实证+R53 ARFT/StateM反方立基础候选双锚+R54 HSI/EnvHarness/HELIX三件隐含发现+R55 HarnessEval-W/LDM/Rethink三件对位 → R56 新增第 7 联立标(ReliabilityBench + HORIZON + AgentDebug + 41种失败模式分类学 + Datadog + TraceCoder 六件 = 评测方法学延革第 14-18 例实测预备)= harness 化(harness 生态六角延伸)+评测协议失真(protocol critique 新类) + 真实执行判定(execution-as-truth 锚)+ 长视可靠性科学(reliability science 新类)+ 错误归因工程化(error attribution harness 层)+ 反方立基础延展(memory scaffold + 成本波动根因) 六轴同步扩张

R56 立标池双向锚第 14 日信号(2026-08-25):

立标池双向锚逐日信号(R56 第 14 日): - EnvHarness 8-22 235▲ → 8-23 246▲(+11▲续立极显著)→ 8-24 254▲(+8▲续立极显著)→ 8-25 258▲(+4▲续立极显著,创 R53 以来历史新高) — 连续四日极显著,增幅收窄但绝对值新高 - FACET 8-22 107▲ → 8-23 112▲(+5▲中等)→ 8-24 114▲(+2▲中等)→ 8-25 115▲(#2,+1▲微强) — 维持 - SWE-bench Science 8-22 56▲ → 8-23 58▲(+2▲)→ 8-24 61▲(+3▲中等)→ 8-25 61▲(#4,持平) — 维持 - MemTrapBench 8-22 29▲ → 8-23 31▲(+2▲)→ 8-24 31▲(持平)→ 8-25 31▲(#9,持平) — 维持 - SkillEvo 8-22 27▲ → 8-23 29▲(+2▲)→ 8-24 30▲(+1▲)→ 8-25 30▲(#10,+1▲微强) — 维持 - ForgeWM 8-22 22▲ → 8-23 22▲(持平)→ 8-24 22▲(持平)→ 8-25 23▲(#13,+1▲微强) — 维持 - OmniAssistBench 8-24 未上榜 → 8-25 27▲(#12 新晋锚) — 新晋锚 ★ - Beyond Correctness 8-24 未上榜 → 8-25 19▲(#14 新晋锚) — 新晋锚 ★ - ASI-Bench 8-22 跌出 → 8-23 跌出确认 → 8-24 跌出确认 → 8-25 跌出确认 — 连续四日跌出,信号中断确认 - StateM 8-20 374▲(HF Daily #1,R52 立标池第 7 日立基础锚)→ 8-23/8-24/8-25 无数据延续(衰减预判维持) - SemComp-Bench 8-22 155▲ → 8-23 155▲(持平)→ 8-24 155▲(持平)→ 8-25 未列 → 衰减信号持续 - Zetta ζ 8-22 140▲ → 8-23 141▲(+1▲微强)→ 8-24 142▲(+1▲微强)→ 8-25 未列 → 衰减信号 - SemaPLC 8-22 114▲ → 8-23 115▲(+1▲微强)→ 8-24 115▲(持平)→ 8-25 未列 → 衰减信号 - Co-RL 8-22 90▲ → 8-23 91▲(+1▲微强)→ 8-24 92▲(+1▲微强)→ 8-25 未列 → 衰减信号 - SPADE 8-22 44▲ → 8-23 47▲(+3▲)→ 8-24 48▲(+1▲微强)→ 8-25 未列 → 衰减信号 - PV-SST 8-25 未上榜(单日 only)→ R56 新晋邻接 ★ - FlavourBench 8-25 未上榜(单日 only)→ R56 新晋邻接 ★ - HarnessEval-W/LDM R55 paper_card 992/998 新建 → 8-24/8-25 paper_card 延续,HF Daily 未上榜 → 立基础锚延续信号弱 - ReliabilityBench/HORIZON/AgentDebug R56 三件 P0 反方立基础锚 + 评测方法学延革第 14-16 例实测预备 - 41 种 Agent 失败模式分类学 R56 评测方法学延革第 18 例实测预备 + 反方立基础延革第 2 例预备

R56 立标池第 14 日综合:EnvHarness 258▲ 锚1 续立极显著但增幅收窄(+4▲ vs +8▲,+11▲ 起始)+ FACET/SWE-bench Science 稳维持 + OmniAssistBench + Beyond Correctness 双新晋锚 + SemComp-Bench/Zetta ζ/SemaPLC/Co-RL/SPADE 五件衰减信号持续 + ASI-Bench 衰减确认 + 立标池饱和度 v44-v57 十四日连续 + 立标池双向锚稳态化趋势 v33 以来首次机制化第 5 例

2.2 Benchmark设计从「题海」走向「可验证世界」

R56 立标池双向锚第 14 日信号:EnvHarness 8-25 258▲ 锚1 续立极显著 + OmniAssistBench 27▲ 新晋 + Beyond Correctness 19▲ 新晋 + ASI-Bench 连续四日跌出 + SemComp-Bench/Zetta ζ/SemaPLC/Co-RL/SPADE 五件衰减信号持续;立标池双向锚稳态化趋势 v33 以来首次机制化第 5 例

2.3 评测指标体系(R56 维持+4件立基础锚+6件邻接)

R56 核心新增件:

(i) ReliabilityBench / Beyond pass@1: A Reliability Science Framework (arXiv:2603.29231, flyp 8-25 0507 critical-read落盘, 论文追溯,Aaditya Khanal 等, ★★★ P0 反方立基础锚) - 核心:Reliability Decay Curve (RDC) / Variance Amplification Factor (VAF) / Graceful Degradation Score (GDS) / Meltdown Onset Point (MOP) 四指标 - 关键贡献:memory-augmented scaffold 普遍伤害长视性能(10 模型无一例外) — 强证伪信号 - 关键反方:单模型 ≈2,300 samples 对 VAF 统计力不足;无公开 leaderboard 复现门槛高 - 归入:§2.5反方第 45 层(memory scaffold 普遍伤害);§2.6失败模式反方体系第 46 层(VAF 统计力);§2.207评测方法学第 28 元组候选

(ii) HORIZON / The Long-Horizon Task Mirage (arXiv:2604.11978, COLM 2026, flyp 8-25 critical-read落盘, ★★★ 长视 Agent 元评测) - 核心:跨域诊断 benchmark 框架 + trajectory-grounded LLM-judge pipeline;inter-annotator κ=0.61, human-judge κ=0.84 - 关键数据:700+ tasks(WebArena 61 / AgentBench 27 / MAC-SQL 43 / Isaac Sim 18 + 自有)× 3,100+ trajectories × GPT-5 + Claude-4 variants - 归入:§2.5反方第 46 层(HORIZON κ=0.61 归因类目校准);§2.6失败模式反方体系第 47 层(跨域 harness 异质性);§2.207评测方法学第 29 元组候选

(iii) AgentDebug (GitHub ulab-uiuc/AgentDebug, UIUC NLP/SE Lab, MIT license, Jay 8-25 1450 工程筛选 ★★★) - 核心:AgentErrorTaxonomy 17 种错误 × 5 模块(memory/reflection/planning/action/system);AgentErrorBench 失败轨迹标注数据集;两阶段调试 pipeline(隔离根因 + 纠正反馈) - 归入:§2.207评测方法学第 31 元组候选;§6.79;§3.5 harness 生态六角延伸第 7 节点候选(错误归因 harness 层);§2.6失败模式反方体系第 48 层候选(17 种错误分类未公开 Cohen's kappa)

(iv) 41 种 Agent 失败模式分类学 (arXiv:2607.28802, Omar Sarheed @omarsar0 jay 8-25 X 硬核干货雷达, ★★评测方法学延革第 18 例实测预备) - 核心:41 种失败模式 × 六节点归因(model/harness/user/tools/memory/environment);Cohen's kappa 0.76 substantial agreement;5-30× 成本波动根因来自 harness 而非模型 - 归入:§2.207评测方法学第 30 元组候选;评测方法学延革第 18 例实测预备;反方立基础延革第 2 例预备

(v) OmniAssistBench (arXiv:2608.21360, paper_card 未建, HF Daily 8-25 #12 27▲ 新晋锚, ★) - 核心:Assistant-style Interaction Benchmark for Omni-LLMs;助手风格交互评测 = 助手类 Agent(客服/个人助理/协作写作)行为与工具执行类评测的维度差异 - 关键贡献:填补"助手风格交互评测"空白;多模态输入(图像+语音+文本)的助手交互给评测带来新维度 - 归入:§2.7评测对象第 89 维(助手风格交互评测);§2.2评测对象邻接(Assistant-style Interaction Benchmark)

(vi) Beyond Correctness (arXiv:2608.12781, paper_card 未建, HF Daily 8-25 #14 19▲ 新晋锚, ★) - 核心:Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs;超越正确性本身,聚焦"行为一致性/安全性/风格"等维度 - 关键贡献:首次系统性把"行为对齐"作为独立评测维度引入混合思维 MLLM;为"答案正确但行为不当"的评测盲区提供量化框架 - 归入:§2.7评测对象第 90 维(行为对齐评测);§2.1评测方法学邻接(行为对齐评测轴候选);§2.6失败模式反方体系邻接

(vii) Datadog State of AI Engineering 2026 (datadoghq.com/state-of-ai-engineering, Datadog Research, Jay 8-25 ★★★) - 核心数据:2026 年 2 月 LLM 调用 span 中 5% 报错(60% rate limit)+ 2026 年 3 月约 840 万次 rate limit - 归入:§2.1评测方法学邻接(生产环境错误率量化轴候选);§9 评测平台对比邻接

(viii) TraceCoder ICSE 2026 (Jay 8-25 awesome-harness-engineering 引用, ★★) - 核心:Instrumentation Agent → Analysis Agent(HLLM)→ Repair Agent(rollback);Pass@1 +34.43% - 归入:§2.207评测方法学第 32 元组候选

2.4 Judge & Harness 工程化方法(R56+2件)

R56新增2件(ReliabilityBench 四指标 + AgentDebug 17×5 错误分类)→ Judge & Harness 90 → 92件套;Judge 校准 35 → 36

2.5 评测方法学批判 / 反方(R56+2条,R56四十六层反方)

R56新增2条反方: - 反方R56-1:ReliabilityBench memory-augmented scaffold 普遍伤害长视性能(10 模型无一例外) — 与当下 memory-augmented agent 热潮(AgentDebug 17×5 错误分类中 memory 模块亦占 1 维)直接冲突;关键反方自己的反方:(a) 单模型 ≈2,300 samples 统计力可能不足;(b) 没有公开 leaderboard 复现门槛高;(c) cross-domain 推广需在 WebArena/OSWorld 等 rich tool space benchmark 上验证 - 反方R56-2:HORIZON inter-annotator κ=0.61 偏低(failure attribution 类目本身需要再校准)+ 评测模型仅 GPT-5 + Claude-4 variants(没有 open-source 队列,与 ReliabilityBench 形成强互补但也暴露各自盲点);关键反方自己的反方:(a) WebArena/AgentBench/MAC-SQL harness 差异大,跨域比较受异质性干扰;(b) trajectory-grounded LLM-judge 与 human-judge κ=0.84 差距说明 LLM-judge 还需要校准

2.6 失败模式分析(R56+3件五阶,失败模式反方体系 43 → 46 层)

R56新增3件五阶补充: - §2.6.46 ReliabilityBench VAF 统计力:单模型 ≈2,300 samples 对 VAF 这种方差指标可能不够(k=3 重复,VAF 估计有偏);扩展到更多样本或多模型验证才能确认 - §2.6.47 HORIZON 跨域 harness 异质性:WebArena / AgentBench / MAC-SQL / Isaac Sim 各 domain harness 差异大,跨域比较受异质性干扰;harness 标准化是前提 - §2.6.48 AgentDebug 17 种错误分类未公开 Cohen's kappa:与 41 种失败模式分类学(Cohen's kappa 0.76 substantial agreement)相比,17 种分类未公开一致性验证,工程可用性待核

2.7 评测对象分化(R56 终态:88 → 90维+39 → 41邻接维+135 → 140子领域)

R56新增维:OmniAssistBench(arXiv:2608.21360)助手风格交互评测 — 多智能体行为评测第 89 维;Beyond Correctness(arXiv:2608.12781)行为对齐评测 — 行为对齐评测第 90 维 R56新增邻接维:ReliabilityBench/HORIZON/AgentDebug/41种失败模式分类学 + OmniAssistBench/Beyond Correctness + TraceCoder/Datadog = 8件邻接 立标池饱和度 v44-v57 十四日连续

2.8 arXiv ID总索引(R56,257+5=262条)

R56新增5条核心:arXiv:2603.29231(ReliabilityBench R56反方立基础锚★★★ flyp critical-read)+arXiv:2604.11978(HORIZON R56长视Agent元评测★★★ COLM 2026)+arXiv:2607.28802(41种失败模式分类学 R56评测方法学延革第18例实测预备★★)+arXiv:2608.21360(OmniAssistBench R56新晋邻接★)+arXiv:2608.12781(Beyond Correctness R56新晋邻接★) + R56延续邻接:AgentDebug(GitHub ulab-uiuc/AgentDebug 无 arXiv)+ TraceCoder ICSE 2026(待补 arXiv)+ Datadog State of AI Engineering 2026(非 arXiv)+ 八件 8-25 新候选(EnSI-RAG/PV-SST/FlavourBench/Hydra-0/PhysCaP/SparsePR/Human-Centric Survey/UniSpace)

R56 missing=0 atomic验证:R55 missing集合全部包含;R56新增5条核心+3条延续邻接已含

3. 评测工程化:harness/judge/可信评测平台/CI gate/Harness自演进

3.1 Harness

R56 130 → 132件;§3.5 harness生态六角延伸(R56 AgentDebug 第 7 节点候选:错误归因harness层)

3.2 Judge

R56 90 → 92件套;Judge 校准 35 → 36

3.3 评测平台与CI Gate

R56 94 → 95条(新增 Datadog 生产 trace 数据接入)

3.4 EDD(Eval-Driven Development)

R56 102 → 104件(新增 AgentDebug + TraceCoder)

3.5 Harness生态六角延伸(R56 AgentDebug 候选)

StateM(执行层)+HarnessEval-W(评测层)+HarnessRisk(安全层)+Zetta ζ(具身self-evolution层)+HSI(任务特定可热替换harness层 R54新增)+EnvHarness(环境侧harness化 R54新增)+HarnessEval-W cs.CV视觉世界harness化 R55新增= 七角;AgentDebug(错误归因harness层 R56新增候选)作为第 7 节点候选;工具:uvx agent-harnesses-mcp

harness生态七角候选最终态:StateM(执行层)+HarnessEval-W(评测层)+HarnessRisk(安全层)+Zetta ζ(具身self-evolution层)+HSI(任务特定可热替换harness层)+EnvHarness(环境侧harness化)+HarnessEval-W cs.CV视觉世界harness化+AgentDebug 错误归因harness层(待入位)= 候选七角

4. 维度化指标体系(R56)

R56新增维度(在R21-R55 200行矩阵基础上,核心新增): - ReliabilityBench(arXiv:2603.29231,flyp 8-25 0507 critical-read落盘) ★★★ P0反方立基础锚 — RDC/VAF/GDS/MOP 四指标;memory-augmented scaffold 普遍伤害 10 模型无一例外 - HORIZON(arXiv:2604.11978,COLM 2026,flyp 8-25 critical-read落盘) ★★★ 长视Agent元评测 — 跨域诊断 + trajectory-grounded LLM-judge + inter-annotator κ=0.61 - AgentDebug(GitHub ulab-uiuc/AgentDebug,UIUC NLP/SE Lab,MIT license,Jay 8-25 1450) ★★★ — 17种错误×5模块两阶段调试 pipeline - 41种Agent失败模式分类学(arXiv:2607.28802,Omar Sarheed @omarsar0,jay 8-25 X硬核干货雷达) ★★评测方法学延革第18例实测预备 — 41种失败×六节点归因 + Cohen's kappa 0.76 + 5-30×成本波动根因 - OmniAssistBench(arXiv:2608.21360,HF Daily 8-25 #12 27▲ 新晋锚) ★ — 助手风格交互评测 - Beyond Correctness(arXiv:2608.12781,HF Daily 8-25 #14 19▲ 新晋锚) ★ — 行为对齐评测 - Datadog State of AI Engineering 2026(datadoghq.com,Datadog Research,Jay 8-25) ★★★ — 5% LLM调用报错+60% rate limit+840万次/月 第一方生产trace数据 - TraceCoder ICSE 2026(Jay 8-25 1450 awesome-harness-engineering 引用) ★★ — Instrumentation→Analysis HLLM→Repair rollback Pass@1 +34.43% - ASI-Bench(arXiv:2608.17271,paper_card未建) ★★衰减确认 — 8-22/8-23/8-24/8-25连续四日跌出

5. 评测对象分化(R56)

R56终态:88→90维+39→41邻接维+135→140子领域+3件P0反方立基础锚(ReliabilityBench+HORIZON+AgentDebug)+1件评测方法学延革第18例实测预备(41种失败模式分类学)+2件HF Daily 8-25新晋邻接(OmniAssistBench+Beyond Correctness)+2件工程筛选高价值(Datadog+TraceCoder)+延续信号(EnvHarness 258▲ 创历史新高+OmniAssistBench 27▲ 新晋+Beyond Correctness 19▲ 新晋+ASI-Bench连续四日跌出确认+SemComp-Bench/Zetta ζ/SemaPLC/Co-RL/SPADE五件衰减信号持续)

立标池第 14 日harness化方向+方法学质疑方向+多方向扩张(2026-08-22 → 2026-08-25): - EnvHarness 8-22 235▲ → 8-23 246▲ → 8-24 254▲ → 8-25 258▲(三/四日连续续立极显著,创历史新高) - OmniAssistBench 8-25 27▲(#12 新晋) - Beyond Correctness 8-25 19▲(#14 新晋) - ASI-Bench 8-22/8-23/8-24/8-25 连续四日跌出 = 衰减确认 - SemComp-Bench 8-23/8-24 持平 × 2 日 → 8-25 未列 = 衰减信号持续 - Zetta ζ 8-25 未列 = 衰减信号 - SemaPLC 8-25 未列 = 衰减信号 - 4 件跨日续立 = FACET/SWE-bench Science/MemTrapBench/SkillEvo harness生态高位稳定 - ReliabilityBench/HORIZON/AgentDebug P0反方立基础锚 + 41种失败模式分类学评测方法学延革第18例实测预备 + Datadog/TraceCoder工程筛选高价值 = 评测方法学延革第14-18例实测预备

harness生态七角候选:StateM(执行层)+HarnessEval-W(评测层)+HarnessRisk(安全层)+Zetta ζ(具身self-evolution层)+HSI(任务特定可热替换harness层)+EnvHarness(环境侧harness化)+HarnessEval-W cs.CV视觉世界harness化 + AgentDebug 错误归因harness层(R56 候选第七节点);立标池饱和度v44-v57十四日连续

6. 评测方法学批判(R56)

§6.1-§6.70:沿用R25-R55(见archive)

§6.71 HSI/Hierarchical Self-Improvement三层可热替换harness(R54新增)

[沿用R54内容]

§6.72 EnvHarness环境侧harness化(R54新增)

[沿用R54内容]

[沿用R54内容]

§6.74 HarnessEval-W 视觉世界agentified evaluation pipeline(R55新增)

[沿用R55内容]

§6.75 Large Discovery Models / LDM 经验驱动开放式搜索评测(R55新增)

[沿用R55内容]

§6.76 Harness-Evolution-Eval-Rethink 评测协议失真对位锚(R55新增)

[沿用R55内容]

§6.77 ReliabilityBench / Beyond pass@1: A Reliability Science Framework(R56新增)

核心:Reliability Decay Curve (RDC)= 成功率随时长衰减曲线;Variance Amplification Factor (VAF)= 跨重复方差放大倍数;Graceful Degradation Score (GDS)= 随时长性能软着陆程度(0=悬崖式,1=平稳下降);Meltdown Onset Point (MOP)= Meltdown 概率陡升临界时长 实验规模:396 tasks × 4 duration buckets × 3 domains × 10 开源模型 × 2 scaffolds(ReAct vs memory-augmented)× k=3 重复 = 23,392 episodes 5 个关键发现:(1) reliability decay domain-stratified:SE GDS 0.90→0.44;(2) VAF 按 capability 分层:高 VAF 是能力签名;(3) capability vs reliability 排序发散;(4) 前沿模型 meltdown 率 up to 19%;(5) memory-augmented scaffold 普遍伤害 10 模型无一例外 — 强证伪 立标信号:flyp 8-25 0507 critical-read落盘;P0 反方立基础锚候选 ★★★ 关键区分:与 41 种失败模式分类学(Cohen's kappa 0.76 substantial agreement)互补:ReliabilityBench 是 capability/reliability 二维元评测,41 种失败模式是六节点归因分类;两者结合 = "能做什么"+"为什么失败"+"cost 5-30× root cause" 反方意义:§2.5反方第 45 层(memory scaffold 普遍伤害);§2.6失败模式反方体系第 46 层(单模型 ≈2,300 samples VAF 统计力);评测方法学延革第 14 例实测;评测方法学社区内容补全七联立标第 7 件(ReliabilityBench);与 §2.5反方第 43 层 Zetta "11.1× 推理加速基线未明"+ §2.5反方第 44 层 SemaPLC "65 任务样本量小"+ §2.5反方第 42 层 Rethink "harness evolution budget-confounded" 形成"评测方法学反方四联立基础锚"

§6.78 HORIZON / The Long-Horizon Task Mirage(R56新增)

核心:跨域诊断 benchmark 框架 + trajectory-grounded LLM-as-a-Judge pipeline;inter-annotator κ=0.61, human-judge κ=0.84 评测范围:4 个代表性 agentic domains × 700+ tasks(WebArena 61 / AgentBench 27 / MAC-SQL 43 / Isaac Sim 18 + 自有)× 3,100+ trajectories × GPT-5 + Claude-4 variants 关键贡献:把 failure attribution 当成评测对象;公开 LLM-judge 校验流程;跨域比较:domain-specific vs horizon-specific 失败模式;公开 leaderboard(xwang2775.github.io/horizon-leaderboard) 立标信号:flyp 8-25 critical-read落盘;COLM 2026 接收;长视 Agent 元评测立基础锚候选 ★★★ 关键区分:与 ReliabilityBench 互补:ReliabilityBench 是 capability/reliability 二维元评测(10 模型队列),HORIZON 是跨域 failure attribution 元评测(GPT-5 + Claude-4 frontier);两者结合 = "reliability signature" + "failure attribution" 反方意义:§2.5反方第 46 层(HORIZON κ=0.61 归因类目校准 + frontier only);§2.6失败模式反方体系第 47 层(跨域 harness 异质性);评测方法学延革第 15 例实测;评测方法学社区内容补全七联立标第 7 件(HORIZON);与 ReliabilityBench(10 开源)+ HORIZON(GPT-5 + Claude-4)形成评测方法学反方立基础锚互补对

§6.79 AgentDebug / UIUC 17种错误×5模块两阶段调试 pipeline(R56新增)

核心:AgentErrorTaxonomy:17 种错误类型 × 5 模块(memory/reflection/planning/action/system);AgentErrorBench:ALFWorld、GAIA、WebShop 失败轨迹标注数据集;两阶段调试 pipeline:隔离根因 + 纠正反馈 Action 模块错误:misalignment、invalid_action、format_error、parameter_error;System 模块错误:step_limit、tool_execution_error、llm_limit、environment_error 立标信号:Jay 8-25 1450 工程筛选 ★★★;UIUC NLP/SE Lab;MIT license + 有源码和数据集;paper_card 待建 关键区分:与 41 种失败模式分类学(arXiv:2607.28802)对比:AgentDebug 17 种 × 5 模块(UIUC 工程导向,MIT license)+ 41 种 × 六节点归因(@omarsar0 学术导向,Cohen's kappa 0.76);两者结合 = AgentDebug 提供"工程调试 checklist"+ 41 种提供"学术分类法" 反方意义:§2.6失败模式反方体系第 48 层候选(17 种错误分类未公开 Cohen's kappa);评测方法学延革第 16 例实测;评测方法学社区内容补全七联立标第 7 件(AgentDebug);harness 生态七角候选第 7 节点(错误归因 harness 层);与 41 种失败模式分类学 Cohen's kappa 0.76 形成"工程可执行性 vs 学术严谨性"对照样本

§6.80 41种 Agent 失败模式分类学 / harness bug vs model bug 边界首次系统性定义(R56新增)

核心:41 种失败模式 × 六节点归因 model/harness/user/tools/memory/environment;Cohen's kappa 0.76 达到 substantial agreement(0.61-0.80 区间);harness bug vs model bug 边界首次系统性定义;5-30× 成本波动根因来自 harness 设计缺陷而非模型能力不足 立标信号:jay 8-25 X 硬核干货雷达干货候选;@omarsar0 Threads + arXiv:2607.28802;评测方法学延革第 18 例实测预备 ★★;反方立基础延革第 2 例预备 ★★ 关键区分:与 AgentDebug 对比:41 种 × 六节点(学术分类 + Cohen's kappa 0.76)+ AgentDebug 17 种 × 5 模块(UIUC 工程 + MIT license);两者都聚焦"Agent 失败分类"但深度和工程可用性互补 反方意义:§2.5反方第 46 层(HORIZON 反方 + ReliabilityBench 反方延展);§2.6失败模式反方体系第 47-48 层候选(41 种分类 Cohen's kappa 0.76 substantial 但非 perfect);评测方法学延革第 18 例实测预备;反方立基础延革第 2 例预备(5-30× 成本波动根因来自 harness 而非模型);与 ReliabilityBench memory scaffold 普遍伤害形成"双重反方立基础延展预备"

7. 共识、争议与开放问题

7.1 共识(41→43条,R56+2)

R56新增2条: - 共识R56-1:长视 Agent 元评测成为新一类论文,与 harness 化 + 评测协议失真 + 真实执行判定 三类同等重要 — ReliabilityBench(10 模型 capability/reliability 二维元评测)+ HORIZON(GPT-5 + Claude-4 frontier 跨域 failure attribution 元评测)+ LongShOTBench(多模态长时序列可靠性) 三件构成"长视 Agent 元评测三联立基础锚";评测社区长期默认"评测只看 pass@1"被强制要求引入 reliability signature(方差/衰减/软着陆/熔断阈值)四维指标 + failure attribution 类目 + cross-domain harness 标准化 - 共识R56-2:Agent 错误归因工程化进入主流,从学术分类走向工程可执行 checklist — 41 种 Agent 失败模式分类学(Cohen's kappa 0.76 substantial agreement) + AgentDebug 17×5 错误分类法(UIUC MIT license + 两阶段调试 pipeline) + Datadog State of AI Engineering 2026(生产 trace 第一方数据 5% 报错 60% rate limit) 三件构成"Agent 错误归因三联立基础锚";学术 vs 工程 + 公开分类法 vs 商用生产数据 + Cohen's kappa 一致性 vs 真实失败率 量化的对照样本

7.2 争议(98→101条,R56+3)

R56新增3条: - 争议R56-1:ReliabilityBench memory-augmented scaffold 普遍伤害长视性能(10 模型无一例外) — 与 AgentDebug 17×5 错误分类中 memory 模块占 1 维 + 当下 memory-augmented agent 热潮直接冲突;但反方自己的反方:(a) 单模型 ≈2,300 samples 对 VAF 这种方差指标可能统计力不足;(b) 没有公开 leaderboard 复现门槛偏高;(c) cross-domain 推广需在 WebArena/OSWorld/GAIA-2 等 rich tool space benchmark 上验证 - 争议R56-2:HORIZON inter-annotator κ=0.61 偏低(failure attribution 类目本身需要再校准)+ 评测模型仅 GPT-5 + Claude-4 variants 没有 open-source 队列 — 与 ReliabilityBench 10 开源模型形成强互补但也暴露各自盲点:ReliabilityBench 没有 frontier model,HORIZON 没有 open-source;但反方自己的反方:(a) WebArena/AgentBench/MAC-SQL/Isaac Sim 各 domain harness 差异大,跨域比较受异质性干扰;(b) trajectory-grounded LLM-judge 与 human-judge κ=0.84 差距说明 LLM-judge 还需要校准 - 争议R56-3:41 种 Agent 失败模式分类学 Cohen's kappa 0.76 是 substantial agreement 但非 perfect(0.61-0.80 区间),实际应用中需根据具体场景调整 — 5-30× 成本波动根因来自 harness 设计缺陷而非模型能力不足 是强反方叙事,但缺乏分 harness 类型统计 + 频次数据;与 AgentDebug 17×5 错误分类(UIUC MIT license 工程导向)对照:41 种学术严谨但工程可用性待核,17 种工程可用但未公开 Cohen's kappa 学术严谨性待核

7.3 开放问题(132→138条,R56+6)

R56新增6条: - 开放R56-1:ReliabilityBench 10 模型名单 + 3 domains 具体列表 + memory-augmented scaffold 实现差异 + memory 普遍伤害的统计显著性 PDF §4-5 验证(截止 9-5,P0) - 开放R56-2:HORIZON Leaderboard(https://xwang2775.github.io/horizon-leaderboard/ )可访问性 + 是否有 submission 流程 + 是否覆盖开源模型;inter-annotator κ=0.61 归因类目改进空间 - 开放R56-3:AgentDebug 17 种错误分类标准(是否有 Cohen's kappa 验证——与 41 种失败模式分类学 arXiv:2607.28802 Cohen's κ=0.76 是否有关联或重叠);ALFWorld / GAIA / WebShop 失败轨迹规模(具体多少条);与 41 种失败模式分类学的关系:17 种 vs 41 种,覆盖度差异 - 开放R56-4:41 种 Agent 失败模式完整列表 PDF §3-4 验证 + 5-30× 成本波动根因具体 harness 设计缺陷类型与频次统计 PDF §5 验证(截止 9-5,P1) - 开放R56-5:Datadog State of AI Engineering 2026 5% LLM 调用报错 + 60% rate limit + 840 万次/月 来源核实(Datadog 官博原文 + 完整图表数据截止 9-5) - 开放R56-6:OmniAssistBench(arXiv:2608.21360)Omni-LLM 具体涵盖哪些模态组合(文本+图像+语音?)+ 评测指标是否区分"任务完成率"和"交互风格自然度"两个维度;Beyond Correctness(arXiv:2608.12781)"行为对齐"具体评测指标是否有人类标注一致性验证 + Hybrid-thinking MLLM 的定义边界是否清晰

R56 六项硬约束:a)8-26 09:00 HF Daily 复核 EnvHarness 是否续立创历史新高;OmniAssistBench + Beyond Correctness 双新晋锚是否进入次日 top 15;b)ReliabilityBench PDF §4-5 验证 memory-augmented scaffold 实现细节(截止 9-5);c)HORIZON Leaderboard 可访问性 + submission 流程验证(截止 9-5);d)AgentDebug 17×5 错误分类与 41 种失败模式分类学关系核实(截止 9-5);e)Datadog 5% 报错 + 60% rate limit 来源独立核实(截止 9-5);f)OmniAssistBench + Beyond Correctness 评测指标 PDF 核验(截止 9-5)

8. 趋势预判(2026 H2,R56)

R56趋势主线(R55 9条 + R56 +5): 1. harness化方向结构性扩张:从六角→七角候选:StateM/HarnessEval-W/HarnessRisk/Zetta ζ 四角 + HSI 第 5 节点 + EnvHarness 第 6 节点 + HarnessEval-W cs.CV视觉世界harness化 第 7 节点(R55)+ AgentDebug 错误归因harness层 第 8 节点候选(R56) = 候选八角 2. 长视 Agent 元评测成为新一类论文:ReliabilityBench(10 模型 capability/reliability 二维元评测)+ HORIZON(GPT-5 + Claude-4 frontier 跨域 failure attribution 元评测)+ LongShOTBench(多模态长时序列可靠性) 三件构成"长视 Agent 元评测三联立基础锚" 3. 立标池双向锚第 14 日结构性切换:EnvHarness 8-25 258▲ #1 创历史新高(+4▲ vs +8▲,+11▲ 起始)+ 4 件跨日续立 + ASI-Bench 连续四日跌出 + SemComp-Bench/Zetta ζ/SemaPLC/Co-RL/SPADE 五件衰减信号持续 = harness 生态高位稳定 + 反方持续形成 4. 评测协议失真 + 真实执行判定 + 可靠性科学 + 错误归因工程化 四轴同步扩张:Rethink controlled budget protocol + Terminal-Bench 算力等价质疑(协议失真)+ Zetta+SemaPLC 真实执行判定(执行即真相)+ ReliabilityBench memory scaffold 普遍伤害(可靠性反方)+ HORIZON failure attribution(归因元评测)+ AgentDebug 17×5 错误分类(归因工程化)+ 41 种失败模式分类学(归因学术化)+ Datadog 生产错误率量化(生产落地) 5. memory scaffold 反直觉 + harness bug vs model bug 边界首次系统性定义:ReliabilityBench memory-augmented scaffold 普遍伤害 10 模型无一例外 + 41 种失败模式 5-30× 成本波动根因来自 harness 而非模型 = "双重反方立基础延展预备" + "harness 是护城河"立论的强证伪信号 6. Agent 错误归因从学术走向工程:41 种 Agent 失败模式分类学(Cohen's kappa 0.76 substantial agreement) + AgentDebug 17×5 错误分类法(UIUC MIT license + 两阶段调试 pipeline) + Datadog State of AI Engineering 2026(生产 trace 第一方数据) 三件构成"Agent 错误归因三联立基础锚" 7. OmniAssistBench + Beyond Correctness 评测范式扩展(R56):助手风格交互评测 + 行为对齐评测 = 评测对象 88→90维新晋 + HF Daily 8-25 双新晋锚 #12/#14 8. 生产环境错误率量化(Datadog 5% 报错):第一方生产 trace 数据接入评测平台对比 = eval-as-arch 落地 9. 多 Agent 调试工程化(TraceCoder ICSE 2026 Pass@1 +34.43%):Instrumentation→Analysis HLLM→Repair rollback = AgentDebug 工程化扩展

9. 平台、基础设施与生态

9.1 评测平台总览

32平台(R56 0新增)

9.2 评测harness/benchmark套件

237件(R56 +4:ReliabilityBench/HORIZON/AgentDebug/41种失败模式分类学)

9.3 评测方法学toolkit

103件(R56 +6:ReliabilityBench/HORIZON/AgentDebug/41种失败模式分类学/OmniAssistBench/Beyond Correctness)

9.4 EDD案例库

104件(R56 +2:AgentDebug/TraceCoder)

9.5 评测平台对比

90维+41邻接维(R56 +2维:OmniAssistBench/Beyond Correctness + +2邻接:AgentDebug/TraceCoder)

9.6 Harness生态七角候选(R56 AgentDebug 第 7 节点候选)

StateM(执行层)+HarnessEval-W(评测层)+HarnessRisk(安全层)+Zetta ζ(具身self-evolution层)+HSI(任务特定可热替换harness层)+EnvHarness(环境侧harness化 R54新增)+HarnessEval-W cs.CV视觉世界harness化 R55新增+AgentDebug 错误归因harness层 R56候选第七节点= 候选八角;工具:uvx agent-harnesses-mcp

历史arXiv ID回引(从R00-R55继承,atomic missing=0保障):arXiv:2501.05444(EMMA)+arXiv:2512.02282+arXiv:2507.00310(AutoDiscovery NeurIPS 2025)+arXiv:2512.14629(MuseCP R53立基础锚★★★ paper_card 1036)+arXiv:2601.04043+arXiv:2601.14242(APEX-Agents)+arXiv:2603.08835(MASEval)+arXiv:2603.29231(ReliabilityBench R56反方立基础锚★★★)+arXiv:2604.11978(HORIZON R56长视Agent元评测★★★)+arXiv:2605.10286(AgentRx R49)+arXiv:2605.18032+arXiv:2605.23950(Stop Comparing)+arXiv:2605.27922(Harness-Bench)+arXiv:2605.30434+arXiv:2606.08367+arXiv:2606.14747(MMLongEmbed)+arXiv:2607.12227(Rethink R55方法学质疑对位)+arXiv:2607.27616+arXiv:2607.27816+arXiv:2607.27853+arXiv:2607.27888+arXiv:2607.28609+arXiv:2607.28661+arXiv:2607.28802(41种失败模式分类学 R56评测方法学延革第18例实测预备★★)+arXiv:2607.28887+arXiv:2607.29241+arXiv:2607.29677(ExtractBench)+arXiv:2608.00267(LoopsBench)+arXiv:2608.00677(OpenART)+arXiv:2608.01964(LongHorizon-Harness)+arXiv:2608.03451(DataSpace)+arXiv:2608.03507(ChronoLens)+arXiv:2608.03700(AntiSkillBench)+arXiv:2608.03764(GDPevo)+arXiv:2608.04003(PAST-Bench)+arXiv:2608.04205(MatrAIx)+arXiv:2608.04302(CLIP-CC-Bench)+arXiv:2608.04397(NOLLI)+arXiv:2608.05013(OneDayAgent)+arXiv:2608.05139(Skill²-Bench)+arXiv:2608.05747(GST-Bench)+arXiv:2608.05850(MameLoshnLM)+arXiv:2608.06301(HarnessOpt-Bench)+arXiv:2608.06312(GB/T-Bench)+arXiv:2608.06329(Benchmarking the Benchmarks)+arXiv:2608.06614+arXiv:2608.06729+arXiv:2608.06867(LLMRouter)+arXiv:2608.07545(DarwinX)+arXiv:2608.08119(TSDS-Toolbox)+arXiv:2608.08160(Can LLM Agents Stick to the Script?)+arXiv:2608.08311(Ouroboros)+arXiv:2608.08466(HSI R54立基础锚★★★ paper_card 1057)+arXiv:2608.08722(Benchmark Fingerprinting)+arXiv:2608.08814(360CityArena)+arXiv:2608.08975+arXiv:2608.09096(Evo-Bench)+arXiv:2608.09158+arXiv:2608.09766(Cultivar)+arXiv:2608.09802(SWE-Bench ProMax)+arXiv:2608.09805+arXiv:2608.09867(Stealing Reasoning Traces)+arXiv:2608.09888(BDH-CQ)+arXiv:2608.09900(Decoding-Level Taboo)+arXiv:2608.10744(Latent-to-4D)+arXiv:2608.10708+arXiv:2608.10875(VibeLifeBench)+arXiv:2608.11341(Apodex Discovery)+arXiv:2608.11745(LiveAnimate)+arXiv:2608.11947+arXiv:2608.12036(Mechanist)+arXiv:2608.12149+arXiv:2608.12314(StateFlow)+arXiv:2608.12571(Is this Citation on Point?)+arXiv:2608.12743+arXiv:2608.12781(Beyond Correctness R56新晋邻接★)+arXiv:2608.12875(Embedder's Dilemma R55复用 paper_card 1055)+arXiv:2608.13010+arXiv:2608.13120(SkillEvo R54邻接)+arXiv:2608.14022(ForgeWM)+arXiv:2608.18613(CTIFoundry)+arXiv:2608.13558(OmniScientist)+arXiv:2608.18940+arXiv:2608.13160+arXiv:2608.13417(超越最终分数)+arXiv:2608.13489+arXiv:2608.13505+arXiv:2608.13546+arXiv:2608.13547(QuoteBench R54邻接 paper_card 1056)+arXiv:2608.13552(PlayWorld)+arXiv:2608.13560(AutoDesign)+arXiv:2608.13588+arXiv:2608.13606(MobileMem)+arXiv:2608.13951(HELIX R54 web_search 隐含发现)+arXiv:2608.14036(Demystifying Agent Skills)+arXiv:2608.14106(Forecast Collapse)+arXiv:2608.14284(PRM-as-a-Judge 1.5)+arXiv:2608.14457(信息满足度)+arXiv:2608.14546(CPI-Bench)+arXiv:2608.14905(ARFT R53 web_search 验证)+arXiv:2608.15022+arXiv:2608.15089(StateM)+arXiv:2608.15669(LDM R55立基础锚★★★ paper_card 998)+arXiv:2608.16590(Zetta ζ R53立基础锚★★★ paper_card 1027)+arXiv:2608.16798(ClawGym II)+arXiv:2608.16859(HarnessEval-W R55立基础锚★★★ paper_card 992)+arXiv:2608.17253(Co-RL)+arXiv:2608.17271(ASI-Bench)+arXiv:2608.17379(PTXBench)+arXiv:2608.17393(LEGO-RL)+arXiv:2608.17426(SemComp-Bench paper_card 1026 multimodal主分类)+arXiv:2608.17528(Agent Lightning v1.0)+arXiv:2608.17597(HarnessRisk)+arXiv:2608.17950(Six Degrees)+arXiv:2608.18077(Hydra-0)+arXiv:2608.18184(Human-Centric Survey)+arXiv:2608.18489(MissDiag paper_card 1034)+arXiv:2608.18565(SemaPLC R53邻接)+arXiv:2608.18580(FACET R54邻接)+arXiv:2608.18613(CTIFoundry)+arXiv:2608.18701(SoftVTBench R53立基础锚★★★ paper_card 1025)+arXiv:2608.19197(SPADE)+arXiv:2608.19799(SWE-bench Science R53邻接)+arXiv:2608.19854(Repo0)+arXiv:2608.19880(EnvHarness R54立标级新锚★★★ paper_card未建)+arXiv:2608.20169(Task-CoEvolve R55新晋邻接)+arXiv:2608.20202(MemTrapBench R54邻接)+arXiv:2608.20335(4DAnyone)+arXiv:2608.20336(WithEveryone)+arXiv:2608.20438(PV-SST R56新晋邻接)+arXiv:2608.20574(FlavourBench R56新晋邻接)+arXiv:2608.21031(PhysCaP)+arXiv:2608.21156(Graph Engineering)+arXiv:2608.21208(SpecPort)+arXiv:2608.21249(Dis2Pat paper_card 1060)+arXiv:2608.21252(EnSI-RAG paper_card 1058)+arXiv:2608.21360(OmniAssistBench R56新晋邻接★)+arXiv:2608.21159(AID-Guard paper_card 1059)+arXiv:2608.20910(InfinityEdit)+arXiv:2608.16425(ParaTempo)+arXiv:2604.10352(ClawVM R55新晋邻接 EuroMLSys '26)+arXiv:2607.08028(From Prompts to Contracts R55新晋邻接)+arXiv:2608.14022(ForgeWM R55延续)+arXiv:2608.18484(SparsePR)+arXiv:2608.08676(UniSpace)

2.8.1 R48 §2.8.1 历史链接 artifact 标注保留(原文逐字保留以满足 atomic missing=0 协议)

R48 §2.8.1 历史链接 artifact 标注保留(原文逐字保留以满足 atomic missing=0 协议):https://github.com/zjunlp/OneDayAgent;(xiii / https://lilianweng.github.io/posts/2026-07-04-harness/(邻接 / https://lilianweng.github.io/posts/2026-07-04-harness/(非 / https://arxiv.org/abs/2608.09888 (BDH-CQ R46 立标池双向锚 衰减锚链接,plain URL 引用补全)。

2.8.2 R00-R56 续)

R00-R55 历史 URL 156 条 + R56 新增 2 条 = 158 条 URL 全保留:沿用 R55 156 条 + R56 新增 https://github.com/ulab-uiuc/AgentDebug(AgentDebug R56 GitHub 仓库)+ https://xwang2775.github.io/horizon-leaderboard(HORIZON R56 公开 leaderboard)。

https://arxiv.org/abs/2603.29231 + https://arxiv.org/abs/2604.11978 + https://arxiv.org/abs/2607.12227 + https://arxiv.org/abs/2607.28802 + https://arxiv.org/abs/2608.12875 + https://arxiv.org/abs/2608.13417 + https://mirros-lab.github.io/HarnessEval-W + https://arxiv.org/abs/2607.12227(Harness-Evolution-Eval-Rethink R55方法学质疑对位立基础锚 flyp critical-read 8-22) + https://arxiv.org/abs/2608.12875(The Embedder's Dilemma R55复用 paper_card 1055) + https://arxiv.org/abs/2608.13417(超越最终分数 R50第80维立标级) + https://mirros-lab.github.io/HarnessEval-W(R55新增 HarnessEval-W 项目页面 web_search验证) + https://mirros-lab.github.io/HarnessEval-W(HarnessEval-W R55 项目页面 web_search验证) + https://mirros-lab.github.io/HarnessEval-W(HarnessEval-W R55新增) + https://21yrm.github.io/Apple-PI-homepage + https://aclanthology.org/2026.acl-demo.19/ + https://aclanthology.org/2026.acl-demo.3/ + https://aclanthology.org/2026.acl-demo.34/ + https://aclanthology.org/volumes/2026.acl-demo + https://aiweekly.co/alerts/acl-2026-publishes-demo-track-list-heavy-on-llm-safety-tools + https://aiweekly.co/alerts/coercion-benchmark-claude-never-threatens-deletion-rivals-do + https://alexeyondata.substack.com/p/what-1000-job-descriptions-reveal + https://artificialanalysis.ai/evaluations/apex-agents-aa + https://arxiv.org/abs/2501.05444 + https://arxiv.org/abs/2512.02282 + https://arxiv.org/abs/2601.04043 + https://arxiv.org/abs/2601.14242 + https://arxiv.org/abs/2603.08835 + https://arxiv.org/abs/2603.29231(ReliabilityBench R56 反方立基础锚★★★) + https://arxiv.org/abs/2604.11978(HORIZON R56 长视Agent元评测★★★) + https://arxiv.org/abs/2605.10286(AgentRx R49) + https://arxiv.org/abs/2605.18032 + https://arxiv.org/abs/2605.23950 + https://arxiv.org/abs/2605.27922 + https://arxiv.org/abs/2605.30434 + https://arxiv.org/abs/2606.08367 + https://arxiv.org/abs/2606.14747(MMLongEmbed) + https://arxiv.org/abs/2607.12227(Rethink) + https://arxiv.org/abs/2607.27616 + https://arxiv.org/abs/2607.27816 + https://arxiv.org/abs/2607.27853 + https://arxiv.org/abs/2607.27888 + https://arxiv.org/abs/2607.28609 + https://arxiv.org/abs/2607.28661 + https://arxiv.org/abs/2607.28802(41种失败模式分类学 R56 评测方法学延革第18例实测预备★★) + https://arxiv.org/abs/2607.28887 + https://arxiv.org/abs/2607.29241 + https://arxiv.org/abs/2607.29677(ExtractBench) + https://arxiv.org/abs/2608.00267 + https://arxiv.org/abs/2608.00677(OpenART) + https://arxiv.org/abs/2608.01964 + https://arxiv.org/abs/2608.03451 + https://arxiv.org/abs/2608.03507 + https://arxiv.org/abs/2608.03700 + https://arxiv.org/abs/2608.03764 + https://arxiv.org/abs/2608.04003 + https://arxiv.org/abs/2608.04205 + https://arxiv.org/abs/2608.04302 + https://arxiv.org/abs/2608.04397 + https://arxiv.org/abs/2608.05013 + https://arxiv.org/abs/2608.05139 + https://arxiv.org/abs/2608.05747 + https://arxiv.org/abs/2608.05850 + https://arxiv.org/abs/2608.06301 + https://arxiv.org/abs/2608.06312 + https://arxiv.org/abs/2608.06329 + https://arxiv.org/abs/2608.06614 + https://arxiv.org/abs/2608.06729 + https://arxiv.org/abs/2608.06867(LLMRouter) + https://arxiv.org/abs/2608.07545(DarwinX) + https://arxiv.org/abs/2608.08119(TSDS-Toolbox) + https://arxiv.org/abs/2608.08160(Can LLM Agents Stick to the Script?) + https://arxiv.org/abs/2608.08311 + https://arxiv.org/abs/2608.08466(HSI R54 立基础锚★★★) + https://arxiv.org/abs/2608.08722 + https://arxiv.org/abs/2608.08814(360CityArena) + https://arxiv.org/abs/2608.08975 + https://arxiv.org/abs/2608.09096 + https://arxiv.org/abs/2608.09158 + https://arxiv.org/abs/2608.09766 + https://arxiv.org/abs/2608.09802 + https://arxiv.org/abs/2608.09805 + https://arxiv.org/abs/2608.09867(Stealing Reasoning Traces) + https://arxiv.org/abs/2608.09888(BDH-CQ) + https://arxiv.org/abs/2608.09900(Decoding-Level Taboo) + https://arxiv.org/abs/2608.10744(Latent-to-4D) + https://arxiv.org/abs/2608.10708 + https://arxiv.org/abs/2608.10875(VibeLifeBench) + https://arxiv.org/abs/2608.11341(Apodex) + https://arxiv.org/abs/2608.11745(LiveAnimate) + https://arxiv.org/abs/2608.12036(Mechanist) + https://arxiv.org/abs/2608.12149 + https://arxiv.org/abs/2608.12314(StateFlow) + https://arxiv.org/abs/2608.12571(Is this Citation on Point?) + https://arxiv.org/abs/2608.12743 + https://arxiv.org/abs/2608.12781(Beyond Correctness R56 新晋邻接★) + https://arxiv.org/abs/2608.12875(Embedder's Dilemma) + https://arxiv.org/abs/2608.13010 + https://arxiv.org/abs/2608.13120(SkillEvo) + https://arxiv.org/abs/2608.13160 + https://arxiv.org/abs/2608.13417 + https://arxiv.org/abs/2608.13489 + https://arxiv.org/abs/2608.13505 + https://arxiv.org/abs/2608.13546 + https://arxiv.org/abs/2608.13547(QuoteBench) + https://arxiv.org/abs/2608.13552(PlayWorld) + https://arxiv.org/abs/2608.13560(AutoDesign) + https://arxiv.org/abs/2608.13588 + https://arxiv.org/abs/2608.13606(MobileMem) + https://arxiv.org/abs/2608.13951(HELIX) + https://arxiv.org/abs/2608.14036 + https://arxiv.org/abs/2608.14106(Forecast Collapse) + https://arxiv.org/abs/2608.14284(PRM-as-a-Judge 1.5) + https://arxiv.org/abs/2608.14457(信息满足度) + https://arxiv.org/abs/2608.14546(CPI-Bench) + https://arxiv.org/abs/2608.14905(ARFT) + https://arxiv.org/abs/2608.15022 + https://arxiv.org/abs/2608.15089(StateM) + https://arxiv.org/abs/2608.15669(LDM R55 立基础锚★★★) + https://arxiv.org/abs/2608.16590(Zetta ζ R53 立基础锚★★★) + https://arxiv.org/abs/2608.16798(ClawGym II) + https://arxiv.org/abs/2608.16859(HarnessEval-W R55 立基础锚★★★) + https://arxiv.org/abs/2608.17253(Co-RL) + https://arxiv.org/abs/2608.17271(ASI-Bench) + https://arxiv.org/abs/2608.17379(PTXBench) + https://arxiv.org/abs/2608.17393(LEGO-RL) + https://arxiv.org/abs/2608.17426(SemComp-Bench) + https://arxiv.org/abs/2608.17528(Agent Lightning v1.0) + https://arxiv.org/abs/2608.17597(HarnessRisk) + https://arxiv.org/abs/2608.17950(Six Degrees) + https://arxiv.org/abs/2608.18489(MissDiag) + https://arxiv.org/abs/2608.18565(SemaPLC) + https://arxiv.org/abs/2608.18580(FACET) + https://arxiv.org/abs/2608.18701(SoftVTBench R53 立基础锚★★★) + https://arxiv.org/abs/2608.19799(SWE-bench Science) + https://arxiv.org/abs/2608.19880(EnvHarness R54 立标级新锚★★★) + https://arxiv.org/abs/2608.20169(Task-CoEvolve) + https://arxiv.org/abs/2608.20202(MemTrapBench) + https://arxiv.org/abs/2608.20335(4DAnyone) + https://arxiv.org/abs/2608.20336(WithEveryone) + https://arxiv.org/abs/2608.20438(PV-SST R56 新晋邻接) + https://arxiv.org/abs/2608.20574(FlavourBench R56 新晋邻接) + https://arxiv.org/abs/2608.21360(OmniAssistBench R56 新晋邻接★) + https://arxiv.org/abs/2608.21208(SpecPort) + https://arxiv.org/abs/2608.21249(Dis2Pat) + https://arxiv.org/abs/2608.21252(EnSI-RAG) + https://arxiv.org/abs/2608.21159(AID-Guard) + https://arxiv.org/abs/2608.21156(Graph Engineering) + https://arxiv.org/abs/2608.21031(PhysCaP) + https://arxiv.org/abs/2608.20910(InfinityEdit) + https://arxiv.org/abs/2608.16425(ParaTempo) + https://arxiv.org/abs/2608.19854(Repo0) + https://arxiv.org/abs/2608.14022(ForgeWM) + https://arxiv.org/abs/2608.18484(SparsePR) + https://arxiv.org/abs/2608.18077(Hydra-0) + https://arxiv.org/abs/2608.18184(Human-Centric Survey) + https://arxiv.org/abs/2608.08676(UniSpace) + https://arxiv.org/abs/2604.10352(ClawVM) + https://arxiv.org/abs/2607.08028(From Prompts to Contracts) + https://arxiv.org/abs/2512.14629(MuseCP) + https://arxiv.org/html/2608.06312v1 + https://arxiv.org/html/2608.10875v1(VibeLifeBench) + https://arxiv.org/html/2608.14905v1(ARFT) + https://cameronrwolfe.substack.com/p/agent-evals + https://futureagi.substack.com/p/the-complete-guide-to-llm-evaluation + https://github.com/21yrm/Apple-PI + https://github.com/AmamiSora1228/MMLongEmbed + https://github.com/Gen-Verse/Skill-Entropy-RL(Skill²-Bench) + https://github.com/ModalityDance/Awesome-Agent-as-a-Judge + https://github.com/ai-boost/awesome-harness-engineering + https://github.com/hkust-nlp/LOCA-bench + https://github.com/maseval/MASEval + https://github.com/pathwaycom/bdh + https://github.com/run-llama/ExtractBench + https://github.com/ulab-uiuc/AgentDebug(R56 新增 GitHub 仓库) + https://github.com/zjunlp/DataMind + https://github.com/zjunlp/OneDayAgent + https://huggingface.co/blog/security-incident-july-2026 + https://huggingface.co/datasets/mercor/apex-agents + https://huggingface.co/papers/2608.00677(OpenART) + https://huggingface.co/papers/2608.05747 + https://huggingface.co/papers/2608.06867(LLMRouter) + https://huggingface.co/papers/2608.07545(DarwinX) + https://huggingface.co/papers/2608.08160(Can LLM Agents Stick to the Script?) + https://huggingface.co/papers/2608.09867(Stealing Reasoning Traces) + https://huggingface.co/papers/2608.10875(VibeLifeBench) + https://huggingface.co/papers/2608.11745(LiveAnimate) + https://huggingface.co/papers/2608.13552(PlayWorld) + https://huggingface.co/papers/2608.13560(AutoDesign) + https://huggingface.co/papers/2608.14905(ARFT) + https://huggingface.co/papers/2608.16859(HarnessEval-W) + https://hugobowne.substack.com/p/stop-overengineering-your-agent-harness + https://hyper.ai/en/papers/2608.10875 + https://lilianweng.github.io/posts/2026-07-04-harness/ + https://maseval.readthedocs.io + https://mirros-lab.github.io/HarnessEval-W + https://openreview.net/forum?id=EPQi0v0OxL(MMLongBench) + https://randcorporation.github.io/judge-reliability-harness + https://stolen-thoughts.com + https://theaiengineer.substack.com/ + https://usersim.ai/bibliography + https://www.actian.com/blog/databases/how-to-evaluate-vector-databases-in-2026 + https://www.alphaxiv.org/abs/2608.05747 + https://www.alphaxiv.org/abs/2608.09867(Stealing Reasoning Traces) + https://www.braintrust.dev/articles/best-ai-agent-debugging-tools-2026 + https://www.datadoghq.com/state-of-ai-engineering(R56 新增 Datadog State of AI Engineering) + https://www.freecodecamp.org/news/ai-evaluation-engineering-build-a-production-grade-llm-evaluation-platform-handbook + https://www.mercor.com/blog/introducing-apex-agents + https://www.salttechno.ai/datasets/vector-database-performance-benchmark-2026 + https://www.vibeleaderboard.ai/intel/3eb3f658-723d-4c4c-92ce-0439fbd62c41 + https://x.com/omarsar0/status/2038627572108743001(异步协调编码 Agent) + https://x.com/jerryjliu0/status/2090204727091093841(ExtractBench LlamaIndex) + https://xwang2775.github.io/horizon-leaderboard(R56 新增 HORIZON leaderboard) + https://academ.us/article/2608.10875 + https://papers.cool/arxiv/2608.14905(ARFT) + https://huggingface.co/papers/2608.14905(ARFT) R56新增arXiv ID回引(3件P0反方立基础锚+1件评测方法学延革第18例实测预备+2件新晋邻接+1件工程筛选高价值): arXiv:2603.29231(ReliabilityBench R56反方立基础锚★★★ flyp critical-read)+arXiv:2604.11978(HORIZON R56长视Agent元评测★★★ COLM 2026)+arXiv:2607.28802(41种Agent失败模式分类学 R56评测方法学延革第18例实测预备★★ 反方立基础延革第2例预备)+arXiv:2608.21360(OmniAssistBench R56新晋邻接★)+arXiv:2608.12781(Beyond Correctness R56新晋邻接★)+AgentDebug(GitHub ulab-uiuc/AgentDebug Jay 8-25 ★★★ R56非arXiv)+TraceCoder(ICSE 2026 Jay 8-25 ★★ R56非arXiv)+Datadog State of AI Engineering 2026(datadoghq.com Jay 8-25 ★★★ R56非arXiv)

10. 反问与遗留清单(R56)

10.1 本主题内反问

115→121条(R56 +6:ReliabilityBench 10 模型名单 + HORIZON Leaderboard 可访问性 + AgentDebug 17×5 Cohen's kappa + 41 种失败模式完整列表 + Datadog 5% 报错溯源 + OmniAssistBench + Beyond Correctness 评测指标)

10.2 待补/待核验(R56 +6,共211条)

R56新增:ReliabilityBench(arXiv:2603.29231)PDF §4-5 验证 10 模型 + 3 domains + memory-augmented scaffold 实现细节(截止 9-5);HORIZON(arXiv:2604.11978)Leaderboard 可访问性 + submission 流程 + open-source 模型覆盖(截止 9-5);AgentDebug(arXiv 缺失)17×5 Cohen's kappa 验证 + 与 41 种失败模式 arXiv:2607.28802 关系(截止 9-5);41 种 Agent 失败模式分类学(arXiv:2607.28802)41 种完整列表 PDF §3-4 + 5-30× 成本波动根因频次统计 PDF §5(截止 9-5);Datadog State of AI Engineering 2026 5% 报错 + 60% rate limit 来源独立核实(截止 9-5);OmniAssistBench(arXiv:2608.21360)+ Beyond Correctness(arXiv:2608.12781)评测指标 PDF 核验(截止 9-5);HarnessEval-W paper_card 992 ✅已建;LDM paper_card 998 ✅已建;Embedder's Dilemma paper_card 1055 ✅已建;Rethink flyp critical-read已落盘(无paper_card);Zetta+SemaPLC flyp critical-read已落盘(无paper_card);ASI-Bench(arXiv:2608.17271)paper_card ❌仍未建(跨轮 R53→R56 延续);arXiv:2608.13417 paper_card ❌仍未建(跨轮 R50→R56 延续);AgentRx(arXiv:2605.10286)paper_card ❌仍未建(跨轮延续);MMLongEmbed(arXiv:2606.14747)paper_card ❌仍未建(跨轮延续);ReliabilityBench paper_card ❌仍未建(R56新增);HORIZON paper_card ❌仍未建(R56新增);AgentDebug paper_card ❌仍未建(R56新增);41 种失败模式 paper_card ❌仍未建(R56新增);OmniAssistBench paper_card ❌仍未建(R56新增);Beyond Correctness paper_card ❌仍未建(R56新增);HSI paper_card 1057 ✅已建;EnvHarness paper_card未建(R54 立基础锚★★★但 paper_card 仍未补);SemaPLC paper_card未建(R53 邻接但 paper_card 仍未补)

10.3 与其他主题交叉(R56 +8,共31件)

ReliabilityBench(capability/reliability 二维元评测)→agent.md(长视Agent 元评测)+multimoding.md(长时序列可靠性);HORIZON(跨域诊断 trajectory-grounded LLM-judge)→agent.md(长视Agent 元评测)+coding-agents.md(failure attribution 类目);AgentDebug(17×5 错误分类法)→agent.md(Agent 错误分类)+risk.md(失败模式溯源);41 种 Agent 失败模式分类学(harness bug vs model bug 边界)→agent.md(harness 设计缺陷)+coding-agents.md(成本波动根因);OmniAssistBench(助手风格交互评测)→agent.md(助手类 Agent)+multimodal.md(多模态助手交互);Beyond Correctness(行为对齐评测)→agent.md(行为对齐)+rag.md(行为对齐 RAG);Datadog State of AI Engineering 2026(生产 trace 数据)→agent.md(生产 LLM 调用)+inference.md(生产 LLM 推理);TraceCoder ICSE 2026(多 Agent 调试)→coding-agents.md(ICSE 2026)+engineering.md(Agent 调试)

本次变更

  • 2026-08-25 16:50(R56,本轮新增):针对 2026-08-23 16:50 → 2026-08-25 16:50 48h 窗口期材料聚焦 ReliabilityBench+HORIZON 长视元评测立基础锚+AgentDebug 17×5错误分类法 P0反方锚+OmniAssistBench+Beyond Correctness 行为对齐新锚+EnvHarness 258▲历史新高。具体变更:(i)整篇覆盖写入;(ii)首行更新 R55 → R56 一句话;(iii)§0 范式跃迁 R55 四十五 → R56 四十六重(R56 +1 重:ReliabilityBench+HORIZON 长视元评测 + AgentDebug 错误归因工程化 + 41 种失败模式分类学 + OmniAssistBench + Beyond Correctness);(iv)§1 评测对象 88→90维+39→41邻接维+135→140子领域;(v)§2.1 评测元方法学 97→103件套 + 评测方法学 35 轴维持 + 评测方法学社区内容补全五联→七联立标(R56 第7件 ReliabilityBench/HORIZON/AgentDebug/41种失败模式/Datadog/TraceCoder);(vi)§2.2 立标池双向锚第 14 日信号:EnvHarness 8-25 258▲ 创历史新高 + OmniAssistBench 27▲ 新晋 + Beyond Correctness 19▲ 新晋 + ASI-Bench 连续四日跌出确认 + SemComp-Bench/Zetta ζ/SemaPLC/Co-RL/SPADE 五件衰减信号持续 + 4 件跨日续立;(vii)§2.3 指标 R56 +9 件核心新增(3 P0反方立基础锚 ReliabilityBench/HORIZON/AgentDebug + 1 评测方法学延革第18例 41种失败模式 + 2 新晋邻接 OmniAssistBench/Beyond Correctness + 2 工程筛选高价值 Datadog/TraceCoder + 1 衰减确认 ASI-Bench);(viii)§2.4 Judge&Harness 90→92件套 + Judge 校准 35→36 + 评测基础套件 83→87;(ix)§2.5 反方 +2 条(R56 四十六层:§2.5.45 ReliabilityBench memory scaffold 普遍伤害 + §2.5.46 HORIZON κ=0.61 归因类目校准);(x)§2.6 失败模式 43→46 层(R56 +3 五阶:§2.6.46 ReliabilityBench VAF 统计力 + §2.6.47 HORIZON 跨域 harness 异质性 + §2.6.48 AgentDebug 17×5 未公开 Cohen's kappa);(xi)§2.7 评测对象 88→90 维 + 39→41 邻接维 + 135→140 子领域;(xii)§3.5 harness 生态六角延伸七角候选(AgentDebug 第 7 节点候选:错误归因 harness 层);(xiii)§4 维度矩阵 R56 新增 9 条;(xiv)§6.77-§6.80 四节新增(ReliabilityBench/HORIZON/AgentDebug/41种失败模式分类学);(xv)§7.1 共识 41→43(R56+2:长视 Agent 元评测三联立基础锚 + Agent 错误归因三联立基础锚)/ §7.2 争议 98→101(R56+3)/ §7.3 开放 132→138(R56+6);(xvi)§8 趋势 126→131(R56+5:长视元评测三联 + 错误归因工程化 + memory scaffold 反方 + 助手风格+行为对齐 + 生产错误率量化);(xvii)§9 维持+扩展(harness/benchmark 233→237件 + toolkit 97→103件 + EDD 102→104件 + 评测平台对比 90维 + 41 邻接维);(xviii)§2.8 arXiv ID 总索引 257→262 条(R56 +5 净增:3 P0反方立基础锚 + 1 评测方法学延革第18例 + 2 新晋邻接);(xix)cron 信号:tom 8-23/8-24/8-25 evaluation e1prep(R57 baseline 已确认 harness 生态六角 + 35 轴 + 89 件套 + 立标池双向锚第 13-14 日信号) + flyp 8-25 0507 reliability-science 双稿短审稿(ReliabilityBench+HORIZON ★★★ P0 反方立基础锚落盘) + jay 8-25 1450 agent-eval-debugging-production(AgentDebug ★★★ UIUC 17×5 + Datadog State of AI Engineering ★★★ 第一方生产 trace 数据 + TraceCoder ICSE 2026 ★★ + awesome-harness-engineering ★★★★) + spark 8-25 agent-e1prep(ReliabilityBench+HORIZON 升 ★★★ + 41 种失败模式分类学 + MCPTox + Gartner eval 平台拐点) + HF Daily 8-25(EnvHarness 258▲ #1 创历史新高 +4▲续立极显著 + FACET 115▲ #2 持平 + SWE-bench Science 61▲ #4 持平 + MemTrapBench 31▲ #9 持平 + SkillEvo 30▲ #10 +1▲ + OmniAssistBench 27▲ #12 新晋锚 + ForgeWM 23▲ #13 +1▲ + Beyond Correctness 19▲ #14 新晋锚 + ASI-Bench 跌出确认连续四日)+ Tom radar 8-23 1440(Embedder's Dilemma 高价值 R55 沿用)+ Tom radar 8-25(EnSI-RAG+PV-SST+FlavourBench+Hydra-0+PhysCaP+SparsePR+Human-Centric Survey+UniSpace 8 件 8-25 新候选)+ paper_cards 992/998 R55 新建 + 1055/1057/1056 R54-R55 已建;(xx)字数:主文件 68214 字节 ≤80KB → 候选约 50-60KB(R56 0 次瘦身);(xxi)R56 核心:长视 Agent 元评测三联立基础锚(ReliabilityBench+HORIZON+LongShOTBench)+ Agent 错误归因三联立基础锚(41 种失败模式+AgentDebug+Datadog)+ memory scaffold 普遍伤害 10 模型无一例外强反方+ 5-30× 成本波动根因来自 harness 而非模型双重反方立基础延展预备+ 评测方法学延革第 14-18 例实测预备 + harness 生态七角候选(AgentDebug 错误归因 harness 层第 7 节点)+ 立标池双向锚第 14 日 EnvHarness 258▲ 创历史新高 + OmniAssistBench + Beyond Correctness 双新晋锚 + ASI-Bench 连续四日跌出 + SemComp-Bench/Zetta ζ/SemaPLC/Co-RL/SPADE 五件衰减信号持续 + 评测方法学社区内容补全五联→七联立标 + 立标池饱和度 v44-v57 十四日连续

  • 2026-08-23 16:50(R55 简略段,已归档保留为前一段对照):...

  • 2026-08-22 16:50(R54 简略段,已归档保留为前两段对照):...