论文

本文提出 M$^3$Exam，一个以查询为中心、基于真实用户-Agent 交互构建的多模态对话记忆基准，涵盖跨模态定位与隐式信息推断等多维度评估。M$^3$Exam is introduced, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference.

本文提出 ForeSci，一个时间受控的基准，用于评估 LLM Agent 是否能从历史证据中做出前瞻性研究判断，并在四种骨干模型上评测原生 LLM、Hybrid RAG 以及三种 research-agent 适配方案。ForeSci is introduced, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence, and agent-based methods improve traceability over Hybrid RAG, while their gains in future-target alignment over native LLMs are modest and task dependent.

本文提出 MMLongEmbed，首个面向长上下文场景评估 MEM 的综合基准，并发现现有架构严重依赖浅层特征匹配，难以捕捉深层语义与结构依赖。This work introduces MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios, and finds that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies.

文中指出，安全的 LLM Agent 需要显式的信任边界、原则化的权限控制、具备溯源能力的 state 管理，以及与真实运行场景对齐的评估实践；现有 benchmark 仍未能充分覆盖长程、具状态、对部署敏感的风险。It is argued that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings, as well as existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks.

在 DeNovoSWE 上对 Qwen3-30B-A3B 进行微调可显著提升长程 SWE 性能，在具有挑战性的 BeyondSWE-Doc2Repo benchmark 上将其得分从 5.8% 提升至 47.2%。Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

介绍 EvoArena 基准套件，将环境变化建模为跨终端、软件与社会领域的渐进式更新序列；并提出 EvoMem，一种基于 patch 的记忆范式，将记忆演化记录为结构化的更新历史，使 Agent 能通过记忆的变化推理环境的演化。EvoArena is introduced, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains, and EvoMem is proposed, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.

本文测试在模型作答时阻止其看到选项标签能否消除位置影响并进而提升性能，并评估了两种不同的偏置缓解策略。This paper test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance, and evaluates two different strategies for mitigating bias.

HarnessRisk 是一个面向生命周期的 benchmark，将 agent harness 安全组织为六个运行阶段，包括 Harness Configuration、Capability Extension、Runtime Operation、State Persistence、Action Control 和 Incident Recovery，发现显式的风险识别并不能可靠地带来安全的行动——某些配置在超过 90% 的运行中检测到风险，同时仍保留显著的攻击成功率。HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery, finds that explicit risk recognition does not reliably lead to safe action as some configurations detect risks in more than 90% of runs while retaining substantial attack success.

PTXBench 提供了一个可审计的测试平台，用于衡量并提升 LLM 利用持续演进 GPU 架构的能力，并表明各 LLM 在架构特定 PTX 能力上仍参差不齐。PTXBench provides an auditable testbed for measuring and improving LLMs'ability to exploit evolving GPU architectures, and shows that architecture-specific PTX capability remains uneven.

实验表明，仅提供触觉本身并不能确保有效的多模态融合，SoftVTBench 为研究策略不仅能否成功，还在于其如何与可形变物体物理交互，以及触觉在何时改善这种交互，提供了统一的视触觉资源Results show that making touch available does not by itself ensure effective multimodal fusion, and SoftVTBench provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.

本文提出一种量化基准优化的方法论，聚焦于音频对参考转写不充分确定的情形，指出高性能模型会表现出基准条件化行为，从而虚高基准得分，却未必反映通用转写能力的真正提升。This work presents a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript, and indicates that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

选取三个前沿混合专家模型在低资源语言上进行推理微调，提出六个可度量的行为维度，且每维度均设门拒绝任何与输出长度相关的指标，并报告其自家评测工具为何失效。Take three frontier mixture-of-experts models and fine-tune them to reason in a low-resource language and propose six behavioural dimensions that make changes measurable, each gated to reject any metric that correlates with output length, and report how their own instruments lied.

介绍 NARU，一个用于评估日语长视频中叙事演进和文化理解推理能力的基准；该工作提出一种基于分层记忆的标注流水线，可将原始视频转换为结构化的事件、叙事和文化标注，并通过任务导向合成与迭代式捷径去除来生成问题。NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video, is introduced, a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal.

本研究对新兴的内存键值存储进行了全面的性能与可行性评估，突出了性能、兼容性与长期可行性（包括项目成熟度、社区支持与持续开发）之间的权衡。This study presents a comprehensive performance and viability assessment of the emerging in-memory key-value stores and highlights trade-offs between performance, compatibility, and long-term viability, including project maturity, community support, and sustained development.

介绍 Dis2Pat，一个从披露到专利（disclosure-to-patent）的数据集，其设计贴合真实专利工作流，要求直接从发明人风格的、去法律化的披露生成完整专利申请；并提出强基线 Patent-MAF，一个可在本地部署、面向专利撰写的多 Agent 框架。Dis2Pat is introduced, a disclosure-to-patent dataset that reflects realistic patenting workflows by requiring the generation of complete patent applications directly from inventor-style, de-legalized disclosures and a strong baseline named Patent-MAF is proposed, a multi-agent framework for locally deployable patent drafting.

本文提出 FlavorBench：一个基于版本化烹饪嵌入模型编译稠密确定性答案映射的基准，并报告了一项 3 seed 的后训练研究——在 Epicure 的 270 条最优答案上对 Qwen3-0.6B checkpoint 进行 LoRA SFT 后，在该任务集上获得 13.3 分的提升。This work introduces FlavorBench: a benchmark for Compiling Dense Deterministic Answer Maps from a Versioned Culinary Embeddings Model and presents a 3-seed post-training study where LoRA SFT of a Qwen3-0.6B checkpoint on 270 optimal answers for Epicure to score on this task-set resulted in a 13.3 point gain.

通过将交互式函数调用沙箱与基于证据的验证相结合，MobilePA-Bench 既可作为实用的诊断基准，也是 Agent 强化学习的交互式基础，加速可靠移动 Agent 的开发。By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.

介绍 ClawProBench：基于 OpenClaw（具备 workspace 工具及浏览、记忆、消息、调度、skill、subagent 等原生能力的实时 agent 运行时）实例化的 trace-aware、runtime-native agent 评估基准。ClawProBench is presented, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents.

本文通过多语言 self-play，正交于知识与综合基准性能对跨语言技能不一致性进行量化，表明技能差异是开发真正多语言模型过程中可衡量且主要的障碍。This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance via multilingual self-play, and shows that skill discrepancies are a measurable major roadblock in the development of truly multilingual models.

评估四个 SOTA 模型发现，将角色置于语义无关的上下文中并不会抑制该角色的关联属性；相反，跨角色的属性集中度会上升（合并 BI $+0.047$）。Evaluating four state-of-the-art models finds that placing a role in a semantically unrelated context does not suppress role-linked attributes; instead, cross-role attribute concentration increases (pooled BI $+0.047$).

提出 EvoGenUI-Bench，一个面向多轮界面维护的基准，包含 150 个五轮任务，共计 750 轮，覆盖三种场景：信息呈现、可执行交互和工具驱动的外部状态。EvoGenUI-Bench is introduced, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state.

本文提出 DramaChain Bench，首个覆盖完整生产链各阶段的短剧基准，并验证最终剧集质量并非仅由视频生成决定。DramaChain Bench is presented, the first short-drama benchmark that evaluates every stage of the complete production chain, and confirms that final episode quality is not governed by video generation alone.

首个系统性研究 LLM-as-a-judge 在工作流 DAG 上对 Agentic 工具调用评判可靠性的基准，区别于面向开放式文本或偏好的通用 LLM-as-a-judge 任务；揭示了当前 LLM judge 的根本局限，并给出面向 Agentic 系统可靠评估的实践指南。The first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation, exposes fundamental limitations of current LLM judges and yields practical guidelines for reliable evaluation in agentic systems.

研究发现，生成的 harness 在代码与搜索/研究任务上仍显著落后于成熟的人工参考方案，而在写作与机器学习实验任务上达到或超过所选参考方案，且执行成本差异巨大。It is found that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost.

本文提出 ASPIRE，面向模糊目标驱动自我演化的基准，表明模糊目标会将搜索资源导向目标解释阶段；并在涵盖 6 类目标的 520 题隐藏专家评测集上评估所得系统。This work introduces ASPIRE, a benchmark for vague-goal-driven self-evolution and shows that vague goals redirect search effort toward goal interpretation, and evaluates the resulting systems on a hidden, expert-authored set of 520 items spanning six goals.

这些发现表明，仅识别成功动作并不足够；Agent 还需将反馈转化为可执行且可迁移的策略。本文给出统一框架以诊断该过程，并定位阻碍 Agent 将交互经验转化为可靠自我提升的瓶颈。These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies, and provide a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.

提出 The Last Translation Benchmark，一组由人工编写并经同行评审的样本，可打破领先的机器翻译模型，并提出新评估方法：每个样本附带手工编写的验证规则，描述该样本上的具体失败案例，从而支持可靠且可操作的未来评估。The Last Translation Benchmark is introduced, a collection of human-authored and peer-reviewed examples that break leading machine translation models and a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation.

该工作提出了 OR-Clarify，一个用于预表述澄清的 benchmark，并提出了 Interactive Optimization (InterOPT)，这是一个两阶段框架，能识别未解决的、影响表述的关键缺口，并据此引导系统决定是提出下一个问题还是停止提问。This work introduces OR-Clarify, a benchmark for pre-formulation clarification and proposes Interactive Optimization (InterOPT), a two-stage framework that identifies unresolved formulation-critical gaps and uses them to guide whether to ask the next question or to stop.

提出 KoNA，一个评估 VLM 选择性不遵从的基准，覆盖五类情形：False Premise、Visual Inaccessibility、Universal Unknown、Task Feasibility 与 Safety。实验结果显示，微调后的模型能够区分可回答部分与需要不遵从的部分，并以符合任务要求的方式作答。KoNA is introduced, a benchmark for evaluating selective non-compliance in VLMs across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety, and results suggest that the fine-tuned models can distinguish between answerable components and those requiring non-compliance and respond in a task-appropriate manner.

VDiff-Bench 提供一个针对性诊断基准，用于评估 MLLM 的比较视觉理解能力，揭示标准单图视觉-语言任务无法捕捉的失败模式。VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.

提出 EvoHarnessBench，一个在工具、技能与 Agent 三个维度上对可控 harness 演化条件下的 Agent 进行评测的 benchmark，并将 harness 演化确立为一项独立挑战：Agent 需要在持续演化的 harness 下保持原有有效行为EvoHarnessBench is introduced, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents), and establishes harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.

AgentAudit 沿能力、接地性、安全与行为四个方面的十个维度评估完整执行轨迹——即指令完整性、规划器、记忆、工具选择、工具调用、工具正确性、对齐、工具忠实性、安全性与执行完整性——以精确定位导致观察到的失败的具体阶段。AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions, namely instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, to pinpoint the exact stage responsible for an observed failure.

识别出 diff 生成能够胜出的一种与架构无关的统一机制：它在短小且空间局部化的编辑上具有竞争力；其类别级优势恰好集中在作者数据集中平均编辑步数最低的两类任务——重构与错误处理/边界用例修复。A single, architecture-independent mechanism behind the conditions where diff-based generation does win is identified: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in the authors' dataset.

WearableQA 由 200 位真实用户的可穿戴时序数据、血液生物标志物和人口统计信息构建的 4,084 道十选一选择题组成，为评估 LLM 在真实可穿戴数据上的推理能力提供了现实且具诊断性的基准。WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.

SWE-Bench Pro Verified 提供了一个更可信的基准来评估软件工程 Agent，其结合了反作弊保护（消除主要泄漏渠道而不干扰正常 Agent 功能）和任务精修（最小限度地修正有缺陷实例中的不一致性）。SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents, which combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances.

本文介绍 StochBench，一个基于 Lean 4 的基准，包含 450 道覆盖不同抽象层次的研究生随机过程问题，每道题配有其自然语言来源，更能代表领域特定的应用数学，同时对高级证明器仍具挑战。StochBench is introduced, a Lean 4 benchmark of 450 graduate stochastic-processes problems at varying abstraction levels, each paired with its natural-language source that better represents domain-specific applied mathematics while remaining challenging for advanced provers.