论文

本文提出 M$^3$Exam，一个以查询为中心、基于真实用户-Agent 交互构建的多模态对话记忆基准，涵盖跨模态定位与隐式信息推断等多维度评估。M$^3$Exam is introduced, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference.

本文提出 ForeSci，一个时间受控的基准，用于评估 LLM Agent 是否能从历史证据中做出前瞻性研究判断，并在四种骨干模型上评测原生 LLM、Hybrid RAG 以及三种 research-agent 适配方案。This work introduces ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence, and evaluates native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones.

本文提出 MMLongEmbed，首个面向长上下文场景评估 MEM 的综合基准，并发现现有架构严重依赖浅层特征匹配，难以捕捉深层语义与结构依赖。This work introduces MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios, and finds that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies.

文中指出，安全的 LLM Agent 需要显式的信任边界、原则化的权限控制、具备溯源能力的 state 管理，以及与真实运行场景对齐的评估实践；现有 benchmark 仍未能充分覆盖长程、具状态、对部署敏感的风险。It is argued that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings, as well as existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks.

在 DeNovoSWE 上对 Qwen3-30B-A3B 进行微调可显著提升长程 SWE 性能，在具有挑战性的 BeyondSWE-Doc2Repo benchmark 上将其得分从 5.8% 提升至 47.2%。Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

介绍 EvoArena 基准套件，将环境变化建模为跨终端、软件与社会领域的渐进式更新序列；并提出 EvoMem，一种基于 patch 的记忆范式，将记忆演化记录为结构化的更新历史，使 Agent 能通过记忆的变化推理环境的演化。EvoArena is introduced, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains, and EvoMem is proposed, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.

本文测试在模型作答时阻止其看到选项标签能否消除位置影响并进而提升性能，并评估了两种不同的偏置缓解策略。This paper test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance, and evaluates two different strategies for mitigating bias.

HarnessRisk 是一个面向生命周期的 benchmark，将 agent harness 安全组织为六个运行阶段，包括 Harness Configuration、Capability Extension、Runtime Operation、State Persistence、Action Control 和 Incident Recovery，发现显式的风险识别并不能可靠地带来安全的行动——某些配置在超过 90% 的运行中检测到风险，同时仍保留显著的攻击成功率。HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery, finds that explicit risk recognition does not reliably lead to safe action as some configurations detect risks in more than 90% of runs while retaining substantial attack success.

PTXBench 提供了一个可审计的测试平台，用于衡量并提升 LLM 利用持续演进 GPU 架构的能力，并表明各 LLM 在架构特定 PTX 能力上仍参差不齐。PTXBench provides an auditable testbed for measuring and improving LLMs'ability to exploit evolving GPU architectures, and shows that architecture-specific PTX capability remains uneven.

实验表明，仅提供触觉本身并不能确保有效的多模态融合，SoftVTBench 为研究策略不仅能否成功，还在于其如何与可形变物体物理交互，以及触觉在何时改善这种交互，提供了统一的视触觉资源Results show that making touch available does not by itself ensure effective multimodal fusion, and SoftVTBench provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.

本文提出一种量化基准优化的方法论，聚焦于音频对参考转写不充分确定的情形，指出高性能模型会表现出基准条件化行为，从而虚高基准得分，却未必反映通用转写能力的真正提升。This work presents a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript, and indicates that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

选取三个前沿混合专家模型在低资源语言上进行推理微调，提出六个可度量的行为维度，且每维度均设门拒绝任何与输出长度相关的指标，并报告其自家评测工具为何失效。Take three frontier mixture-of-experts models and fine-tune them to reason in a low-resource language and propose six behavioural dimensions that make changes measurable, each gated to reject any metric that correlates with output length, and report how their own instruments lied.

长视频理解任务超越了孤立事件的检索，需要追踪不断演化的叙事并解读可能隐含的社会含义。然而，现有 benchmark 很少联合评估这些能力，尤其是在高语境、非英语媒体中。为弥补这一空白，我们提出 NARU，一个用于评估日语长视频中叙事演化与文化理解推理的 benchmark。NARU 包含 1,481 个问题，源自 155 个总时长 146.8 小时的视频，涵盖四个叙事维度和五个文化维度Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimens

本研究对新兴的内存键值存储进行了全面的性能与可行性评估，突出了性能、兼容性与长期可行性（包括项目成熟度、社区支持与持续开发）之间的权衡。This study presents a comprehensive performance and viability assessment of the emerging in-memory key-value stores and highlights trade-offs between performance, compatibility, and long-term viability, including project maturity, community support, and sustained development.

尽管近期 LLM 在单项专利起草任务上取得不错的成绩，却未触及现实专利起草的核心挑战——从早期发明材料直接生成完整且法律上一致的专利申请。已有工作多假设输入处于后期、高度结构化或已具法律风格，但实际工作流程始于发明人编写的非正式、去法律化的 disclosure。为弥合差距，我们提出 Dis2Pat，一个反映真实[专利撰写场景]的 disclosure-to-patent 数据集。While recent large language models (LLMs) have achieved promising results on individual patent drafting tasks, they fundamentally fail to investigate the core challenge of real-world patent drafting: generating a complete and legally coherent patent application directly from early-stage invention materials. Prior work predominantly assumes later-stage, highly structured, or already legalistic inputs. However, real patenting workflows begin with informal, de-legalized disclosures authored by inventors. To bridge the gap, we introduce Dis2Pat, a disclosure-to-patent dataset that reflects realist

开放式语言模型基准通常继承一种评判方式：人类偏好面板、另一个模型，或脆弱的精确匹配答案。我们提出 FlavourBench，一个自动化基准，其中版本化的烹饪系统提供密集、可执行的真值。每个任务给出八种食材，要求选择三食材组合；在模型执行前，Epicure 对全部 56 种可能组合打分。我们在相同的核心任务集上评估了 27 个 frontier 端点，覆盖替换、配对与受限组合共 534 道任务。每个被排名的模型在每个面板上恰好有 89 个有效回答，且 famOpen-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and fam

随着端侧 LLM Agent 演变为个人副驾驶，移动操作系统已成为该范式的关键试验场，亟需严格的能力评测。然而现有基准可分为两类，各自存在关键盲区：以 GUI 为中心的基准仅测试表层屏幕操作，忽略了后台工具调用与长程规划；而静态 function-calling 基准依赖离线 API 匹配，与真实运行时约束脱节。为弥合这一差距，我们提出 MobilePA-Bench，一个交互式、有状态、以工具为中心的基准。As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark

本文提出了 SeedRG，一个用于缓解 knowledge leakage 并应对 benchmark aging 问题的半合成 benchmark 生成 pipeline。SeedRG is introduced, a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging.

本文提出一个无偏的多进程 evaluation 框架，能够有效分散 client 端负载，从而在每秒数千次 query 以上的生产规模下实现对 LLM 的精确、可复现 profiling。This work proposes an unbiased, multi-process evaluation framework that effectively distributes client-side load, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.

本文提出 RAGCap-Bench，一个面向能力的 benchmark，用于对 agentic RAG workflow 中的中间任务进行细粒度评测，并构建了典型 LLM 错误的分类体系以设计针对性评测问题。This work proposes RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows, and constructs a taxonomy of typical LLM errors to design targeted evaluation questions.

本工作提出 AlphaEval，一个基于真实生产环境的基准，包含来自七家在其核心业务中部署 AI Agent 的公司的 94 个任务，覆盖六个 O*NET (Occupational Information Network) 领域；并贡献了一套从需求到基准的构建框架，将从需求到评估的完整流程标准化。This work presents AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains, and contributes a requirement-to-benchmark construction framework that standardizes the entire pipeline from requirement to evaluation.

提出 RoboDojo，一个面向通用机器人操作策略综合评估的仿真-真机统一基准，将 30 种策略集成到 XPolicyLab 并在 RoboDojo 上进行评测，建立了公开的排行榜与系统性的策略性能分析。RoboDojo is introduced, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies that integrates 30 policies into XPolicyLab and evaluates them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance.

确认了丰富资源语言与代表性不足语言之间在数学推理性能上存在持续差距，性能更优主要与更强的指令遵循能力相关，并提出了完全开源的数据集、数据采集流程与评估框架。A persistent gap in mathematical reasoning performance between high-resource and underrepresented languages is confirmed, with stronger results largely associated with better instruction-following ability, and a fully open-source dataset, data acquisition pipeline, and evaluation framework is introduced.

该工作提出 HETERQA，一个包含 857 个 QA 对的综合性基准，涵盖五个异构来源的记录检索，并表明 HETERQA 为异构来源下的记录检索提供了有效的测试平台，为未来检索方法留下了显著空间。This work introduces HETERQA, a comprehensive benchmark with 857 QA pairs for record retrieval over five heterogeneous sources and indicates that HETERQA provides an effective testbed for record retrieval over heterogeneous sources and leaves substantial room for future retrieval methods.

静脉识别是一种安全生物特征技术，常受限于标注数据稀缺与成像差异；而面向自然图像设计的增强策略可能破坏其关键的细粒度拓扑与纹理。本文提出 AGVBench，在 5 个公开掌/指静脉数据集、7 种骨干网络（含经典 CNN、视觉 Transformer 及静脉专用模型）上评测 30 种代表性增强策略。结果显示，多图混合类方法（如 MixUp、PuzzleMix……Vein recognition is a secure biometric technology often constrained by limited annotated data and imaging variations. While data augmentation mitigates this, strategies designed for natural images may disrupt the fine-grained topology and textures essential for identity discrimination. We present AGVBench, which evaluates 30 representative augmentation strategies on five public palm- and finger-vein datasets with seven backbone architectures, covering classic CNNs, vision transformers, and vein-specific recognition models. Our results show that multi-image mixing methods (e.g., MixUp, PuzzleMi

提出"自主策略演化"评估范式：在固定交互预算下，由 harness-model Agent 反复编辑可执行策略系统；并在 EvoPolicyGym 中实例化，该基准基于一组紧凑型交互式 RL 环境构建，用于评测 Agent 如何迭代改进已探索策略。This work introduces Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget, and instantiates this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments that evaluates how agents iteratively improve explored policies.

BeyondArena 是首个面向表格数据的统一整体基准，支持多种任务类型（IID、时序、分组），覆盖样本量与特征维度的不同尺度，并涵盖来自广泛学科的多样化特征类型。BeyondArena is the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types from a broad range of disciplines.

提出 PerceptionRubrics——一个基于评分量表的评估框架，旨在弥合饱和的基准分数与真实场景脆弱性之间的差距，并验证了严格的感知保真是可靠生成的前提。PerceptionRubrics is introduced, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness, validating that strict perceptual fidelity is the prerequisite for reliable generation.

所得到的模型在大多数数据集-预测步长组合上优于先前的线性预测器，并在八个基准中的六个上超越 Transformer、MLP 和 CNN 基线；同时它还可作为对数据本身的诊断工具，揭示那些被更大模型默默吸收进其学习参数中的结构。The resulting models beat prior linear forecasters on most dataset-horizon entries and exceed Transformer, MLP, and CNN baselines on six of eight benchmarks, and serve as a diagnostic on the data itself, revealing structures that larger models absorb silently into their learned parameters.

提出 GauntletBench，一个用于评估 Agent 在挑战性场景中泛化能力的 Web 基准，聚焦于三种被低估的能力（时间感知、图形理解与 3D 推理），揭示了当前 Agent 能力与复杂真实场景所需能力之间的巨大差距。GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), is introduced, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios.

一个包含 382 个真实企业任务、覆盖 6 种专业角色和 22 项程序性技能的基准，用于评估技能在任务、角色和模型骨干间的迁移能力，发现部分技能可在任务和模型间广泛泛化，而另一些则专化为角色特定工作流，在迁移时失去效力。A benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones finds that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer.

结果表明仅评估求解器或仅评估答案是不足的：Agent 的差异不仅体现在发现关键 contingency 上，还体现在验证预算的使用、显式提交、类型强制、重复验证、基于证据的报告以及缓解行为等方面。The results show why solver-only or answer-only evaluation is insufficient: agents are distinguished not only by top-contingency discovery, but also by validation-budget use, explicit submission, type coercions, duplicate validations, evidence-backed reporting, and mitigation behavior.

本文提出 AdvancedMathBench，一个用于评估高级数学推理能力的 benchmark 套件，并推出 VerifierBench，包含 888 条模型生成的证明轨迹及专家 ground truth，用于评估模型能否正确判断证明有效性并给出合理的验证理由。This work introduces AdvancedMathBench, a benchmark suite designed to evaluate advanced mathematical reasoning capabilities, and introduces VerifierBench, consisting of 888 model-generated proof trajectories paired with expert ground truth, to evaluate whether models can correctly judge proof validity and provide sound verification rationales.

本文提出 MCLASH，一个多语言道德决策 benchmark，用于捕捉跨语言的文化情境化道德直觉和社会规范；并提出 MET（Multilingual Ethics with Theory-grounded reasoning），一种基于心理学与哲学专家策划的理论依据的两步提示方法。This work introduces MCLASH, a multilingual moral decision-making benchmark to capture culturally situated moral intuitions and social norms across languages, and proposes MET (Multilingual Ethics with Theory-grounded reasoning), a two-step prompting method built on expert-curated, theory-based grounds drawn from psychology and philosophy.

开发了自动化评分流水线，用于评估多种模型，包括开源权重模型、闭源语言模型、视觉语言模型和图像生成模型，结果显示没有单一模型在所有任务类型上占优，且某些任务对所有评估模型仍具挑战性。An automated grading pipeline is developed to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models, and shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models.

提出 LakeQuest，一个经人工校验的基准，包含 9,846 个 QA 对，用于在真实数据湖场景下评估端到端检索与综合 pipeline，并揭示现代问答系统中的关键失效模式。LakeQuest is introduced, a human-validated benchmark of 9,846 QA pairs designed to evaluate the end-to-end retrieve-and-synthesize pipeline over realistic data lakes and exposes critical failure modes in modern QA systems.