论文

本文提出 M$^3$Exam，一个以查询为中心、基于真实用户-Agent 交互构建的多模态对话记忆基准，涵盖跨模态定位与隐式信息推断等多维度评估。M$^3$Exam is introduced, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference.

本文提出 V-RAGBench——一个由 ⟨query, evidence chunk, answer⟩ 三元组构成的基准，可对检索与生成进行忠实且解耦的评估；同时提出 CARVE，一种在多种配置下并行运行检索器、并通过分块自适应重排序为每个分块挑选最优配置的简易方法。V-RAGBench is introduced, a benchmark of $\langle$ query, evidence chunk, answer$\rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk.

本文提出 ForeSci，一个时间受控的基准，用于评估 LLM Agent 是否能从历史证据中做出前瞻性研究判断，并在四种骨干模型上评测原生 LLM、Hybrid RAG 以及三种 research-agent 适配方案。ForeSci is introduced, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence, and agent-based methods improve traceability over Hybrid RAG, while their gains in future-target alignment over native LLMs are modest and task dependent.

本文提出 π-Bench，一个用于评估主动式协助能力的基准，包含跨 5 个领域特定用户画像的 100 个多轮任务，用于评估 Agent 在长交互中预见并满足用户需求的能力，联合衡量长周期轨迹中的主动性与任务完成度，更贴近真实使用场景。$-Bench is introduced, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas that evaluates agents'ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use.

本文提出 MMLongEmbed，首个面向长上下文场景评估 MEM 的综合基准，并发现现有架构严重依赖浅层特征匹配，难以捕捉深层语义与结构依赖。This work introduces MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios, and finds that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies.

Agents' Last Exam（ALE）是一个面向长时序、具有经济价值且结果可验证的真实任务的 AI Agent 评测基准，旨在弥合基准测试表现与 GDP 相关影响之间的差距，而非仅仅作为排行榜。Agents'Last Exam (ALE) is introduced, a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes, intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

提出一个面向 RAG 的 AI 系统基准测试（RAGPerf）框架，用于刻画 RAG pipeline 的系统行为，并证明其引入的性能开销可忽略不计。The design and implementation of a RAG-based AI system benchmarking (RAGPerf) framework for characterizing the system behaviors of RAG pipelines is presented and it is shown that RAGPerf incurs negligible performance overhead.

在两台生产超级计算机上对三种 SOTA 向量数据库进行大规模评估，扩展至 64 个计算节点上的 256 个分布式 worker，结果显示：工作负载特性可能限制延迟下降，增加核心反而会降低查询吞吐，且从 16 扩展至 256 个 worker 仅带来 5.46 倍的性能提升。A large-scale evaluation of three state-of-the-art vector databases on two production supercomputers, scaling to 256 distributed workers across 64 compute nodes, reveals that workload characteristics can limit latency reduction, additional cores can reduce query throughput, and scaling from 16 to 256 workers only yields a 5.46x improvement.

本文在 KV cache 约束下对 LLM 推理进行理论建模，提出一种新型批处理与调度算法，在有效管理 KV cache 内存的同时最小化推理延迟，并通过在合成数据集上与后视最优的对比展示其强劲的实证性能。This work model LLM inference with KV cache constraints theoretically and proposes a novel batching and scheduling algorithm that minimizes inference latency while effectively managing the KV cache's memory, and demonstrates the algorithm's strong empirical performance by comparing it to the hindsight optimal in a synthetic dataset.

本文构建一种针对 GitHub issue 的持久会话评估方法，锚定在单一 base commit，对线性顺序探索与非线性、领域范围的并行 agentic 探索进行比较。This work constructs an approach for persistent-session evaluation of GitHub issues anchored at a single base commit, and compares linear sequential exploration against non-linear, domain-scoped parallel agentic exploration.

结果表明，在所评估的协调者–工作者设定下，多 Agent 系统中的隐私风险主要由架构层面的协调通道决定，而非仅取决于最终输出行为：风险来源于对标准输出级防御不可见的内部通道。Results suggest, within the evaluated coordinator-worker setting, that privacy risk in multi-agent systems is strongly shaped by architectural coordination channels rather than final-output behavior alone: it arises from internal channels that remain invisible to standard output-level defenses.

文中指出，安全的 LLM Agent 需要显式的信任边界、原则化的权限控制、具备溯源能力的 state 管理，以及与真实运行场景对齐的评估实践；现有 benchmark 仍未能充分覆盖长程、具状态、对部署敏感的风险。It is argued that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings, as well as existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks.

在 DeNovoSWE 上对 Qwen3-30B-A3B 进行微调可显著提升长程 SWE 性能，在具有挑战性的 BeyondSWE-Doc2Repo benchmark 上将其得分从 5.8% 提升至 47.2%。Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

介绍 EvoArena 基准套件，将环境变化建模为跨终端、软件与社会领域的渐进式更新序列；并提出 EvoMem，一种基于 patch 的记忆范式，将记忆演化记录为结构化的更新历史，使 Agent 能通过记忆的变化推理环境的演化。EvoArena is introduced, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains, and EvoMem is proposed, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.

介绍 MCP-Persona，这是首个专为评估 Agent 在真实场景、个性化 MCP 工具上的表现而设计的基准，并揭示了当前 Agent 在个性化工具使用上的显著不足，从而凸显该基准在发现并解决这些局限上的关键作用。MCP-Persona is introduced, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools and demonstrates their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations.

本文测试在模型作答时阻止其看到选项标签能否消除位置影响并进而提升性能，并评估了两种不同的偏置缓解策略。This paper test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance, and evaluates two different strategies for mitigating bias.

HarnessRisk 是一个面向生命周期的 benchmark，将 agent harness 安全组织为六个运行阶段，包括 Harness Configuration、Capability Extension、Runtime Operation、State Persistence、Action Control 和 Incident Recovery，发现显式的风险识别并不能可靠地带来安全的行动——某些配置在超过 90% 的运行中检测到风险，同时仍保留显著的攻击成功率。HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery, finds that explicit risk recognition does not reliably lead to safe action as some configurations detect risks in more than 90% of runs while retaining substantial attack success.

本文设计了一项对比研究，结合受控定量实验与配对轨迹分析，并将观察结果归纳为一个包含三个高层类别与十二种 Skill 使用模式的分类法，表明当噪声轨迹转化为稳定执行的过程性锚点时，Skill 便会发挥作用。This work designs a contrastive study that combines controlled quantitative experiments with paired trajectory analysis and consolidates observations into a taxonomy of three high-level categories and twelve skill-use modes, showing that skills work when noisy trajectories become procedural anchors that stabilize execution.

PTXBench 提供了一个可审计的测试平台，用于衡量并提升 LLM 利用持续演进 GPU 架构的能力，并表明各 LLM 在架构特定 PTX 能力上仍参差不齐。PTXBench provides an auditable testbed for measuring and improving LLMs'ability to exploit evolving GPU architectures, and shows that architecture-specific PTX capability remains uneven.

实验表明，仅提供触觉本身并不能确保有效的多模态融合，SoftVTBench 为研究策略不仅能否成功，还在于其如何与可形变物体物理交互，以及触觉在何时改善这种交互，提供了统一的视触觉资源Results show that making touch available does not by itself ensure effective multimodal fusion, and SoftVTBench provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.

在代表性视频生成模型上的实验表明，在保持参考图像中任务相关语义一致性的同时实现预期结果仍然具有挑战性Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.

结果为兼容的开源权重语言模型检查点建立了一种被动、无需数据的溯源信号，且该投影配对信号出现在六个及更多语言模型系列中The results establish a passive, data-free provenance signal for compatible open-weight language-model checkpoints, and the projection-pairing signal appears across six language-model families and beyond.

提出 AdaPop（自适应流行度）方法，将局部 token 置信度与源自外部代理的逐事实流行度相关指数相结合，并通过双上升控制器在每个 epoch 调整 retain 惩罚来自动平衡遗忘与保留。The AdaPop (Adaptive Popularity) method is proposed, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy, and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch.

MissDiag 将聚合鲁棒性度量转化为类型化的诊断归因，为在不完备知识下比较、诊断和压力测试 KGQA 与 KG-RAG 系统提供了更具可解释性的基础。By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.

本文提出一种量化基准优化的方法论，聚焦于音频对参考转写不充分确定的情形，指出高性能模型会表现出基准条件化行为，从而虚高基准得分，却未必反映通用转写能力的真正提升。This work presents a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript, and indicates that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

一项配对消融实验在保留仓库与可执行工程上下文的同时移除显式科学指导，表明科学知识并非一律有益：可靠信息能约束修复、提升平均表现与 token 效率，而错位指导则会诱发锚定，不必然提升精确修复成功率。A paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context shows that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success.

选取三个前沿混合专家模型在低资源语言上进行推理微调，提出六个可度量的行为维度，且每维度均设门拒绝任何与输出长度相关的指标，并报告其自家评测工具为何失效。Take three frontier mixture-of-experts models and fine-tune them to reason in a low-resource language and propose six behavioural dimensions that make changes measurable, each gated to reject any metric that correlates with output length, and report how their own instruments lied.

介绍 NARU，一个用于评估日语长视频中叙事演进和文化理解推理能力的基准；该工作提出一种基于分层记忆的标注流水线，可将原始视频转换为结构化的事件、叙事和文化标注，并通过任务导向合成与迭代式捷径去除来生成问题。NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video, is introduced, a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal.

本研究对新兴的内存键值存储进行了全面的性能与可行性评估，突出了性能、兼容性与长期可行性（包括项目成熟度、社区支持与持续开发）之间的权衡。This study presents a comprehensive performance and viability assessment of the emerging in-memory key-value stores and highlights trade-offs between performance, compatibility, and long-term viability, including project maturity, community support, and sustained development.

介绍 Dis2Pat，一个从披露到专利（disclosure-to-patent）的数据集，其设计贴合真实专利工作流，要求直接从发明人风格的、去法律化的披露生成完整专利申请；并提出强基线 Patent-MAF，一个可在本地部署、面向专利撰写的多 Agent 框架。Dis2Pat is introduced, a disclosure-to-patent dataset that reflects realistic patenting workflows by requiring the generation of complete patent applications directly from inventor-style, de-legalized disclosures and a strong baseline named Patent-MAF is proposed, a multi-agent framework for locally deployable patent drafting.

在所测试的同伴排序信息流下，鲁棒的结论是词法层面的趋同，而非对一般意见的捕获，也不是合成 LLM-Agent 群体中的一般性协调优势；该结果未估计其对人类或真实生产平台的影响。The robust result is lexical convergence under the tested peer-ranked feed, not general opinion capture or a general coordination advantage in the synthetic LLM-agent populations; it does not estimate effects on people or production platforms.

本文提出 FlavorBench：一个基于版本化烹饪嵌入模型编译稠密确定性答案映射的基准，并报告了一项 3 seed 的后训练研究——在 Epicure 的 270 条最优答案上对 Qwen3-0.6B checkpoint 进行 LoRA SFT 后，在该任务集上获得 13.3 分的提升。This work introduces FlavorBench: a benchmark for Compiling Dense Deterministic Answer Maps from a Versioned Culinary Embeddings Model and presents a 3-seed post-training study where LoRA SFT of a Qwen3-0.6B checkpoint on 270 optimal answers for Epicure to score on this task-set resulted in a 13.3 point gain.

我们提出 Hydra-0，一种以动作流为条件的通用世界模型，将机器人动作表示为像素运动。这种共享的视觉接口使得跨具身、任务、环境和视频生成 backbone 的通用世界建模与控制成为可能，学习动作在不同场景下的后果。我们的最佳配置相比动作条件 baseline，机器人运动误差降低 90.4%，物体运动误差降低 60.2%，同时支持零样本组合与数据高效适配。在 RoboLab 基准上，Hydra-0 在 replayeWe introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replaye

通过将交互式函数调用沙箱与基于证据的验证相结合，MobilePA-Bench 既可作为实用的诊断基准，也是 Agent 强化学习的交互式基础，加速可靠移动 Agent 的开发。By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.

基于语音的应用在接入检索模块之前需先通过自动语音识别（ASR）处理口头查询，因此 ASR 错误会以固定的上游约束进入 pipeline。我们通过实验验证标准检索增强生成（RAG）的两项扩展——实体图链接与迭代式 query 改写——是吸收还是放大了这些错误。基于神经 TTS 合成的四种英语口音，我们在三个多跳 QA 基准（HotpotQA、2WikiMultiHopQA 和 MuSiQue）上评测四种 RAG 配置，以干净文本 oracle 为对照。尽管结构上更丰富的 configuratioSpeech-based applications pass spoken queries through automatic speech recognition (ASR) before any retrieval module, so ASR errors enter the pipeline as a fixed upstream constraint. We empirically test whether two extensions to standard retrieval-augmented generation (RAG), entity-graph linking and iterative reformulation, absorb or amplify these errors. Using four English accents synthesized through neural TTS, we evaluate four RAG configurations on three multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA and MuSiQue) against a clean-text oracle. Although the structurally richer configuratio

本文提出 GameXpert-Bench，将游戏开发的三个生命周期阶段以 coding agent 操作化为三条互补的基准轨道，并发现现有 agent 在生成可玩基础框架和实现显式需求方面更可靠，而在发现缺陷、验证运行时行为以及跨变更保持功能一致性方面能力较弱。GameXpert-Bench is introduced, which operationalizes the three lifecycle stages of game development with a coding agent as three complementary benchmark tracks, and finds current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.