论文

本文提出 M$^3$Exam，一个以查询为中心、基于真实用户-Agent 交互构建的多模态对话记忆基准，涵盖跨模态定位与隐式信息推断等多维度评估。M$^3$Exam is introduced, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference.

本文提出 V-RAGBench——一个由 ⟨query, evidence chunk, answer⟩ 三元组构成的基准，可对检索与生成进行忠实且解耦的评估；同时提出 CARVE，一种在多种配置下并行运行检索器、并通过分块自适应重排序为每个分块挑选最优配置的简易方法。V-RAGBench is introduced, a benchmark of $\langle$ query, evidence chunk, answer$\rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk.

本文提出 ForeSci，一个时间受控的基准，用于评估 LLM Agent 是否能从历史证据中做出前瞻性研究判断，并在四种骨干模型上评测原生 LLM、Hybrid RAG 以及三种 research-agent 适配方案。This work introduces ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence, and evaluates native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones.

本文提出 π-Bench，一个用于评估主动式协助能力的基准，包含跨 5 个领域特定用户画像的 100 个多轮任务，用于评估 Agent 在长交互中预见并满足用户需求的能力，联合衡量长周期轨迹中的主动性与任务完成度，更贴近真实使用场景。$-Bench is introduced, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas that evaluates agents'ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use.

本文提出 MMLongEmbed，首个面向长上下文场景评估 MEM 的综合基准，并发现现有架构严重依赖浅层特征匹配，难以捕捉深层语义与结构依赖。This work introduces MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios, and finds that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies.

Agents' Last Exam（ALE）是一个面向长时序、具有经济价值且结果可验证的真实任务的 AI Agent 评测基准，旨在弥合基准测试表现与 GDP 相关影响之间的差距，而非仅仅作为排行榜。Agents'Last Exam (ALE) is introduced, a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes, intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

提出一个面向 RAG 的 AI 系统基准测试（RAGPerf）框架，用于刻画 RAG pipeline 的系统行为，并证明其引入的性能开销可忽略不计。The design and implementation of a RAG-based AI system benchmarking (RAGPerf) framework for characterizing the system behaviors of RAG pipelines is presented and it is shown that RAGPerf incurs negligible performance overhead.

在两台生产超级计算机上对三种 SOTA 向量数据库进行大规模评估，扩展至 64 个计算节点上的 256 个分布式 worker，结果显示：工作负载特性可能限制延迟下降，增加核心反而会降低查询吞吐，且从 16 扩展至 256 个 worker 仅带来 5.46 倍的性能提升。A large-scale evaluation of three state-of-the-art vector databases on two production supercomputers, scaling to 256 distributed workers across 64 compute nodes, reveals that workload characteristics can limit latency reduction, additional cores can reduce query throughput, and scaling from 16 to 256 workers only yields a 5.46x improvement.

本文在 KV cache 约束下对 LLM 推理进行理论建模，提出一种新型批处理与调度算法，在有效管理 KV cache 内存的同时最小化推理延迟，并通过在合成数据集上与后视最优的对比展示其强劲的实证性能。This work model LLM inference with KV cache constraints theoretically and proposes a novel batching and scheduling algorithm that minimizes inference latency while effectively managing the KV cache's memory, and demonstrates the algorithm's strong empirical performance by comparing it to the hindsight optimal in a synthetic dataset.

本文构建一种针对 GitHub issue 的持久会话评估方法，锚定在单一 base commit，对线性顺序探索与非线性、领域范围的并行 agentic 探索进行比较。This work constructs an approach for persistent-session evaluation of GitHub issues anchored at a single base commit, and compares linear sequential exploration against non-linear, domain-scoped parallel agentic exploration.

结果表明，在所评估的协调者–工作者设定下，多 Agent 系统中的隐私风险主要由架构层面的协调通道决定，而非仅取决于最终输出行为：风险来源于对标准输出级防御不可见的内部通道。Results suggest, within the evaluated coordinator-worker setting, that privacy risk in multi-agent systems is strongly shaped by architectural coordination channels rather than final-output behavior alone: it arises from internal channels that remain invisible to standard output-level defenses.

文中指出，安全的 LLM Agent 需要显式的信任边界、原则化的权限控制、具备溯源能力的 state 管理，以及与真实运行场景对齐的评估实践；现有 benchmark 仍未能充分覆盖长程、具状态、对部署敏感的风险。It is argued that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings, as well as existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks.

在 DeNovoSWE 上对 Qwen3-30B-A3B 进行微调可显著提升长程 SWE 性能，在具有挑战性的 BeyondSWE-Doc2Repo benchmark 上将其得分从 5.8% 提升至 47.2%。Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

介绍 EvoArena 基准套件，将环境变化建模为跨终端、软件与社会领域的渐进式更新序列；并提出 EvoMem，一种基于 patch 的记忆范式，将记忆演化记录为结构化的更新历史，使 Agent 能通过记忆的变化推理环境的演化。EvoArena is introduced, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains, and EvoMem is proposed, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.

介绍 MCP-Persona，这是首个专为评估 Agent 在真实场景、个性化 MCP 工具上的表现而设计的基准，并揭示了当前 Agent 在个性化工具使用上的显著不足，从而凸显该基准在发现并解决这些局限上的关键作用。MCP-Persona is introduced, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools and demonstrates their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations.

本文测试在模型作答时阻止其看到选项标签能否消除位置影响并进而提升性能，并评估了两种不同的偏置缓解策略。This paper test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance, and evaluates two different strategies for mitigating bias.

HarnessRisk 是一个面向生命周期的 benchmark，将 agent harness 安全组织为六个运行阶段，包括 Harness Configuration、Capability Extension、Runtime Operation、State Persistence、Action Control 和 Incident Recovery，发现显式的风险识别并不能可靠地带来安全的行动——某些配置在超过 90% 的运行中检测到风险，同时仍保留显著的攻击成功率。HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery, finds that explicit risk recognition does not reliably lead to safe action as some configurations detect risks in more than 90% of runs while retaining substantial attack success.

本文设计了一项对比研究，结合受控定量实验与配对轨迹分析，并将观察结果归纳为一个包含三个高层类别与十二种 Skill 使用模式的分类法，表明当噪声轨迹转化为稳定执行的过程性锚点时，Skill 便会发挥作用。This work designs a contrastive study that combines controlled quantitative experiments with paired trajectory analysis and consolidates observations into a taxonomy of three high-level categories and twelve skill-use modes, showing that skills work when noisy trajectories become procedural anchors that stabilize execution.

PTXBench 提供了一个可审计的测试平台，用于衡量并提升 LLM 利用持续演进 GPU 架构的能力，并表明各 LLM 在架构特定 PTX 能力上仍参差不齐。PTXBench provides an auditable testbed for measuring and improving LLMs'ability to exploit evolving GPU architectures, and shows that architecture-specific PTX capability remains uneven.

实验表明，仅提供触觉本身并不能确保有效的多模态融合，SoftVTBench 为研究策略不仅能否成功，还在于其如何与可形变物体物理交互，以及触觉在何时改善这种交互，提供了统一的视触觉资源Results show that making touch available does not by itself ensure effective multimodal fusion, and SoftVTBench provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.

在代表性视频生成模型上的实验表明，在保持参考图像中任务相关语义一致性的同时实现预期结果仍然具有挑战性Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.

结果为兼容的开源权重语言模型检查点建立了一种被动、无需数据的溯源信号，且该投影配对信号出现在六个及更多语言模型系列中The results establish a passive, data-free provenance signal for compatible open-weight language-model checkpoints, and the projection-pairing signal appears across six language-model families and beyond.

提出 AdaPop（自适应流行度）方法，将局部 token 置信度与源自外部代理的逐事实流行度相关指数相结合，并通过双上升控制器在每个 epoch 调整 retain 惩罚来自动平衡遗忘与保留。The AdaPop (Adaptive Popularity) method is proposed, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy, and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch.

MissDiag 将聚合鲁棒性度量转化为类型化的诊断归因，为在不完备知识下比较、诊断和压力测试 KGQA 与 KG-RAG 系统提供了更具可解释性的基础。By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.

本文提出一种量化基准优化的方法论，聚焦于音频对参考转写不充分确定的情形，指出高性能模型会表现出基准条件化行为，从而虚高基准得分，却未必反映通用转写能力的真正提升。This work presents a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript, and indicates that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

一项配对消融实验在保留仓库与可执行工程上下文的同时移除显式科学指导，表明科学知识并非一律有益：可靠信息能约束修复、提升平均表现与 token 效率，而错位指导则会诱发锚定，不必然提升精确修复成功率。A paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context shows that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success.

选取三个前沿混合专家模型在低资源语言上进行推理微调，提出六个可度量的行为维度，且每维度均设门拒绝任何与输出长度相关的指标，并报告其自家评测工具为何失效。Take three frontier mixture-of-experts models and fine-tune them to reason in a low-resource language and propose six behavioural dimensions that make changes measurable, each gated to reject any metric that correlates with output length, and report how their own instruments lied.

长视频理解任务超越了孤立事件的检索，需要追踪不断演化的叙事并解读可能隐含的社会含义。然而，现有 benchmark 很少联合评估这些能力，尤其是在高语境、非英语媒体中。为弥补这一空白，我们提出 NARU，一个用于评估日语长视频中叙事演化与文化理解推理的 benchmark。NARU 包含 1,481 个问题，源自 155 个总时长 146.8 小时的视频，涵盖四个叙事维度和五个文化维度Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimens

本研究对新兴的内存键值存储进行了全面的性能与可行性评估，突出了性能、兼容性与长期可行性（包括项目成熟度、社区支持与持续开发）之间的权衡。This study presents a comprehensive performance and viability assessment of the emerging in-memory key-value stores and highlights trade-offs between performance, compatibility, and long-term viability, including project maturity, community support, and sustained development.

尽管近期 LLM 在单项专利起草任务上取得不错的成绩，却未触及现实专利起草的核心挑战——从早期发明材料直接生成完整且法律上一致的专利申请。已有工作多假设输入处于后期、高度结构化或已具法律风格，但实际工作流程始于发明人编写的非正式、去法律化的 disclosure。为弥合差距，我们提出 Dis2Pat，一个反映真实[专利撰写场景]的 disclosure-to-patent 数据集。While recent large language models (LLMs) have achieved promising results on individual patent drafting tasks, they fundamentally fail to investigate the core challenge of real-world patent drafting: generating a complete and legally coherent patent application directly from early-stage invention materials. Prior work predominantly assumes later-stage, highly structured, or already legalistic inputs. However, real patenting workflows begin with informal, de-legalized disclosures authored by inventors. To bridge the gap, we introduce Dis2Pat, a disclosure-to-patent dataset that reflects realist

大语言模型（LLM）agent 的群体行为无法用单 agent 基准刻画。我们提出 PV-SST，一个基于同行投票的社交平台测试平台，并报告一项独立冻结、预先注册的等曝光实验，涵盖四个话题、四个未使用种子、四个开源权重模型家族以及三个预设的更大模型变体。该实验包含 448 次试验和 112 个完整的"模型 × 话题 × 种子"区组。相对于仅话题对照条件，由同行生成的点赞排序的上一轮同行帖子信息流，会提升最终轮在两端的词汇相似度Population-level behavior in large-language-model (LLM) agents cannot be characterized by single-agent benchmarks. We introduce PV-SST, a peer-voted social-platform testbed, and report a separately frozen, preregistered matched-exposure experiment spanning four topics, four unused seeds, four open-weight model families, and three prespecified larger variants. The experiment comprises 448 trials and 112 complete model-by-topic-by-seed blocks. Relative to a topic-only control, a feed of previous-round peer posts ranked by peer-generated likes increases final-round lexical similarity in both the

开放式语言模型基准通常继承一种评判方式：人类偏好面板、另一个模型，或脆弱的精确匹配答案。我们提出 FlavourBench，一个自动化基准，其中版本化的烹饪系统提供密集、可执行的真值。每个任务给出八种食材，要求选择三食材组合；在模型执行前，Epicure 对全部 56 种可能组合打分。我们在相同的核心任务集上评估了 27 个 frontier 端点，覆盖替换、配对与受限组合共 534 道任务。每个被排名的模型在每个面板上恰好有 89 个有效回答，且 famOpen-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and fam

我们提出 Hydra-0，一种以动作流为条件的通用世界模型，将机器人动作表示为像素运动。这种共享的视觉接口使得跨具身、任务、环境和视频生成 backbone 的通用世界建模与控制成为可能，学习动作在不同场景下的后果。我们的最佳配置相比动作条件 baseline，机器人运动误差降低 90.4%，物体运动误差降低 60.2%，同时支持零样本组合与数据高效适配。在 RoboLab 基准上，Hydra-0 在 replayeWe introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replaye

随着端侧 LLM Agent 演变为个人副驾驶，移动操作系统已成为该范式的关键试验场，亟需严格的能力评测。然而现有基准可分为两类，各自存在关键盲区：以 GUI 为中心的基准仅测试表层屏幕操作，忽略了后台工具调用与长程规划；而静态 function-calling 基准依赖离线 API 匹配，与真实运行时约束脱节。为弥合这一差距，我们提出 MobilePA-Bench，一个交互式、有状态、以工具为中心的基准。As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark

基于语音的应用在接入检索模块之前需先通过自动语音识别（ASR）处理口头查询，因此 ASR 错误会以固定的上游约束进入 pipeline。我们通过实验验证标准检索增强生成（RAG）的两项扩展——实体图链接与迭代式 query 改写——是吸收还是放大了这些错误。基于神经 TTS 合成的四种英语口音，我们在三个多跳 QA 基准（HotpotQA、2WikiMultiHopQA 和 MuSiQue）上评测四种 RAG 配置，以干净文本 oracle 为对照。尽管结构上更丰富的 configuratioSpeech-based applications pass spoken queries through automatic speech recognition (ASR) before any retrieval module, so ASR errors enter the pipeline as a fixed upstream constraint. We empirically test whether two extensions to standard retrieval-augmented generation (RAG), entity-graph linking and iterative reformulation, absorb or amplify these errors. Using four English accents synthesized through neural TTS, we evaluate four RAG configurations on three multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA and MuSiQue) against a clean-text oracle. Although the structurally richer configuratio

近期大语言模型（LLM）已能作为编码 Agent，根据自然语言请求构建完整游戏。游戏开发尤为严苛，因为程序逻辑、视觉与音频内容、界面、交互和可玩性必须在同一可执行制品中协同工作。因此衡量该能力需要同时评测游戏产品与开发过程。现有基准通常通过评估最终制品或孤立的开发阶段来评测 LLM 的游戏开发能力。我们对完整人机协作开发过程的分析Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development