论文

识别出 diff 生成能够胜出的一种与架构无关的统一机制：它在短小且空间局部化的编辑上具有竞争力；其类别级优势恰好集中在作者数据集中平均编辑步数最低的两类任务——重构与错误处理/边界用例修复。A single, architecture-independent mechanism behind the conditions where diff-based generation does win is identified: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in the authors' dataset.

WearableQA 由 200 位真实用户的可穿戴时序数据、血液生物标志物和人口统计信息构建的 4,084 道十选一选择题组成，为评估 LLM 在真实可穿戴数据上的推理能力提供了现实且具诊断性的基准。WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.

SWE-Bench Pro Verified 提供了一个更可信的基准来评估软件工程 Agent，其结合了反作弊保护（消除主要泄漏渠道而不干扰正常 Agent 功能）和任务精修（最小限度地修正有缺陷实例中的不一致性）。SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents, which combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances.

本文介绍 StochBench，一个基于 Lean 4 的基准，包含 450 道覆盖不同抽象层次的研究生随机过程问题，每道题配有其自然语言来源，更能代表领域特定的应用数学，同时对高级证明器仍具挑战。StochBench is introduced, a Lean 4 benchmark of 450 graduate stochastic-processes problems at varying abstraction levels, each paired with its natural-language source that better represents domain-specific applied mathematics while remaining challenging for advanced provers.

引入 MetroLLM-Bench，一个包含 955 个用例的基准，用于测试语言模型作为交通信息亭策略层的能力，并评估了来自六个厂商的 26 个模型，其中 23 个被排名。This work introduces MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk, and evaluates twenty-six models from six vendors, of which twenty-three are ranked.

Video-LLM 的时间推理基准常以语言为中介，留有来自选项措辞、答案相关性或语言先验带来的语言捷径空间，因此提出 TempCloze，一个用于评估 Video-LLM 视觉时间推理能力的视频完形填空基准。Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correlations, or language priors, so TempCloze is introduced, a video cloze benchmark for evaluating visual temporal reasoning in Video-LLMs.

本文提出了 SeedRG，一个用于缓解 knowledge leakage 并应对 benchmark aging 问题的半合成 benchmark 生成 pipeline。SeedRG is introduced, a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging.

研究面向实现的研究方法规范的可编码化就绪度，定义为其是否为胜任的实现者或编码 Agent 提供足够的方法学信息，以在不引入未支持假设的情况下构建预期方法。This work studies the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.

本文提出 Benchmark Radar，一个用于 AI 基准检索与发现的活体数据库和搜索引擎，覆盖 LLM 评测、Agent 与工具使用基准、代码、推理、安全及领域评测。Benchmark Radar is presented, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations.

提出了 DataFlex-RL，一个在统一 GRPO 方案下比较数据策略的评测平台；研究发现，改变数据策略会显著影响训练过程，但相对于均匀训练并未带来可复现的提升。DataFlex-RL, an evaluation platform for comparing choices under a common GRPO recipe, is introduced, finding that changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.

提出即插即用的 Feature Recovery Module (FRM)，可在保持宿主网络冻结的前提下，将退化编码器特征映射到与干净图像对齐的表征；该模块提升了场景级检测、CLIP/SigLIP2 特征恢复以及全部四项物体级 VLM 任务，且退化越严重增益越大。The Feature Recovery Module (FRM), a plug-and-play module that maps degraded encoder features to pristine-aligned representations while keeping the host frozen, is proposed, which improves scene-level detection, CLIP/SigLIP2 feature recovery, and all four object-level VLM tasks, with larger gains under more severe degradation.

对突发物理危险的反应既是具身智能的重要检验，也是将多模态大语言模型 (MLLMs) 部署为家庭机器人决策核心的硬性要求；ReactHuman 是首个面向类人反应式决策的物理驱动基准。Reacting to sudden physical hazards is both a meaningful test of embodied intelligence and a hard requirement for deploying multimodal large language models (MLLMs) as the decision core of household robots, and ReactHuman is the first physics-grounded benchmark for human-like reactive decision-making.

推出 Atria Dawn Preview，一个面向科学研究与工程工作流的基础 Agent 语言模型，旨在拓展真实场景下 Agent 生产力的前沿。Atria Dawn Preview is introduced, a foundation agentic language model designed for scientific research and engineering workflows, with the goal of expanding the frontier of agent productivity in the real world.

论文提出了 E2A-Bench，一个面向金融图表推理的 969 查询基准，由 323 个 HS300 成分股在三种输入模态下构建，并附带由 OHLCV 确定性派生的证据锚点；结果表明金融 VLM 评估应追溯从证据到决策的完整链路，而非依赖单一幻觉分数。E2A-Bench is introduced, a 969-query benchmark for financial chart reasoning, constructed from 323 HS300 constituents under three input modalities with deterministic OHLCV-derived evidence anchors, and results show that financial VLM evaluation should trace the full evidence-to-action chain rather than rely on a single hallucination score.

本文提出一个无偏的多进程 evaluation 框架，能够有效分散 client 端负载，从而在每秒数千次 query 以上的生产规模下实现对 LLM 的精确、可复现 profiling。This work proposes an unbiased, multi-process evaluation framework that effectively distributes client-side load, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.

论文提出了 ModularRSI，一个与基准解耦、对比式、模块化的可泛化 harness 进化框架，对同一任务下成功与失败的轨迹进行对比，并跨任务聚合证据以识别反复出现的行为缺陷。ModularRSI is proposed, a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution that contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies.

论文报告了来自 OpenAI、Anthropic、xAI 与 Google DeepMind 的六类前沿模型实验，并使用 epistemic jailbreak 一词来指称随请求具体性增加而伴随出现的技术溯源严谨性丧失现象。This paper reports experiments across six frontier model types from OpenAI, Anthropic, xAI, and Google DeepMind, and uses the term epistemic jailbreak for the accompanying loss of discipline in technical provenance as requested specificity increases.

该工作提出 PACT（Pressure-Applied Compliance Testing），一个用于评估 AI agent 在压力下遵守规则的基准，覆盖十二个受监管的企业领域与四十八个场景，每个场景均为真实的多轮对话。This work introduces PACT (Pressure-Applied Compliance Testing), a benchmark for rule-following under pressure in AI agents assisting employees in daily tasks across twelve regulated enterprise domains and forty-eight scenarios, each set in a realistic multi-turn conversation.

本文提出 V\={a}kQA，一个覆盖六个领域、包含 2,001 对事实型问答的泰卢固语 SQA 基准；并观察到：泰卢固语措辞保留了翻译中会丢失的文化特异性，语音输入引入的音近混淆会改变问题含义，级联 ASR-MT 误差会逐步叠加放大。V\={a}kQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, is introduced and it is observed that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively.

本文提出一个围绕物理世界推理四个互补维度构建的综合评估框架，并构建了一系列多样化新任务，要求模型跨模态整合互补信息。This work introduces a comprehensive evaluation framework organized around four complementary dimensions of physical world reasoning, and constructs a diverse set of novel tasks that require models to integrate complementary information across modalities.

本文提出 TeleAntiFraud 2.0，采用 Mixed-Tree Anti-Fraud Generation Pipeline 构建，并基于月度冻结评测协议进行评估，确立了近域构造与 collapse-aware 报告作为在现实易混淆条件下评测音频电信诈骗模型的核心要求。This work presents TeleAntiFraud 2.0, constructed with the Mixed-Tree Anti-Fraud Generation Pipeline and evaluated under a monthly frozen evaluation protocol, establishing near-domain construction and collapse-aware reporting as core requirements for evaluating audio-based telecom-fraud models under realistic confusable conditions.

OmniVChat-Studio，一种用于合成单轮与多轮音视频对话的多 Agent 数据引擎；以及 OmniVChat-RL，一种在 OmniVChat 中同时针对回复正确性、效率与风格设计奖励的强化学习奖励方案。OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues and OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat.

APort Vault 是面向工具使用 AI Agent 支付授权的基准。它在公开 CTF 活动中重放了人类编写的 4,371 次针对真实支付 Agent 的攻击，覆盖 8 家实验室的 14 个模型、五种策略配置与两条重放轨道，每条轨道分别在有/无确定性的 pre-action check（实现 Open Agent Passport (OAP) 规范）下执行，共计完成 225,964 次评测。我们每次评测报告五个独立事件，因为将它们合并正是 Agent 基准产生无法经得起审查的数字的方式。请求很常见，且其速率差异巨大APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed. We report five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review. Requests are common and their rate differs far

一项基于任务向量迁移的实验：在计算了价值偏好方向对应的任务向量后，将其相对于通用指令跟随向量进行正交化处理，该方法能够有效隔离出特定价值偏好的方向，从而通过任务算术获得具有相反立场的模型。A task vector transfer based experiment where after computing the task vectors for a direction of value preference the authors orthogonalize it with respect to the general instruction following vector shows that this method is effective in isolating the direction of the specific value preference that can successfully be used to conduct task arithmetic to obtain a model with the opposite stance.

本文提出 Gricea，一个开放科学平台，将研究表示为可配置、可部署的研究 artifact，研究者可以运行、检查、共享和复用；该平台展示了如何通过共享的研究 artifact 构建、复现和扩展 CAI 研究，从而借助开放科学实现知识的累积构建。This work presents Gricea, an open-science platform representing studies as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse, demonstrating Gricea's support for constructing, reproducing, and extending CAI studies through shared research artifacts, enabling cumulative knowledge building through open science.

研究发现，较弱的 Agent 常常在产出有效成果之前即告失败，而较强的 Agent 则越来越多地在结构、几何与构造过程要求上失败；CADWorld 揭示了通用 GUI 能力与可靠执行持久、可验证工程工作流之间的差距。It is found that weaker agents often fail before producing a valid artifact, whereas stronger agents increasingly fail on structural, geometric, and construction-process requirements, and CADWorld exposes a gap between general GUI competence and reliable execution of persistent, verifiable engineering workflows.

GameHorizon Suite 是一套统一的数据与评估套件，可在多个时间跨度下衡量不同模型族的游戏能力，为跨水平跨度与跨模型族的游戏能力评估提供标准化标尺。The GameHorizon Suite, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families, can provide a standardized yardstick for evaluating gameplay capabilities across horizons and model families.

提出将变异分析作为 kernel-benchmark 预言机的充分性度量：通过确定性规则向 188 个 KernelBench 问题的已验证 CUDA 实现中注入 10 个可编译故障，其中 7,384 个具备独立击杀见证；任何测试协议均按其检出比例评分。Mutation analysis as an adequacy metric for kernel-benchmark oracles is introduced: deterministic rules inject 10 compilable faults into verified CUDA implementations of 188 KernelBench problems, 7{,}384 of them with an independent kill witness; any test protocol is scored by the fraction it detects.

构建 Taste-Bench，一个由 Agent 在工程与研究任务中产生的轨迹自动构建的品味问题基准，并证明品味是可训练的。Taste-Bench is built, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks, and it is shown that taste can be trained.

本文提出 AgenticRAGTracer，这是首个主要由大语言模型自动构建、专为支持逐步验证而设计的 Agentic RAG 基准。AgenticRAGTracer is introduced, the first Agentic RAG benchmark that is primarily constructed automatically by large language models and designed to support step-by-step validation, and is primarily constructed automatically by large language models and designed to support step-by-step validation.

介绍 Tri-PvP，一个包含 8,000 样本、跨越视觉、音频与文本的三模态冲突基准，揭示了证据形式偏倚的系统性不对称：模型在视觉上更偏向感知信号，而在音频上更偏向命题性信号。Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, is introduced, revealing a systematic asymmetry in evidence-form bias: models exhibit a stronger bias toward perceptual signal in vision but propositional in audio.

EMBODIEDSWE-GEN 将 coding agent 的单一解决方案扩展为大规模多样化轨迹用于训练 VLA，并表明仅在 coding agent 生成的仿真演示上微调的 VLA，即可在真实机器人上完成长时任务。EMBODIEDSWE-GEN, which expands a single solution from coding agent into large diverse trajectories for training a VLA, and shows that a VLA finetuned solely on coding-agent-generated simulation demonstrations completes a long-horizon task on real robot.

介绍 HappyWorld-Bench，一个用于评估生成世界在 Agent 交互过程中是否保持可靠的综合基准，并强调对世界模型的评估不仅应看视觉质量，还应考察状态一致性以及其对动作和干预响应的正确性。HappyWorld-Bench is introduced, a comprehensive benchmark that evaluates whether generated worlds remain reliable as agents interact with them, and highlights the need to evaluate world models not only by visual quality, but also by state consistency and the correctness of their responses to actions and interventions.

本文主张每个 NLP 子领域应将主要性能指标与一个校准分数配对，呼吁将校准视为每个模型的基本属性而非边缘话题。It is argued that each NLP subfield should pair its main performance metric with a calibration score and call for treating calibration as an essential property of every model rather than a niche topic.

本文提出 OmniEchoBench，一个面向空间音视频感知与音-视-语言导航的统一基准，以及一个空间感知的全模态模型，该模型在预训练语义音频通路之外引入 FOA 空间编码器。OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation, and a spatially aware omni-modal model, which introduces an FOA spatial encoder alongside a pretrained semantic audio pathway.

本文提出 RLCDAlignBench，在十类对齐失败上对 Jev 进行基准测试：谄媚、越狱、欺骗、提示注入、幻觉、隐私侵犯、社会偏见、奖励黑客、不确定性隐瞒与权力寻求。RLCDAlignBench is presented, which benchmarks Jev on ten alignment failures: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking.