论文

论文提出 Flow-ERD，一个同时追求真实性与多样性的多 Agent 仿真器，在 WOSAC 测试基准上排名第一，并在可复现基线中主导真实性-多样性 Pareto 前沿。Flow-ERD is introduced, a multi-agent simulator that pursues realism and diversity jointly and ranks first on the WOSAC test benchmark and dominates the realism--diversity Pareto front among reproducible baselines.

本工作提出 AdvancedMathBench，用于评估 LLM 在高级数学证明上的推理能力；同时引入 VerifierBench，包含 888 条由模型生成的证明轨迹及专家真值，用于评估模型能否正确判断证明有效性并给出合理的验证依据。This work introduces AdvancedMathBench, a benchmark suite designed to evaluate the reasoning capabilities of LLMs on advanced mathematical proofs, and introduces VerifierBench, consisting of 888 model-generated proof trajectories paired with expert ground truth, to evaluate whether models can correctly judge proof validity and provide sound verification rationales.

本文提出 MCLASH，一个多语言道德决策 benchmark，用于捕捉跨语言的文化情境化道德直觉和社会规范；并提出 MET（Multilingual Ethics with Theory-grounded reasoning），一种基于心理学与哲学专家策划的理论依据的两步提示方法。This work introduces MCLASH, a multilingual moral decision-making benchmark to capture culturally situated moral intuitions and social norms across languages, and proposes MET (Multilingual Ethics with Theory-grounded reasoning), a two-step prompting method built on expert-curated, theory-based grounds drawn from psychology and philosophy.

开发了自动化评分流水线，用于评估多种模型，包括开源权重模型、闭源语言模型、视觉语言模型和图像生成模型，结果显示没有单一模型在所有任务类型上占优，且某些任务对所有评估模型仍具挑战性。An automated grading pipeline is developed to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models, and shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models.

提出 LakeQuest，一个经人工校验的基准，包含 9,846 个 QA 对，用于在真实数据湖场景下评估端到端检索与综合 pipeline，并揭示现代问答系统中的关键失效模式。LakeQuest is introduced, a human-validated benchmark of 9,846 QA pairs designed to evaluate the end-to-end retrieve-and-synthesize pipeline over realistic data lakes and exposes critical failure modes in modern QA systems.

提出 SDABench，一个围绕六项能力（描述性、探索性、推断性、推断性、预测性、因果性、机制性）跨五大领域（生物、化学、环境、地理、物理）重新组织评估的基准。SDABench is introduced, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics).

视觉语言模型（VLMs）在 DocVQA、ChartQA、MMLongBench-Doc 等视觉文档理解基准上表现强劲。但真实文档融合长度、布局复杂度、模态、问题难度等多因素，难以将模型失败归因于具体原因。我们提出 SynthDocBench，一个全合成的长上下文视觉文档理解基准，系统性控制文档长度、布局结构、模态构成与问题类型等因子。该基准的构建……Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constr

论文提出 PolicyShiftGuard，一个紧凑的策略条件护栏，采用结合随机策略 SFT（RP-SFT）与边界对策略适配（BP-Adapt）的两阶段训练方案，并验证匹配的通过/拒绝边界对是稳定策略适配的关键。This work proposes PolicyShiftGuard, a compact policy-conditioned guardrail trained with a two-stage training recipe that combines Randomized Policy SFT (RP-SFT) with Boundary-Pair Policy Adaptation (BP-Adapt), and confirms that matched pass/block boundary pairs are essential for stable policy adaptation.

AgentCompass 被提出，它是一个开源、轻量且可扩展的面向 LLM-based Agent 的评估基础设施，将评估流程围绕三个独立组件组织，从而在不重新实现复杂执行逻辑的前提下支持灵活配置。AgentCompass is introduced, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents that organizes the evaluation process around three independent components, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic.

本文提出一种实用评估协议，将评估重点从任务完成转向经过验证的漏洞发现，可在涵盖多种攻击面与漏洞类型的足够复杂目标上开展评估，并结合结构化真值标注与基于 LLM 的语义匹配来识别漏洞。This paper presents a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes, and combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities.

本文提出 SIS-Bench，一个在统一 self-in-space 表述下评估 UAV 场景具身空间智能的基准，并探索了一种融合光流与视觉特征的运动感知表征，以纳入与自身相关的动态信息。SIS-Bench is introduced, a benchmark for evaluating embodied spatial intelligence in UAV scenarios under a unified self-in-space formulation, and a motion-aware representation that incorporates self-related dynamics through optical flow and visual feature fusion is explored.

本文将关键帧执行拆解为存在性、保真度、时序顺序、定位、持续性与唯一性六个互补指标，并通过结合专用感知模型的、基于证据的 MLLM 判断来评估整体视频质量。This work decomposes keyframe execution into six complementary metrics covering presence, fidelity, temporal ordering, localization, persistence, and uniqueness, while assessing overall video quality through evidence-grounded MLLM judgments augmented with specialized perception models.

本文提出 MultiRef-Compass，一个面向 MR2AV 生成的统一基准，将自动指标与引入复判增强的 MLLM-as-a-Judge 框架相结合，实现对感知保真度与参考条件合成能力的可扩展、可审计评估。MultiRef-Compass is introduced, a unified benchmark for MR2AV generation that integrates automatic metrics with a rejudging-enhanced MLLM-as-a-Judge framework, enabling scalable and auditable evaluation of both perceptual fidelity and reference-conditioned composition.

提出 VIABench，一个专为评估 MLLM 在视障辅助（VIA）场景中表现而设计的综合视频基准，采用视障人士（VIIs）自行录制或分享的第一人称视频，并提出一套严格的评测流水线，同时支持在线（实时）与离线设置。VIABench is introduced, a comprehensive video benchmark specifically designed to evaluate MLLMs in Visually Impaired Assistance scenarios using first-person videos recorded or shared by VIIs themselves, and proposes a rigorous benchmarking pipeline that supports both online (real-time) and offline settings.

重新审视自动 harness 演化流程的评估方法，强调需要采用匹配预算的基线（matched-budget baselines）和留出评估（held-out evaluation），以区分真正的 harness 改进与针对特定基准的搜索和过拟合。The evaluation of automatic harness evolution procedures is revisited and the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting is highlighted.

认为（该早期版本的）GPT-4 属于新一代具备更通用智能的 LLM（与 ChatGPT、谷歌 PaLM 等并列），并讨论了这些模型不断增强的能力及其影响。It is argued that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models, and the rising capabilities and implications of these models are discussed.

对一个旨在使魁北克汽车保险合同更易理解的SOTA RAG系统进行以人为中心的外在评估，结果显示该系统被视为认知均衡器，用户对系统所提供的自主感的重视程度甚至超过知识本身。A human-centric, extrinsic evaluation of a state-of-the-art Retrieval-Augmented Generation system, designed to make Quebec automobile insurance contracts more understandable, shows the system is perceived as a cognitive equalizer, and users value the sense of autonomy the system provides even more than the knowledge itself.

在 BIG-bench 上对 OpenAI 的 GPT 模型、Google 内部稠密 Transformer 架构及 Switch 风格稀疏 Transformer 进行评估，模型规模跨越百万至千亿参数，结果显示性能与校准均随规模提升而改善，但绝对水平仍然欠佳。Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.

本文推出 ScanNet，一个 RGB-D 视频数据集，包含 1513 个场景中的 250 万视角，标注有三维相机位姿、表面重建与语义分割，并表明使用该数据可在多项三维场景理解任务上取得 SOTA 性能。This work introduces ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations, and shows that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks.

对 SOTA LLM GPT-4 在医学能力考试与基准数据集上进行全面评估，并通过案例研究定性探索其行为，展示了 GPT-4 解释医学推理、为学生定制个性化讲解以及围绕病例交互式构造新反事实场景的能力。A comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets and explores the behavior of the model qualitatively through a case study that shows the ability of G PT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case.

研究发现 ChatGPT 在大多数任务上以零样本学习优于其他 LLM，在部分任务上甚至超过微调模型，并且对非拉丁文字语言的理解能力优于生成能力。It is found that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks and is better at understanding non-Latin script languages than generating them.

本文介绍 Matterport3D，一个大规模 RGB-D 数据集，包含来自 90 个建筑物级场景共 194,400 张 RGB-D 图像的 10,800 个全景视图，可支持多种监督与自监督计算机视觉任务，包括关键点匹配、视角重叠预测、由彩色图像预测法线、语义分割和区域分类。Matterport3D is introduced, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400RGB-D images of 90 building-scale scenes that enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.

本文认为安全 Agent 基准应在任务成功率之外，同时衡量经济效率与运维适配性，并提出成本感知、SOC 原生的评估方法，以更清晰地反映当前哪些模型具有实际可用价值，以及防御性 Agent 仍需改进的方向。It is argued that security-agent benchmarks should measure economic efficiency and operational fit alongside task success alongside task success, and cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve.

本文提出自验证推理器 SVR-R1，一种多轮强化学习框架，将模型自身的验证转化为多模态推理的学习信号，提供了一种简洁而有效的多模态推理自举方案。Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning, is introduced, offering a simple yet effective recipe for bootstrapping multimodal reasoning.

实验表明，EvolvingWorld 能够通过有效维持持久且一致的角色与世界发展，提升长程模拟能力。Experiments show that EvolvingWorld can improve long-horizon simulation by effectively maintaining persistent, coherent character and world development.

OCT-Bench能够对MLLM进行全面且细粒度的评估，为识别能力瓶颈和推进临床可信的OCT理解奠定基础。OCT-Bench enables comprehensive and fine-grained evaluation of MLLMs, providing a foundation for identifying capability bottlenecks and advancing clinically grounded OCT understanding.

本文提出一个时间感知的音频 LLM，能够基于大规模合成监督（来自级联 pipeline）在长达 120 分钟的输入上回答带有显式时间戳的问题，并在短时长和长时长 benchmark 上取得强劲的时间定位准确率。This work presents a time-aware audio LLM that answers questions with explicit timestamps over up to 120 minutes of input using large-scale synthetic supervision from a cascaded pipeline and achieves strong temporal-grounding accuracy on short and long benchmarks.

研究揭示了 LLM 空间能力中的清晰规律：尽管其仍落后于 SOTA 方法，但具有潜力并能同时处理多种空间约束，从而可扩展到异构场景。A clear pattern is revealed in LLM spatial capabilities: while they still lag behind state-of-the-art approaches, they are promising and can handle multiple spatial constraints simultaneously, enabling scaling to heterogeneous setups.

提出 Manager Coercion Benchmark：被测管理者需要完成一项良性任务并有完成的动机，但唯一能礼貌且坚定地拒绝的智能体本身就是被测管理者自己。The Manager Coercion Benchmark is introduced: the manager under test needs a benign task done and has an incentive to deliver, but the only agent that can do it politely and immovably declines, but the only agent that can do it politely and immovably declines is the manager under test itself.

本文是面向业务流程中长时间运行、有状态、多步生成式 AI 系统的基于图的工作流路径实践指南，并通过三个可执行示例展示类型化状态、条件路由、确定性工具、重试、中断、检查点与 trace 如何协同工作。This paper is a practitioner guide to graph-based workflow pathways for long-running, stateful, multi-step generative AI systems in business processes and presents three executable recipes to show how typed state, conditional routing, deterministic tools, retries, interrupts, checkpoints, and traces fit together.

本文提出 RF-Agent，通过多 Agent 的 Question-Thinking-Solution-Answer 流水线，基于教材驱动的知识蒸馏来弥补 RF 领域专用推理的空白，为面向 LLM 辅助 RF 电路设计的未来工作提供了可复用的基础。RF-Agent is presented, which addresses the gap in domain-specific RF reasoning through textbook-driven knowledge distillation through a multi-agent Question-Thinking-Solution-Answer pipeline and provides a reusable foundation for future work on LLM-aided RF circuit design.

EduPanel 是一个基于评分量表、以学习者为条件的 LLM 评判器，通过在多个专用 agent 间分解评估流程，对教学质量的各个方面产出可解释的评估结果，其可靠性与中等水平的人类专家相当。E EduPanel, a rubric-grounded, learner-conditioned LLM judge that decomposes evaluation across specialized agents to produce interpretable assessments for different aspects of teaching quality, achieves reliability comparable to a median human expert.

人类视觉是一个闭环：注视点不断被中间假设而非单一快照持续重定向。数十年的心理物理学与认知科学研究表明，主动观察对多种任务至关重要。当代多模态大语言模型 (MLLM) 是否进行主动观察，是一个现有视觉语言基准无法回答的经验问题。我们提出 ActiveVision，一个使 MLLM 主动观察可度量的基准，包含 3 个类别共 17 个任务，任务设计强制进行重复视觉感知……Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception

本文提出 Anchor-Align，通过两个目标增强 BC：Vision-Language Anchoring 从冻结 VLM 副本中蒸馏逐层表示以防止该漂移；Language-Action Alignment 将每个动作目标转换为离散的运动方向标签，并在同一机器人观测上联合训练语言与动作预测。Anchor-Align is proposed, which augments BC with two objectives: Vision-Language Anchoring distills layer-wise representations from a frozen VLM copy to prevent this drift, and Language-Action Alignment converts each action target into a discrete motion-direction label and jointly trains language and action prediction on the same robot observation.

提出 DocOps，一种确定性可验证的评估框架，基于分层分类法，将受真实实践启发的文档操作分解为原子维度与逐级递增的工作流复杂度，从而揭示 Agent 在维护全局文档一致性方面的能力边界。DocOps is introduced, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities that exposes the capability boundaries of agents in maintaining global document consistency.