论文

在 BIG-bench 上对 OpenAI 的 GPT 模型、Google 内部稠密 Transformer 架构及 Switch 风格稀疏 Transformer 进行评估，模型规模跨越百万至千亿参数，结果显示性能与校准均随规模提升而改善，但绝对水平仍然欠佳。Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.

本文推出 ScanNet，一个 RGB-D 视频数据集，包含 1513 个场景中的 250 万视角，标注有三维相机位姿、表面重建与语义分割，并表明使用该数据可在多项三维场景理解任务上取得 SOTA 性能。This work introduces ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations, and shows that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks.

对 SOTA LLM GPT-4 在医学能力考试与基准数据集上进行全面评估，并通过案例研究定性探索其行为，展示了 GPT-4 解释医学推理、为学生定制个性化讲解以及围绕病例交互式构造新反事实场景的能力。A comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets and explores the behavior of the model qualitatively through a case study that shows the ability of G PT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case.

研究发现 ChatGPT 在大多数任务上以零样本学习优于其他 LLM，在部分任务上甚至超过微调模型，并且对非拉丁文字语言的理解能力优于生成能力。It is found that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks and is better at understanding non-Latin script languages than generating them.

本文介绍 Matterport3D，一个大规模 RGB-D 数据集，包含来自 90 个建筑物级场景共 194,400 张 RGB-D 图像的 10,800 个全景视图，可支持多种监督与自监督计算机视觉任务，包括关键点匹配、视角重叠预测、由彩色图像预测法线、语义分割和区域分类。Matterport3D is introduced, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400RGB-D images of 90 building-scale scenes that enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.

实验表明，EvolvingWorld 能够通过有效维持持久且一致的角色与世界发展，提升长程模拟能力。Experiments show that EvolvingWorld can improve long-horizon simulation by effectively maintaining persistent, coherent character and world development.

OCT-Bench能够对MLLM进行全面且细粒度的评估，为识别能力瓶颈和推进临床可信的OCT理解奠定基础。OCT-Bench enables comprehensive and fine-grained evaluation of MLLMs, providing a foundation for identifying capability bottlenecks and advancing clinically grounded OCT understanding.

本文提出一个时间感知的音频 LLM，能够基于大规模合成监督（来自级联 pipeline）在长达 120 分钟的输入上回答带有显式时间戳的问题，并在短时长和长时长 benchmark 上取得强劲的时间定位准确率。This work presents a time-aware audio LLM that answers questions with explicit timestamps over up to 120 minutes of input using large-scale synthetic supervision from a cascaded pipeline and achieves strong temporal-grounding accuracy on short and long benchmarks.

研究揭示了 LLM 空间能力中的清晰规律：尽管其仍落后于 SOTA 方法，但具有潜力并能同时处理多种空间约束，从而可扩展到异构场景。A clear pattern is revealed in LLM spatial capabilities: while they still lag behind state-of-the-art approaches, they are promising and can handle multiple spatial constraints simultaneously, enabling scaling to heterogeneous setups.

本文提出 Manager Coercion Benchmark：被测 manager 拥有一个良性任务并有完成动机，但唯一能够礼貌且坚定拒绝执行任务的 agent，正是被测 manager 本身。The Manager Coercion Benchmark is introduced: the manager under test needs a benign task done and has an incentive to deliver, but the only agent that can do it politely and immovably declines, but the only agent that can do it politely and immovably declines is the manager under test.

本文是面向业务流程中长时间运行、有状态、多步生成式 AI 系统的基于图的工作流路径实践指南，并通过三个可执行示例展示类型化状态、条件路由、确定性工具、重试、中断、检查点与 trace 如何协同工作。This paper is a practitioner guide to graph-based workflow pathways for long-running, stateful, multi-step generative AI systems in business processes and presents three executable recipes to show how typed state, conditional routing, deterministic tools, retries, interrupts, checkpoints, and traces fit together.

本文提出 RF-Agent，通过多 Agent 的 Question-Thinking-Solution-Answer 流水线，基于教材驱动的知识蒸馏来弥补 RF 领域专用推理的空白，为面向 LLM 辅助 RF 电路设计的未来工作提供了可复用的基础。RF-Agent is presented, which addresses the gap in domain-specific RF reasoning through textbook-driven knowledge distillation through a multi-agent Question-Thinking-Solution-Answer pipeline and provides a reusable foundation for future work on LLM-aided RF circuit design.

EduPanel 是一个基于评分量表、以学习者为条件的 LLM 评判器，通过在多个专用 agent 间分解评估流程，对教学质量的各个方面产出可解释的评估结果，其可靠性与中等水平的人类专家相当。E EduPanel, a rubric-grounded, learner-conditioned LLM judge that decomposes evaluation across specialized agents to produce interpretable assessments for different aspects of teaching quality, achieves reliability comparable to a median human expert.

人类视觉是一个闭环：注视点不断被中间假设而非单一快照持续重定向。数十年的心理物理学与认知科学研究表明，主动观察对多种任务至关重要。当代多模态大语言模型 (MLLM) 是否进行主动观察，是一个现有视觉语言基准无法回答的经验问题。我们提出 ActiveVision，一个使 MLLM 主动观察可度量的基准，包含 3 个类别共 17 个任务，任务设计强制进行重复视觉感知……Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception

本文提出 Anchor-Align，通过两个目标增强 BC：Vision-Language Anchoring 从冻结 VLM 副本中蒸馏逐层表示以防止该漂移；Language-Action Alignment 将每个动作目标转换为离散的运动方向标签，并在同一机器人观测上联合训练语言与动作预测。Anchor-Align is proposed, which augments BC with two objectives: Vision-Language Anchoring distills layer-wise representations from a frozen VLM copy to prevent this drift, and Language-Action Alignment converts each action target into a discrete motion-direction label and jointly trains language and action prediction on the same robot observation.

提出 DocOps，一种确定性可验证的评估框架，基于分层分类法，将受真实实践启发的文档操作分解为原子维度与逐级递增的工作流复杂度，从而揭示 Agent 在维护全局文档一致性方面的能力边界。DocOps is introduced, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities that exposes the capability boundaries of agents in maintaining global document consistency.

提出 ENTRAP-VL（ENTRainment Assessment Probe for Vision and Language），一个由人工策展的 1,500 条数据的数据集，涵盖八个类别，按一个跨双轴的分类体系组织，并划分为文本诱发流和视觉诱发流。ENTRAP-VL (ENTRainment Assessment Probe for Vision and Language), a manually curated dataset of 1,500 items across eight categories, organized by a taxonomy that spans two axes and split into a textual-entrainment stream and a visual-entrainment stream, is introduced.

提出 ProVisE（Protocolized Visual Evaluation），一个与基准无关的框架，通过受协议约束的视觉问答从图像生成模型中抽取答案，并将其解析为与原始指标兼容的结构化预测，揭示了像素空间表达与基于文本推理的互补优势。ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics, is proposed and revealed, revealing complementary strengths of pixel-space expression and text-based reasoning.

本文提出 K12-KGraph，一个从人民教育出版社官方教材中提取的、与课程对齐的知识图谱，覆盖小学、初中和高中阶段的数学、物理、化学和生物，并由此衍生出 K12-Bench，一个包含 23,640 道题的多选基准，涵盖五类任务族，表明文本监督与视觉监督具有互补性。This work introduces K12-KGraph, a curriculum-aligned knowledge graph extracted from official People's Education Press textbooks in mathematics, physics, chemistry, and biology across primary, middle, and high school, and derives K12-Bench, a 23,640-question multi-select benchmark with five task families, showing that textual and visual supervision are complementary.

研究发现，ChatGPT 最成功地被用作数学助手，用于查询事实、充当数学搜索引擎和知识库接口；GPT-4 还可被用于本科水平的数学问题，但在研究生难度的题目上表现不佳。It is found that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a Mathematical search engine and knowledge base interface, and GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty.

EvalPlus——一个用于严格基准测试 LLM 生成代码功能正确性的代码合成评估框架，通过 LLM 与基于 mutation 的策略驱动的自动测试输入生成器，为给定评估数据集补充大量新生成的测试用例。EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code and augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM and mutation-based strategies.

提出 DataPrep-Bench，首个统一基准，在共享的下游任务 grounding 协议下，对 LLM 驱动的数据准备在六个领域、多种 base model 上的两类能力进行联合评估。DataPrep-Bench is introduced, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models of LLM-driven data preparation.

本研究考察是否存在一条实用路径，在没有坐标界面、且无需高成本区域级监督的条件下提升归因效果，并指出了这样一条可行路径。A study investigates whether there is a practical path to improve attribution without a coordinate interface and without costly region-level supervision, and indicates a practical path to improve attribution without a coordinate interface and without costly region-level supervision.

即使 NVIDIA 控制流 ISA 与重汇聚（reconvergence）机制持续演进，Divergence 仍能保持稳定且可预期的性能开销。Divergence retains a stable and predictable performance cost even as NVIDIA's control-flow ISA and reconvergence mechanisms continue to evolve.

本文提出 IndicTalk，目前最大的多语言印度语码混合（code-mixed）会话语料库，包含超过 13,28,604 段事件驱动的多轮对话，覆盖 9 种印度语言的 18 种语言变体，将公开发布以支持低资源印度语多语言会话 AI 的开发与评估。IndicTalk is presented, one of the largest multilingual Indic code-mixed conversational corpora, comprising over 13,28,604 event-grounded multi-turn conversations across 18 language varieties covering 9 Indic languages and will be released to support the development and evaluation of multilingual conversational AI for underrepresented Indic languages.

本文提出 InMind，一项包含 125 个任务、经专家校验、跨越十个生活领域的基准，其中 113 个任务基于可引用的公开来源；本文将此类失效模式命名为 implicit-association blind spot，并提出一种极简诊断探针（diagnostic probe），在 query 到达前保持 memory 可见即可恢复大部分性能差距。This work introduces InMind, a 125-task, expert-verified benchmark spanning ten life domains, with 113 tasks grounded in citable public sources, and calls this failure mode the implicit-association blind spot, and introduces a minimal diagnostic probe that keeps memory visible before the query arrives recovers most of the gap.

PerceptionBench 通过诊断前沿 MLLMs 在 42 项现有基准上响应中的最早失效点，并构建一个感知分支定义十种原子感知能力的错误分类法，为衡量与诊断 MLLMs 视觉感知边界提供了一项能力级（capability-level）标准。PerceptionBench provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs, by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks and constructing an error taxonomy whose perception branch defines ten atomic perceptual capabilities.

介绍 RepoReasoner，一个用于评估仓库级代码推理的 benchmark，评估两种互补能力：Output Prediction，衡量跨文件的细粒度、有状态执行推理；Call Chain Prediction，在噪声上下文下评估高层架构依赖理解。RepoReasoner is introduced, a benchmark for evaluating repository-level code reasoning that assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context.

介绍 HANDBOOK_md，一个包含 65 个 agentic 任务的 benchmark，模拟员工遵循公司手册的方式；每项任务对 10 份基础手册之一进行修改，变动评分所依赖的具体规则和阈值，因此没有任何两个任务共享同一套策略。HANDBOOK_md is presented, a benchmark of 65 agentic tasks modeled on how employees follow company handbooks, and every task modifies one of 10 base handbooks, altering the specific rules and thresholds on which grading depends, so no two tasks share the same set of policies.

一项受控的种子干预实验发现，与随机非 gold 上下文相比，由检索得到的初始上下文以更少的后续探索取得更高的文件 F1，而 oracle gold 上下文仍存在可观的提升空间。A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.

本文介绍 CLBench-V，一个多模态上下文学习 benchmark，围绕三个维度组织任务——上下文 grounding、新信息应用与新知识学习——以解决定位上下文使用失效位置的难题。This work introduces CLBench-V, a benchmark for multimodal context learning that addresses the difficulty of localizing where context use breaks down by organizing tasks around three dimensions: context grounding, new information application, and new knowledge learning.

介绍 StealthBench，一个跨越六个 OPSEC 维度衡量自主攻击性安全 agent 操作隐蔽性的 benchmark，并以公共 benchmark 形式发布，以支持隐蔽感知 agent 的开发及自主攻击性安全部署中的自动化 OPSEC 监控。StealthBench, a benchmark that measures operational stealth in autonomous offensive-security agents across six operational security (OPSEC) dimensions, is introduced and released as a public benchmark to support both the development of stealth-aware agents and automated OPSEC monitoring for autonomous offensive-security deployments.

提出知识对齐的多跳检索器 KAMR，区分受 query 强约束的 anchor triplet 和弱对齐但在结构上与 anchor 相连的 connected triplet，持续提升多跳检索及下游问答性能。A knowledge-aligned multi-hop retriever, KAMR, which distinguishes anchor triplets that are strongly constrained by the query from connected triplets that are weakly aligned yet structurally linked to the anchors, which consistently improves multi-hop retrieval and downstream question answering performance.

提出 SpecFirst，一个两阶段框架，在代码合成前强制进行需求 elicitation，并证明显式需求工程阶段是从零构建程序的有效范式。This work presents SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis, and demonstrates that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.

提出 PALATE（Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation），一个基于用户模拟器的可扩展 RPA benchmark，可针对具体 user-RPA 对给出可解释的评估，避免将系统压缩为单一、与用户无关的排名。PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation with Tailored Evaluation), a scalable RPA benchmark built on user simulators, produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.