论文

确认了丰富资源语言与代表性不足语言之间在数学推理性能上存在持续差距，性能更优主要与更强的指令遵循能力相关，并提出了完全开源的数据集、数据采集流程与评估框架。A persistent gap in mathematical reasoning performance between high-resource and underrepresented languages is confirmed, with stronger results largely associated with better instruction-following ability, and a fully open-source dataset, data acquisition pipeline, and evaluation framework is introduced.

该工作提出 HETERQA，一个包含 857 个 QA 对的综合性基准，涵盖五个异构来源的记录检索，并表明 HETERQA 为异构来源下的记录检索提供了有效的测试平台，为未来检索方法留下了显著空间。This work introduces HETERQA, a comprehensive benchmark with 857 QA pairs for record retrieval over five heterogeneous sources and indicates that HETERQA provides an effective testbed for record retrieval over heterogeneous sources and leaves substantial room for future retrieval methods.

静脉识别是一种安全生物特征技术，常受限于标注数据稀缺与成像差异；而面向自然图像设计的增强策略可能破坏其关键的细粒度拓扑与纹理。本文提出 AGVBench，在 5 个公开掌/指静脉数据集、7 种骨干网络（含经典 CNN、视觉 Transformer 及静脉专用模型）上评测 30 种代表性增强策略。结果显示，多图混合类方法（如 MixUp、PuzzleMix……Vein recognition is a secure biometric technology often constrained by limited annotated data and imaging variations. While data augmentation mitigates this, strategies designed for natural images may disrupt the fine-grained topology and textures essential for identity discrimination. We present AGVBench, which evaluates 30 representative augmentation strategies on five public palm- and finger-vein datasets with seven backbone architectures, covering classic CNNs, vision transformers, and vein-specific recognition models. Our results show that multi-image mixing methods (e.g., MixUp, PuzzleMi

提出"自主策略演化"评估范式：在固定交互预算下，由 harness-model Agent 反复编辑可执行策略系统；并在 EvoPolicyGym 中实例化，该基准基于一组紧凑型交互式 RL 环境构建，用于评测 Agent 如何迭代改进已探索策略。This work introduces Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget, and instantiates this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments that evaluates how agents iteratively improve explored policies.

BeyondArena 是首个面向表格数据的统一整体基准，支持多种任务类型（IID、时序、分组），覆盖样本量与特征维度的不同尺度，并涵盖来自广泛学科的多样化特征类型。BeyondArena is the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types from a broad range of disciplines.

提出 PerceptionRubrics——一个基于评分量表的评估框架，旨在弥合饱和的基准分数与真实场景脆弱性之间的差距，并验证了严格的感知保真是可靠生成的前提。PerceptionRubrics is introduced, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness, validating that strict perceptual fidelity is the prerequisite for reliable generation.

所得到的模型在大多数数据集-预测步长组合上优于先前的线性预测器，并在八个基准中的六个上超越 Transformer、MLP 和 CNN 基线；同时它还可作为对数据本身的诊断工具，揭示那些被更大模型默默吸收进其学习参数中的结构。The resulting models beat prior linear forecasters on most dataset-horizon entries and exceed Transformer, MLP, and CNN baselines on six of eight benchmarks, and serve as a diagnostic on the data itself, revealing structures that larger models absorb silently into their learned parameters.

提出 GauntletBench，一个用于评估 Agent 在挑战性场景中泛化能力的 Web 基准，聚焦于三种被低估的能力（时间感知、图形理解与 3D 推理），揭示了当前 Agent 能力与复杂真实场景所需能力之间的巨大差距。GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), is introduced, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios.

一个包含 382 个真实企业任务、覆盖 6 种专业角色和 22 项程序性技能的基准，用于评估技能在任务、角色和模型骨干间的迁移能力，发现部分技能可在任务和模型间广泛泛化，而另一些则专化为角色特定工作流，在迁移时失去效力。A benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones finds that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer.

结果表明仅评估求解器或仅评估答案是不足的：Agent 的差异不仅体现在发现关键 contingency 上，还体现在验证预算的使用、显式提交、类型强制、重复验证、基于证据的报告以及缓解行为等方面。The results show why solver-only or answer-only evaluation is insufficient: agents are distinguished not only by top-contingency discovery, but also by validation-budget use, explicit submission, type coercions, duplicate validations, evidence-backed reporting, and mitigation behavior.

本工作提出 AdvancedMathBench，用于评估 LLM 在高级数学证明上的推理能力；同时引入 VerifierBench，包含 888 条由模型生成的证明轨迹及专家真值，用于评估模型能否正确判断证明有效性并给出合理的验证依据。This work introduces AdvancedMathBench, a benchmark suite designed to evaluate the reasoning capabilities of LLMs on advanced mathematical proofs, and introduces VerifierBench, consisting of 888 model-generated proof trajectories paired with expert ground truth, to evaluate whether models can correctly judge proof validity and provide sound verification rationales.

本文提出 MCLASH，一个多语言道德决策 benchmark，用于捕捉跨语言的文化情境化道德直觉和社会规范；并提出 MET（Multilingual Ethics with Theory-grounded reasoning），一种基于心理学与哲学专家策划的理论依据的两步提示方法。This work introduces MCLASH, a multilingual moral decision-making benchmark to capture culturally situated moral intuitions and social norms across languages, and proposes MET (Multilingual Ethics with Theory-grounded reasoning), a two-step prompting method built on expert-curated, theory-based grounds drawn from psychology and philosophy.

开发了自动化评分流水线，用于评估多种模型，包括开源权重模型、闭源语言模型、视觉语言模型和图像生成模型，结果显示没有单一模型在所有任务类型上占优，且某些任务对所有评估模型仍具挑战性。An automated grading pipeline is developed to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models, and shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models.

提出 LakeQuest，一个经人工校验的基准，包含 9,846 个 QA 对，用于在真实数据湖场景下评估端到端检索与综合 pipeline，并揭示现代问答系统中的关键失效模式。LakeQuest is introduced, a human-validated benchmark of 9,846 QA pairs designed to evaluate the end-to-end retrieve-and-synthesize pipeline over realistic data lakes and exposes critical failure modes in modern QA systems.

提出 SDABench，一个围绕六项能力（描述性、探索性、推断性、推断性、预测性、因果性、机制性）跨五大领域（生物、化学、环境、地理、物理）重新组织评估的基准。SDABench is introduced, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics).

视觉语言模型（VLMs）在 DocVQA、ChartQA、MMLongBench-Doc 等视觉文档理解基准上表现强劲。但真实文档融合长度、布局复杂度、模态、问题难度等多因素，难以将模型失败归因于具体原因。我们提出 SynthDocBench，一个全合成的长上下文视觉文档理解基准，系统性控制文档长度、布局结构、模态构成与问题类型等因子。该基准的构建……Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constr

AgentCompass 被提出，它是一个开源、轻量且可扩展的面向 LLM-based Agent 的评估基础设施，将评估流程围绕三个独立组件组织，从而在不重新实现复杂执行逻辑的前提下支持灵活配置。AgentCompass is introduced, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents that organizes the evaluation process around three independent components, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic.

本文提出一种实用评估协议，将评估重点从任务完成转向经过验证的漏洞发现，可在涵盖多种攻击面与漏洞类型的足够复杂目标上开展评估，并结合结构化真值标注与基于 LLM 的语义匹配来识别漏洞。This paper presents a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes, and combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities.

本文提出 SIS-Bench，一个在统一 self-in-space 表述下评估 UAV 场景具身空间智能的基准，并探索了一种融合光流与视觉特征的运动感知表征，以纳入与自身相关的动态信息。SIS-Bench is introduced, a benchmark for evaluating embodied spatial intelligence in UAV scenarios under a unified self-in-space formulation, and a motion-aware representation that incorporates self-related dynamics through optical flow and visual feature fusion is explored.

重新审视自动 harness 演化流程的评估方法，强调需要采用匹配预算的基线（matched-budget baselines）和留出评估（held-out evaluation），以区分真正的 harness 改进与针对特定基准的搜索和过拟合。The evaluation of automatic harness evolution procedures is revisited and the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting is highlighted.

认为（该早期版本的）GPT-4 属于新一代具备更通用智能的 LLM（与 ChatGPT、谷歌 PaLM 等并列），并讨论了这些模型不断增强的能力及其影响。It is argued that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models, and the rising capabilities and implications of these models are discussed.

在 BIG-bench 上对 OpenAI 的 GPT 模型、Google 内部稠密 Transformer 架构及 Switch 风格稀疏 Transformer 进行评估，模型规模跨越百万至千亿参数，结果显示性能与校准均随规模提升而改善，但绝对水平仍然欠佳。Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.

对 SOTA LLM GPT-4 在医学能力考试与基准数据集上进行全面评估，并通过案例研究定性探索其行为，展示了 GPT-4 解释医学推理、为学生定制个性化讲解以及围绕病例交互式构造新反事实场景的能力。A comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets and explores the behavior of the model qualitatively through a case study that shows the ability of G PT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case.

研究揭示了 LLM 空间能力中的清晰规律：尽管其仍落后于 SOTA 方法，但具有潜力并能同时处理多种空间约束，从而可扩展到异构场景。A clear pattern is revealed in LLM spatial capabilities: while they still lag behind state-of-the-art approaches, they are promising and can handle multiple spatial constraints simultaneously, enabling scaling to heterogeneous setups.

提出 DocOps，一种确定性可验证的评估框架，基于分层分类法，将受真实实践启发的文档操作分解为原子维度与逐级递增的工作流复杂度，从而揭示 Agent 在维护全局文档一致性方面的能力边界。DocOps is introduced, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities that exposes the capability boundaries of agents in maintaining global document consistency.

提出 ProVisE（Protocolized Visual Evaluation），一个与基准无关的框架，通过受协议约束的视觉问答从图像生成模型中抽取答案，并将其解析为与原始指标兼容的结构化预测，揭示了像素空间表达与基于文本推理的互补优势。ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics, is proposed and revealed, revealing complementary strengths of pixel-space expression and text-based reasoning.

本文提出 K12-KGraph，一个从人民教育出版社官方教材中提取的、与课程对齐的知识图谱，覆盖小学、初中和高中阶段的数学、物理、化学和生物，并由此衍生出 K12-Bench，一个包含 23,640 道题的多选基准，涵盖五类任务族，表明文本监督与视觉监督具有互补性。This work introduces K12-KGraph, a curriculum-aligned knowledge graph extracted from official People's Education Press textbooks in mathematics, physics, chemistry, and biology across primary, middle, and high school, and derives K12-Bench, a 23,640-question multi-select benchmark with five task families, showing that textual and visual supervision are complementary.

研究发现，ChatGPT 最成功地被用作数学助手，用于查询事实、充当数学搜索引擎和知识库接口；GPT-4 还可被用于本科水平的数学问题，但在研究生难度的题目上表现不佳。It is found that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a Mathematical search engine and knowledge base interface, and GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty.

EvalPlus——一个用于严格基准测试 LLM 生成代码功能正确性的代码合成评估框架，通过 LLM 与基于 mutation 的策略驱动的自动测试输入生成器，为给定评估数据集补充大量新生成的测试用例。EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code and augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM and mutation-based strategies.

本研究在 Reachy Mini 机器人平台上，通过系统提示、检索增强生成 (RAG) 和有状态的提示编排，将选定的 KBD 需求落地实现，表明 KBD 可以塑造负责任的机器人行为，并有望提升机器人辅助学习中的学习效果。This study operationalized selected KBD requirements in the Reachy Mini robot platform through system prompting, retrieval-augmented generation, and stateful prompt orchestration, indicating that KBD can shape responsible robot behavior and potentially increase learning effectiveness in robot-supported learning.

即使 NVIDIA 控制流 ISA 与重汇聚（reconvergence）机制持续演进，Divergence 仍能保持稳定且可预期的性能开销。Divergence retains a stable and predictable performance cost even as NVIDIA's control-flow ISA and reconvergence mechanisms continue to evolve.

本文提出 InMind，一项包含 125 个任务、经专家校验、跨越十个生活领域的基准，其中 113 个任务基于可引用的公开来源；本文将此类失效模式命名为 implicit-association blind spot，并提出一种极简诊断探针（diagnostic probe），在 query 到达前保持 memory 可见即可恢复大部分性能差距。This work introduces InMind, a 125-task, expert-verified benchmark spanning ten life domains, with 113 tasks grounded in citable public sources, and calls this failure mode the implicit-association blind spot, and introduces a minimal diagnostic probe that keeps memory visible before the query arrives recovers most of the gap.

PerceptionBench 通过诊断前沿 MLLMs 在 42 项现有基准上响应中的最早失效点，并构建一个感知分支定义十种原子感知能力的错误分类法，为衡量与诊断 MLLMs 视觉感知边界提供了一项能力级（capability-level）标准。PerceptionBench provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs, by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks and constructing an error taxonomy whose perception branch defines ten atomic perceptual capabilities.

介绍 RepoReasoner，一个用于评估仓库级代码推理的 benchmark，评估两种互补能力：Output Prediction，衡量跨文件的细粒度、有状态执行推理；Call Chain Prediction，在噪声上下文下评估高层架构依赖理解。RepoReasoner is introduced, a benchmark for evaluating repository-level code reasoning that assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context.

提出 PALATE（Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation），一个基于用户模拟器的可扩展 RPA benchmark，可针对具体 user-RPA 对给出可解释的评估，避免将系统压缩为单一、与用户无关的排名。PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation with Tailored Evaluation), a scalable RPA benchmark built on user simulators, produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.

LlamaExtract Agentic Plus在三项指标上均排名第一，准确度可与coding agent相媲美而成本仅为其一小部分，是首个同时在大规模下对数值准确性、记录完整性、grounding与实测成本进行打分的方法。LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost, and is the first to score value accuracy, record completeness at scale, grounding, and measured cost together.