论文

本文提出 M$^3$Exam，一个以查询为中心、基于真实用户-Agent 交互构建的多模态对话记忆基准，涵盖跨模态定位与隐式信息推断等多维度评估。M$^3$Exam is introduced, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference.

本文命名并研究这两条研究脉络之间的模式：其递归单元是配备文件系统工具、代码执行与规划的完整 Agent harness，而非无工具的模型调用，并给出针对长上下文推理的受控评估。This work names and studies the pattern between these two lines of work, where the recursive unit is a full agent harness with filesystem tools, code execution, and planning rather than a model call with no tools, and provides a controlled evaluation on long-context reasoning.

本文提出 ForeSci，一个时间受控的基准，用于评估 LLM Agent 是否能从历史证据中做出前瞻性研究判断，并在四种骨干模型上评测原生 LLM、Hybrid RAG 以及三种 research-agent 适配方案。This work introduces ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence, and evaluates native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones.

本文提出 MMLongEmbed，首个面向长上下文场景评估 MEM 的综合基准，并发现现有架构严重依赖浅层特征匹配，难以捕捉深层语义与结构依赖。This work introduces MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios, and finds that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies.

一个两级框架将手动 harness 工程转变为自动化 harness 工程，并更进一步——将"自动化本身的设计"也自动化。A two-level framework shifts manual harness engineering into automated harness engineering, and takes one step further --automating the design of the automation itself.

提出 Agentic Harness Engineering（AHE），一个通过三个相互匹配的 observability 支柱应对 harness 工程挑战的闭环，将每一次编辑转化为可证伪的契约，使 harness 演进能够自主进行而不退化为试错。Agentic Harness Engineering (AHE) is introduced, a closed loop that addresses harness engineering challenges through three matched observability pillars that turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error.

本文提出 Natural-Language Agent Harnesses，即可编辑的、描述运行级 harness 策略的文档，以及 Intelligent Harness Runtime（IHR），一个将上述文档解释为 agent 调用、交接、状态更新、验证门控与 artifact 契约的共享运行时。This paper introduces Natural-Language Agent Harnesses, editable documents that describe run-level harness policy, and Intelligent Harness Runtime (IHR), a shared runtime that interprets these documents into agent calls, handoffs, state updates, validation gates, and artifact contracts.

文中指出，安全的 LLM Agent 需要显式的信任边界、原则化的权限控制、具备溯源能力的 state 管理，以及与真实运行场景对齐的评估实践；现有 benchmark 仍未能充分覆盖长程、具状态、对部署敏感的风险。It is argued that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings, as well as existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks.

在 DeNovoSWE 上对 Qwen3-30B-A3B 进行微调可显著提升长程 SWE 性能，在具有挑战性的 BeyondSWE-Doc2Repo benchmark 上将其得分从 5.8% 提升至 47.2%。Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

本综述系统梳理了超越基础 Skill 创建的 Skill 演化与评估图景，将其归纳为四种范式：执行反馈、轨迹蒸馏、压缩与强化学习，并指出了构建可泛化、高效且可验证安全的 Skill 生态的开放方向。This survey systematically examines the landscape of skill evolution and evaluation beyond foundational skill creation into four distinct paradigms, spanning execution feedback, trajectory distillation, compression, and reinforcement learning, and identifies open directions for building skill ecosystems that are generalizable, efficient, and verifiably safe.

文章认为基于实体的分解能形成对原始信息更精炼的表示，并有助于降低索引与生成过程中的噪声；在端到端 QA 评测中，VectorRAG 表现优于标准 GraphRAG，且接近当前 SOTA 图方法的效果。It is argued that entity-based decomposition yields a more distilled representation of original information, and additionally serves to reduce noise in the indexing, and generation process, and on end to end QA evaluation VectorRAG performs better than standard GraphRAG and almost as good as current SOTA graph-based solutions.

介绍 EvoArena 基准套件，将环境变化建模为跨终端、软件与社会领域的渐进式更新序列；并提出 EvoMem，一种基于 patch 的记忆范式，将记忆演化记录为结构化的更新历史，使 Agent 能通过记忆的变化推理环境的演化。EvoArena is introduced, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains, and EvoMem is proposed, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.

本研究对新兴的内存键值存储进行了全面的性能与可行性评估，突出了性能、兼容性与长期可行性（包括项目成熟度、社区支持与持续开发）之间的权衡。This study presents a comprehensive performance and viability assessment of the emerging in-memory key-value stores and highlights trade-offs between performance, compatibility, and long-term viability, including project maturity, community support, and sustained development.

本文提出了 SeedRG，一个用于缓解 knowledge leakage 并应对 benchmark aging 问题的半合成 benchmark 生成 pipeline。SeedRG is introduced, a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging.

本文聚焦 LLM 驱动的 kernel generation 领域，给出现有方法的结构化综述，涵盖 LLM-based 方法与 agentic optimization workflow，并系统梳理了支撑该领域学习与评测的数据集与 benchmark。This survey addresses the gap in LLM-driven kernel generation by providing a structured overview of existing approaches, spanning LLM-based approaches and agentic optimization workflows, and systematically organizing the datasets and benchmarks that underpin learning and evaluation in this domain.

本文提出一个无偏的多进程 evaluation 框架，能够有效分散 client 端负载，从而在每秒数千次 query 以上的生产规模下实现对 LLM 的精确、可复现 profiling。This work proposes an unbiased, multi-process evaluation framework that effectively distributes client-side load, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.

本文通过大规模超参数搜索，系统地重新评估了 Vanilla LoRA 以及九个代表性 LoRA 变体，发现不同 LoRA 方法偏好的学习率区间各异，并将最优学习率区间的差异归因于最大 Hessian 特征值的变化，与经典学习理论相吻合。This work systematically re-evaluate nine representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches, finding that different LoRA methods favor distinct learning rate ranges and attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.

本工作提出 AlphaEval，一个基于真实生产环境的基准，包含来自七家在其核心业务中部署 AI Agent 的公司的 94 个任务，覆盖六个 O*NET (Occupational Information Network) 领域；并贡献了一套从需求到基准的构建框架，将从需求到评估的完整流程标准化。This work presents AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains, and contributes a requirement-to-benchmark construction framework that standardizes the entire pipeline from requirement to evaluation.

提出 RoboDojo，一个面向通用机器人操作策略综合评估的仿真-真机统一基准，将 30 种策略集成到 XPolicyLab 并在 RoboDojo 上进行评测，建立了公开的排行榜与系统性的策略性能分析。RoboDojo is introduced, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies that integrates 30 policies into XPolicyLab and evaluates them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance.

确认了丰富资源语言与代表性不足语言之间在数学推理性能上存在持续差距，性能更优主要与更强的指令遵循能力相关，并提出了完全开源的数据集、数据采集流程与评估框架。A persistent gap in mathematical reasoning performance between high-resource and underrepresented languages is confirmed, with stronger results largely associated with better instruction-following ability, and a fully open-source dataset, data acquisition pipeline, and evaluation framework is introduced.

该工作提出 HETERQA，一个包含 857 个 QA 对的综合性基准，涵盖五个异构来源的记录检索，并表明 HETERQA 为异构来源下的记录检索提供了有效的测试平台，为未来检索方法留下了显著空间。This work introduces HETERQA, a comprehensive benchmark with 857 QA pairs for record retrieval over five heterogeneous sources and indicates that HETERQA provides an effective testbed for record retrieval over heterogeneous sources and leaves substantial room for future retrieval methods.

结果表明，强大的 LLM 能够产出一系列合理思路，但其范围仍比人类研究品味更窄，并存在系统性偏移。It is suggested that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.

静脉识别是一种安全生物特征技术，常受限于标注数据稀缺与成像差异；而面向自然图像设计的增强策略可能破坏其关键的细粒度拓扑与纹理。本文提出 AGVBench，在 5 个公开掌/指静脉数据集、7 种骨干网络（含经典 CNN、视觉 Transformer 及静脉专用模型）上评测 30 种代表性增强策略。结果显示，多图混合类方法（如 MixUp、PuzzleMix……Vein recognition is a secure biometric technology often constrained by limited annotated data and imaging variations. While data augmentation mitigates this, strategies designed for natural images may disrupt the fine-grained topology and textures essential for identity discrimination. We present AGVBench, which evaluates 30 representative augmentation strategies on five public palm- and finger-vein datasets with seven backbone architectures, covering classic CNNs, vision transformers, and vein-specific recognition models. Our results show that multi-image mixing methods (e.g., MixUp, PuzzleMi

提出"自主策略演化"评估范式：在固定交互预算下，由 harness-model Agent 反复编辑可执行策略系统；并在 EvoPolicyGym 中实例化，该基准基于一组紧凑型交互式 RL 环境构建，用于评测 Agent 如何迭代改进已探索策略。This work introduces Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget, and instantiates this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments that evaluates how agents iteratively improve explored policies.

应在真实类别分布下使用跨网络评估来判断部署就绪度，而非仅依赖域内准确率。Deployment readiness should be assessed using cross-network evaluation under realistic class distributions, rather than within-domain accuracy alone, to suggest deployment readiness should be assessed using cross-network evaluation under realistic class distributions, rather than within-domain accuracy alone.

BeyondArena 是首个面向表格数据的统一整体基准，支持多种任务类型（IID、时序、分组），覆盖样本量与特征维度的不同尺度，并涵盖来自广泛学科的多样化特征类型。BeyondArena is the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types from a broad range of disciplines.

提出 PerceptionRubrics——一个基于评分量表的评估框架，旨在弥合饱和的基准分数与真实场景脆弱性之间的差距，并验证了严格的感知保真是可靠生成的前提。PerceptionRubrics is introduced, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness, validating that strict perceptual fidelity is the prerequisite for reliable generation.

所得到的模型在大多数数据集-预测步长组合上优于先前的线性预测器，并在八个基准中的六个上超越 Transformer、MLP 和 CNN 基线；同时它还可作为对数据本身的诊断工具，揭示那些被更大模型默默吸收进其学习参数中的结构。The resulting models beat prior linear forecasters on most dataset-horizon entries and exceed Transformer, MLP, and CNN baselines on six of eight benchmarks, and serve as a diagnostic on the data itself, revealing structures that larger models absorb silently into their learned parameters.

提出 GauntletBench，一个用于评估 Agent 在挑战性场景中泛化能力的 Web 基准，聚焦于三种被低估的能力（时间感知、图形理解与 3D 推理），揭示了当前 Agent 能力与复杂真实场景所需能力之间的巨大差距。GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), is introduced, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios.

研究发现重建质量与表示质量是解耦的；在所考察的任务中，没有任何单一方法能够在所有任务上稳定取得最优表现。It is found that reconstruction and representation quality are decoupled, and no single method consistently performs best across the tasks considered here and no single method consistently performs best across the tasks considered here.

一个包含 382 个真实企业任务、覆盖 6 种专业角色和 22 项程序性技能的基准，用于评估技能在任务、角色和模型骨干间的迁移能力，发现部分技能可在任务和模型间广泛泛化，而另一些则专化为角色特定工作流，在迁移时失去效力。A benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones finds that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer.

结果表明仅评估求解器或仅评估答案是不足的：Agent 的差异不仅体现在发现关键 contingency 上，还体现在验证预算的使用、显式提交、类型强制、重复验证、基于证据的报告以及缓解行为等方面。The results show why solver-only or answer-only evaluation is insufficient: agents are distinguished not only by top-contingency discovery, but also by validation-budget use, explicit submission, type coercions, duplicate validations, evidence-backed reporting, and mitigation behavior.

本文提出 AdvancedMathBench，一个用于评估高级数学推理能力的 benchmark 套件，并推出 VerifierBench，包含 888 条模型生成的证明轨迹及专家 ground truth，用于评估模型能否正确判断证明有效性并给出合理的验证理由。This work introduces AdvancedMathBench, a benchmark suite designed to evaluate advanced mathematical reasoning capabilities, and introduces VerifierBench, consisting of 888 model-generated proof trajectories paired with expert ground truth, to evaluate whether models can correctly judge proof validity and provide sound verification rationales.

本文提出 MCLASH，一个多语言道德决策 benchmark，用于捕捉跨语言的文化情境化道德直觉和社会规范；并提出 MET（Multilingual Ethics with Theory-grounded reasoning），一种基于心理学与哲学专家策划的理论依据的两步提示方法。This work introduces MCLASH, a multilingual moral decision-making benchmark to capture culturally situated moral intuitions and social norms across languages, and proposes MET (Multilingual Ethics with Theory-grounded reasoning), a two-step prompting method built on expert-curated, theory-based grounds drawn from psychology and philosophy.

开发了自动化评分流水线，用于评估多种模型，包括开源权重模型、闭源语言模型、视觉语言模型和图像生成模型，结果显示没有单一模型在所有任务类型上占优，且某些任务对所有评估模型仍具挑战性。An automated grading pipeline is developed to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models, and shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models.

介绍其潜在原因的理论基础，阐明强化学习算法的渐近性能在性能排名与数据规模之间不存在单调关系。The theoretical foundations of the underlying causes outlining that the asymptotic performance of reinforcement learning algorithms does not have a monotone relationship between performance rankings and data-regimes are introduced.