论文

本文提出 M$^3$Exam，一个以查询为中心、基于真实用户-Agent 交互构建的多模态对话记忆基准，涵盖跨模态定位与隐式信息推断等多维度评估。M$^3$Exam is introduced, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference.

本文命名并研究这两条研究脉络之间的模式：其递归单元是配备文件系统工具、代码执行与规划的完整 Agent harness，而非无工具的模型调用，并给出针对长上下文推理的受控评估。This work names and studies the pattern between these two lines of work, where the recursive unit is a full agent harness with filesystem tools, code execution, and planning rather than a model call with no tools, and provides a controlled evaluation on long-context reasoning.

本文提出 ForeSci，一个时间受控的基准，用于评估 LLM Agent 是否能从历史证据中做出前瞻性研究判断，并在四种骨干模型上评测原生 LLM、Hybrid RAG 以及三种 research-agent 适配方案。This work introduces ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence, and evaluates native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones.

本文提出 MMLongEmbed，首个面向长上下文场景评估 MEM 的综合基准，并发现现有架构严重依赖浅层特征匹配，难以捕捉深层语义与结构依赖。This work introduces MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios, and finds that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies.

一个两级框架将手动 harness 工程转变为自动化 harness 工程，并更进一步——将"自动化本身的设计"也自动化。A two-level framework shifts manual harness engineering into automated harness engineering, and takes one step further --automating the design of the automation itself.

提出 Agentic Harness Engineering（AHE），一个通过三个相互匹配的 observability 支柱应对 harness 工程挑战的闭环，将每一次编辑转化为可证伪的契约，使 harness 演进能够自主进行而不退化为试错。Agentic Harness Engineering (AHE) is introduced, a closed loop that addresses harness engineering challenges through three matched observability pillars that turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error.

本文提出 Natural-Language Agent Harnesses，即可编辑的、描述运行级 harness 策略的文档，以及 Intelligent Harness Runtime（IHR），一个将上述文档解释为 agent 调用、交接、状态更新、验证门控与 artifact 契约的共享运行时。This paper introduces Natural-Language Agent Harnesses, editable documents that describe run-level harness policy, and Intelligent Harness Runtime (IHR), a shared runtime that interprets these documents into agent calls, handoffs, state updates, validation gates, and artifact contracts.

本文对 LLM agent 评估这一新兴领域进行了深入综述，提出一个二维分类体系，沿评估目标维度组织已有工作，为系统性评估提供框架，使研究者与从业者能够面向真实场景部署评估 LLM agent。An in-depth overview of the emerging field of LLM agent evaluation is provided, introducing a two-dimensional taxonomy that organizes existing work along evaluation objectives and provides a framework for systematic assessment, enabling researchers and practitioners to evaluate LLM agents for real-world deployment.

文中指出，安全的 LLM Agent 需要显式的信任边界、原则化的权限控制、具备溯源能力的 state 管理，以及与真实运行场景对齐的评估实践；现有 benchmark 仍未能充分覆盖长程、具状态、对部署敏感的风险。It is argued that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings, as well as existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks.

在 DeNovoSWE 上对 Qwen3-30B-A3B 进行微调可显著提升长程 SWE 性能，在具有挑战性的 BeyondSWE-Doc2Repo benchmark 上将其得分从 5.8% 提升至 47.2%。Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

本综述系统梳理了超越基础 Skill 创建的 Skill 演化与评估图景，将其归纳为四种范式：执行反馈、轨迹蒸馏、压缩与强化学习，并指出了构建可泛化、高效且可验证安全的 Skill 生态的开放方向。This survey systematically examines the landscape of skill evolution and evaluation beyond foundational skill creation into four distinct paradigms, spanning execution feedback, trajectory distillation, compression, and reinforcement learning, and identifies open directions for building skill ecosystems that are generalizable, efficient, and verifiably safe.

文章认为基于实体的分解能形成对原始信息更精炼的表示，并有助于降低索引与生成过程中的噪声；在端到端 QA 评测中，VectorRAG 表现优于标准 GraphRAG，且接近当前 SOTA 图方法的效果。It is argued that entity-based decomposition yields a more distilled representation of original information, and additionally serves to reduce noise in the indexing, and generation process, and on end to end QA evaluation VectorRAG performs better than standard GraphRAG and almost as good as current SOTA graph-based solutions.

介绍 EvoArena 基准套件，将环境变化建模为跨终端、软件与社会领域的渐进式更新序列；并提出 EvoMem，一种基于 patch 的记忆范式，将记忆演化记录为结构化的更新历史，使 Agent 能通过记忆的变化推理环境的演化。EvoArena is introduced, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains, and EvoMem is proposed, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.

本文测试在模型作答时阻止其看到选项标签能否消除位置影响并进而提升性能，并评估了两种不同的偏置缓解策略。This paper test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance, and evaluates two different strategies for mitigating bias.

该工作数学形式化了 Transformer 如何执行抽象推理，并提出一种新颖的严格几何签名用于评估事实可靠性，证明了深度 LLM 潜空间天然组织为小世界网络。This work mathematically formalizes how transformers execute abstract reasoning and provides a novel, strictly geometric signature for evaluating factual reliability, proving that deep LLM latent spaces natively organize into Small-World networks.

HarnessRisk 是一个面向生命周期的 benchmark，将 agent harness 安全组织为六个运行阶段，包括 Harness Configuration、Capability Extension、Runtime Operation、State Persistence、Action Control 和 Incident Recovery，发现显式的风险识别并不能可靠地带来安全的行动——某些配置在超过 90% 的运行中检测到风险，同时仍保留显著的攻击成功率。HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery, finds that explicit risk recognition does not reliably lead to safe action as some configurations detect risks in more than 90% of runs while retaining substantial attack success.

PTXBench 提供了一个可审计的测试平台，用于衡量并提升 LLM 利用持续演进 GPU 架构的能力，并表明各 LLM 在架构特定 PTX 能力上仍参差不齐。PTXBench provides an auditable testbed for measuring and improving LLMs'ability to exploit evolving GPU architectures, and shows that architecture-specific PTX capability remains uneven.

实验表明，仅提供触觉本身并不能确保有效的多模态融合，SoftVTBench 为研究策略不仅能否成功，还在于其如何与可形变物体物理交互，以及触觉在何时改善这种交互，提供了统一的视触觉资源Results show that making touch available does not by itself ensure effective multimodal fusion, and SoftVTBench provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.

本文提出 Zetta，一种闭环具身 harness，在保持基础策略冻结的同时在线演化基于代码的运行时评判器与恢复技能，表明闭环 harness 的自进化为可靠的物理智能开辟了一条可扩展的路径Zetta is presented, a closed-loop embodied harness that evolves code-based runtime critics and recovery skills online while keeping the base policy frozen, and shows that closed-loop harness self-evolution opens a scaling path for reliable physical intelligence.

提出首个 MuseCP 评估框架，涵盖四类音乐 facet，使用细粒度且量身定制的指标来捕捉音乐属性的细微变化，并希望为开发更有效、更可靠、具备强大 MuseCP 能力的音乐编辑策略提供实践指导。The first MuseCP evaluation framework is introduced that covers four categories of music facets with fine-grained and well-tailored metrics to capture nuanced changes in music attributes and hopes it can offer practical guidance for developing more effective and reliable music editing strategies with strong MuseCP capability.

本文提出一种量化基准优化的方法论，聚焦于音频对参考转写不充分确定的情形，指出高性能模型会表现出基准条件化行为，从而虚高基准得分，却未必反映通用转写能力的真正提升。This work presents a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript, and indicates that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

选取三个前沿混合专家模型在低资源语言上进行推理微调，提出六个可度量的行为维度，且每维度均设门拒绝任何与输出长度相关的指标，并报告其自家评测工具为何失效。Take three frontier mixture-of-experts models and fine-tune them to reason in a low-resource language and propose six behavioural dimensions that make changes measurable, each gated to reject any metric that correlates with output length, and report how their own instruments lied.

长视频理解任务超越了孤立事件的检索，需要追踪不断演化的叙事并解读可能隐含的社会含义。然而，现有 benchmark 很少联合评估这些能力，尤其是在高语境、非英语媒体中。为弥补这一空白，我们提出 NARU，一个用于评估日语长视频中叙事演化与文化理解推理的 benchmark。NARU 包含 1,481 个问题，源自 155 个总时长 146.8 小时的视频，涵盖四个叙事维度和五个文化维度Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimens

我们提出 TinyCast，一个注意力无关的零样本预测器，仅用 146,505 个参数输出预测分布，其前提是在该规模下，上下文中的周期结构值得通过计算而非学习方式得到。一个零参数谱检测器给出主导周期，上下文按其相位进行折叠，再由一个膨胀卷积编码器和一个分块自回归分位数解码器建模其余部分。它在 GIFT-Eval 榜单上所有可确认参数量的零样本条目中体积最小；在概率准确性方面，它划定了 size-accuracy 前沿。We introduce TinyCast, an attention-free zero-shot forecaster that emits a predictive distribution from 146,505 parameters, on the premise that at this size the periodic structure of a context is worth computing rather than learning. A zero-parameter spectral detector supplies the dominant periods, the context is folded on their phase, and a dilated convolutional encoder and a block-autoregressive quantile decoder model the rest. It is smaller than every zero-shot entry on the GIFT-Eval board whose parameter count can be established. On probabilistic accuracy it defines the size-accuracy front

现代 LLM Agent 的改进通常依赖于人工修改 prompt、工具或工作流，而围绕模型的可执行支架——harness——在部署后一般被视为固定不变的产物。本文研究一种替代方案：harness 是任务特定的且持续可进化的，每个任务族维护各自的 harness，通过固定的任务注入接缝在不同迭代间热替换，并依据环境反馈进行改写。我们提出分层自改进（HSI），在该框架中，一个冻结的 LLM M 在三层Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the harness---is typically treated as a fixed artifact after deployment. This work studies an alternative where the harness is task-specific and continuously evolvable: each task family maintains its own harness, which is hot-swapped across iterations through a fixed task-injection seam and rewritten using environment feedback. We introduce Hierarchical Self-Improvement (HSI), a framework in which a single frozen LLM M operates across three hierarchical

本研究对新兴的内存键值存储进行了全面的性能与可行性评估，突出了性能、兼容性与长期可行性（包括项目成熟度、社区支持与持续开发）之间的权衡。This study presents a comprehensive performance and viability assessment of the emerging in-memory key-value stores and highlights trade-offs between performance, compatibility, and long-term viability, including project maturity, community support, and sustained development.

尽管近期 LLM 在单项专利起草任务上取得不错的成绩，却未触及现实专利起草的核心挑战——从早期发明材料直接生成完整且法律上一致的专利申请。已有工作多假设输入处于后期、高度结构化或已具法律风格，但实际工作流程始于发明人编写的非正式、去法律化的 disclosure。为弥合差距，我们提出 Dis2Pat，一个反映真实[专利撰写场景]的 disclosure-to-patent 数据集。While recent large language models (LLMs) have achieved promising results on individual patent drafting tasks, they fundamentally fail to investigate the core challenge of real-world patent drafting: generating a complete and legally coherent patent application directly from early-stage invention materials. Prior work predominantly assumes later-stage, highly structured, or already legalistic inputs. However, real patenting workflows begin with informal, de-legalized disclosures authored by inventors. To bridge the gap, we introduce Dis2Pat, a disclosure-to-patent dataset that reflects realist

开放式语言模型基准通常继承一种评判方式：人类偏好面板、另一个模型，或脆弱的精确匹配答案。我们提出 FlavourBench，一个自动化基准，其中版本化的烹饪系统提供密集、可执行的真值。每个任务给出八种食材，要求选择三食材组合；在模型执行前，Epicure 对全部 56 种可能组合打分。我们在相同的核心任务集上评估了 27 个 frontier 端点，覆盖替换、配对与受限组合共 534 道任务。每个被排名的模型在每个面板上恰好有 89 个有效回答，且 famOpen-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and fam

以人为本的智能正在基础模型时代演进，越来越强调规模、可迁移性与通用建模。然而，它尚未与基础模型充分融合以取得可比拟的进展。更重要的是，这一广阔领域的近期进展仍分散在不同任务、模态与研究社区之间，其内在的概念与方法学联系尚不清晰。为弥合这些分歧并重新思考基础模型时代的以人为本智能，我们提出一套全谱系的人类上下文分类法Human-centric intelligence is evolving in the foundation-model era, with growing emphasis on scale, transferability, and general-purpose modeling. Yet it has not fully integrated with foundation models to achieve the comparable progress seen in them. More importantly, recent advances across this broad landscape remain fragmented across tasks, modalities, and research communities, leaving their intrinsic conceptual and methodological connections unclear. To bridge these divides and rethink human-centric intelligence in the foundation-model era, we introduce a full-spectrum human context taxonom

随着端侧 LLM Agent 演变为个人副驾驶，移动操作系统已成为该范式的关键试验场，亟需严格的能力评测。然而现有基准可分为两类，各自存在关键盲区：以 GUI 为中心的基准仅测试表层屏幕操作，忽略了后台工具调用与长程规划；而静态 function-calling 基准依赖离线 API 匹配，与真实运行时约束脱节。为弥合这一差距，我们提出 MobilePA-Bench，一个交互式、有状态、以工具为中心的基准。As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark

我们提出一种通过自适应验证任务选择来实现 LLM harness 高效优化的新方法。Harness 优化基于验证性能迭代改写 harness 代码，无需更新底层模型权重即可获得显著性能提升。然而现有方法在每次迭代中对固定的验证集进行完整评估，即便某些任务随 harness 演进区分度下降，仍产生高昂的评测成本。我们提出 Task-CoEvolve，通过应对两项挑战使验证任务与 harness 协同演进：seWe present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose Task-CoEvolve, which co-evolves the validation tasks with the harness by addressing two challenges: se

大语言模型日益被期望执行复杂工作流，其成功依赖于维持相互依赖的约束并产出满足严格端到端验证的产物。然而成功的执行经验通常在单次运行后就丢失了，迫使后续模型从头重新发现策略和失败模式。我们研究能否通过 EvoMap 将此类经验外部化并复用，其中验证器确认的执行轨迹被整合为结构化的 Gene。为评估该设定，我们引入长工作流基准Large language models are increasingly expected to execute complex workflows whose success depends on maintaining interdependent constraints and producing artifacts that satisfy strict end-to-end verification. Yet successful execution experience is typically lost after a single run, forcing subsequent models to rediscover strategies and failure modes from scratch. We study whether such experience can instead be externalized and reused through EvoMap, where verifier-confirmed execution trajectories are consolidated into structured Gene. To evaluate this setting, we introduce the Long-Workflow B

本文提出了 SeedRG，一个用于缓解 knowledge leakage 并应对 benchmark aging 问题的半合成 benchmark 生成 pipeline。SeedRG is introduced, a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging.

本文聚焦 LLM 驱动的 kernel generation 领域，给出现有方法的结构化综述，涵盖 LLM-based 方法与 agentic optimization workflow，并系统梳理了支撑该领域学习与评测的数据集与 benchmark。This survey addresses the gap in LLM-driven kernel generation by providing a structured overview of existing approaches, spanning LLM-based approaches and agentic optimization workflows, and systematically organizing the datasets and benchmarks that underpin learning and evaluation in this domain.

本文提出一个无偏的多进程 evaluation 框架，能够有效分散 client 端负载，从而在每秒数千次 query 以上的生产规模下实现对 LLM 的精确、可复现 profiling。This work proposes an unbiased, multi-process evaluation framework that effectively distributes client-side load, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.

本文提出 RAGCap-Bench，一个面向能力的 benchmark，用于对 agentic RAG workflow 中的中间任务进行细粒度评测，并构建了典型 LLM 错误的分类体系以设计针对性评测问题。This work proposes RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows, and constructs a taxonomy of typical LLM errors to design targeted evaluation questions.