论文

本文提出 M$^3$Exam，一个以查询为中心、基于真实用户-Agent 交互构建的多模态对话记忆基准，涵盖跨模态定位与隐式信息推断等多维度评估。M$^3$Exam is introduced, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference.

本文命名并研究这两条研究脉络之间的模式：其递归单元是配备文件系统工具、代码执行与规划的完整 Agent harness，而非无工具的模型调用，并给出针对长上下文推理的受控评估。This work names and studies the pattern between these two lines of work, where the recursive unit is a full agent harness with filesystem tools, code execution, and planning rather than a model call with no tools, and provides a controlled evaluation on long-context reasoning.

本文提出 ForeSci，一个时间受控的基准，用于评估 LLM Agent 是否能从历史证据中做出前瞻性研究判断，并在四种骨干模型上评测原生 LLM、Hybrid RAG 以及三种 research-agent 适配方案。This work introduces ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence, and evaluates native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones.

本文提出 MMLongEmbed，首个面向长上下文场景评估 MEM 的综合基准，并发现现有架构严重依赖浅层特征匹配，难以捕捉深层语义与结构依赖。This work introduces MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios, and finds that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies.

一个两级框架将手动 harness 工程转变为自动化 harness 工程，并更进一步——将"自动化本身的设计"也自动化。A two-level framework shifts manual harness engineering into automated harness engineering, and takes one step further --automating the design of the automation itself.

提出 Agentic Harness Engineering（AHE），一个通过三个相互匹配的 observability 支柱应对 harness 工程挑战的闭环，将每一次编辑转化为可证伪的契约，使 harness 演进能够自主进行而不退化为试错。Agentic Harness Engineering (AHE) is introduced, a closed loop that addresses harness engineering challenges through three matched observability pillars that turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error.

本文提出 Natural-Language Agent Harnesses，即可编辑的、描述运行级 harness 策略的文档，以及 Intelligent Harness Runtime（IHR），一个将上述文档解释为 agent 调用、交接、状态更新、验证门控与 artifact 契约的共享运行时。This paper introduces Natural-Language Agent Harnesses, editable documents that describe run-level harness policy, and Intelligent Harness Runtime (IHR), a shared runtime that interprets these documents into agent calls, handoffs, state updates, validation gates, and artifact contracts.

文中指出，安全的 LLM Agent 需要显式的信任边界、原则化的权限控制、具备溯源能力的 state 管理，以及与真实运行场景对齐的评估实践；现有 benchmark 仍未能充分覆盖长程、具状态、对部署敏感的风险。It is argued that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings, as well as existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks.

在 DeNovoSWE 上对 Qwen3-30B-A3B 进行微调可显著提升长程 SWE 性能，在具有挑战性的 BeyondSWE-Doc2Repo benchmark 上将其得分从 5.8% 提升至 47.2%。Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

本综述系统梳理了超越基础 Skill 创建的 Skill 演化与评估图景，将其归纳为四种范式：执行反馈、轨迹蒸馏、压缩与强化学习，并指出了构建可泛化、高效且可验证安全的 Skill 生态的开放方向。This survey systematically examines the landscape of skill evolution and evaluation beyond foundational skill creation into four distinct paradigms, spanning execution feedback, trajectory distillation, compression, and reinforcement learning, and identifies open directions for building skill ecosystems that are generalizable, efficient, and verifiably safe.

文章认为基于实体的分解能形成对原始信息更精炼的表示，并有助于降低索引与生成过程中的噪声；在端到端 QA 评测中，VectorRAG 表现优于标准 GraphRAG，且接近当前 SOTA 图方法的效果。It is argued that entity-based decomposition yields a more distilled representation of original information, and additionally serves to reduce noise in the indexing, and generation process, and on end to end QA evaluation VectorRAG performs better than standard GraphRAG and almost as good as current SOTA graph-based solutions.

介绍 EvoArena 基准套件，将环境变化建模为跨终端、软件与社会领域的渐进式更新序列；并提出 EvoMem，一种基于 patch 的记忆范式，将记忆演化记录为结构化的更新历史，使 Agent 能通过记忆的变化推理环境的演化。EvoArena is introduced, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains, and EvoMem is proposed, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.

本研究对新兴的内存键值存储进行了全面的性能与可行性评估，突出了性能、兼容性与长期可行性（包括项目成熟度、社区支持与持续开发）之间的权衡。This study presents a comprehensive performance and viability assessment of the emerging in-memory key-value stores and highlights trade-offs between performance, compatibility, and long-term viability, including project maturity, community support, and sustained development.

本文提出了 SeedRG，一个用于缓解 knowledge leakage 并应对 benchmark aging 问题的半合成 benchmark 生成 pipeline。SeedRG is introduced, a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging.

本文聚焦 LLM 驱动的 kernel generation 领域，给出现有方法的结构化综述，涵盖 LLM-based 方法与 agentic optimization workflow，并系统梳理了支撑该领域学习与评测的数据集与 benchmark。This survey addresses the gap in LLM-driven kernel generation by providing a structured overview of existing approaches, spanning LLM-based approaches and agentic optimization workflows, and systematically organizing the datasets and benchmarks that underpin learning and evaluation in this domain.

本文提出一个无偏的多进程 evaluation 框架，能够有效分散 client 端负载，从而在每秒数千次 query 以上的生产规模下实现对 LLM 的精确、可复现 profiling。This work proposes an unbiased, multi-process evaluation framework that effectively distributes client-side load, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.

本文通过大规模超参数搜索，系统地重新评估了 Vanilla LoRA 以及九个代表性 LoRA 变体，发现不同 LoRA 方法偏好的学习率区间各异，并将最优学习率区间的差异归因于最大 Hessian 特征值的变化，与经典学习理论相吻合。This work systematically re-evaluate nine representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches, finding that different LoRA methods favor distinct learning rate ranges and attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.

本工作提出 AlphaEval，一个基于真实生产环境的基准，包含来自七家在其核心业务中部署 AI Agent 的公司的 94 个任务，覆盖六个 O*NET (Occupational Information Network) 领域；并贡献了一套从需求到基准的构建框架，将从需求到评估的完整流程标准化。This work presents AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains, and contributes a requirement-to-benchmark construction framework that standardizes the entire pipeline from requirement to evaluation.

认为（该早期版本的）GPT-4 属于新一代具备更通用智能的 LLM（与 ChatGPT、谷歌 PaLM 等并列），并讨论了这些模型不断增强的能力及其影响。It is argued that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models, and the rising capabilities and implications of these models are discussed.

发现对 GPT 语言模型进行重复采样，是为困难 prompt 生成可用解的一种出乎意料的有效策略，并讨论了部署强大代码生成技术在安全性、安全保障与经济等方面的潜在更广泛影响。It is found that repeated sampling from the GPT language model is a surprisingly effective strategy for producing working solutions to difficult prompts, and the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics are discussed.

本文对 LLM 的评估方法进行了全面综述，围绕三个关键维度展开：评估什么、在何处评估、如何评估，并为 LLM 评估领域的研究者提供了宝贵洞见。This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate, and offers invaluable insights to researchers in the realm of LLMs evaluation.

分析表明，借助 LLM，美国约 15% 的工作任务可在保持同等质量的前提下显著提速完成，意味着 LLM 驱动的软件将对底层模型经济影响的规模化产生实质性作用。The analysis suggests that, with access to an LLM, about 15% of all worker tasks in the US could be completed significantly faster at the same level of quality, implying that LLM-powered software will have a substantial effect on scaling the economic impacts of the underlying models.

研究发现，ChatGPT 最成功地被用作数学助手，用于查询事实、充当数学搜索引擎和知识库接口；GPT-4 还可被用于本科水平的数学问题，但在研究生难度的题目上表现不佳。It is found that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a Mathematical search engine and knowledge base interface, and GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty.

EvalPlus——一个用于严格基准测试 LLM 生成代码功能正确性的代码合成评估框架，通过 LLM 与基于 mutation 的策略驱动的自动测试输入生成器，为给定评估数据集补充大量新生成的测试用例。EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code and augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM and mutation-based strategies.