论文

提出 Simplified Sparse Attention——一种更简洁的稀疏注意力方案，无需任何架构改动，在检索增强生成中优于全注意力；并扩展为分层 gist-of-gist 变体，在高达 32× 的高压缩比下保持或提升精度，同时实现对数级解码复杂度。Simplified Sparse Attention is introduced, a simpler approach to sparse attention that requires no architectural changes that outperforms full attention in retrieval-augmented generation and extends to a hierarchical gist-of-gist variant that achieves log-linear decoding complexity while maintaining or improving accuracy at high compression ratios up to 32x.

提出 Erase-then-Delta Attention (EDA)，一种将"在哪里擦除"与"在哪里写入"解耦的内存更新规则；研究表明循环记忆模型不仅应决定写入什么，还应决定擦除哪些陈旧信息以及擦除的位置。Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write, is proposed, suggesting that recurrent memory models should decide not only what to write, but also what stale information to erase and where.

本文从前瞻视角重新审视 token 重要性，提出衡量压缩 token 对未来上下文影响的新指标 Forward Influence，以及融合信息论信号的熵感知 KV cache 压缩框架 InfoKV。This paper revisits token importance from a forward-looking perspective and introduces Forward Influence, a metric that measures how compressed tokens affect future contexts, and proposes InfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals.

提出 SAC，这是首个针对稀疏注意力模型优化的高效解耦 KV Cache 系统，利用 CXL（Compute Express Link）的低延迟、cache-line 粒度 load/store 语义，将基于 CXL 的解耦确立为新兴稀疏注意力模型的优越基础设施。SAC is proposed, the first efficient disaggregated KV cache system optimized for sparse attention models, which leverages the low-latency, cache-line granularity load/store semantics of Compute Express Link (CXL), establishing CXL-based disaggregation as the superior infrastructure for emerging sparse attention models.

本文通过收益–成本视角研究 PLT 循环次数选择：额外循环可精炼表示，但 CLP 也会在每次循环边界引入位置错配，由此解释 PLT 在两次循环时趋于饱和的现象，并为循环次数选择提供诊断依据。This study studies PLT loop-count selection through a gain--cost view: an extra loop may refine representations, but CLP also introduces a positional mismatch at each loop boundary, explaining PLT's saturation at two loops and providing diagnostics for loop-count selection.

提出 Sparse Delta Memory，一种通过稀疏寻址方案将门控线性 RNN 隐状态容量扩展数个数量级的架构，在上下文学习与长上下文检索任务上显著提升性能。Sparse Delta Memory is introduced, an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme and significantly improves performance on in-context learning and long-context retrieval tasks.

本文对 softmax 注意力与四种近期的循环线性注意力架构（DeltaNet、Gated DeltaNet、Kimi Delta Attention 与 Gated DeltaNet-2）进行对比研究，明确阐述它们在表达能力、记忆衰减、擦写控制、训练吞吐量与实现复杂度上的差异。A comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 is presented, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity.

Soofi S 30B-A3B 是一个主权开源的 MoE 混合 Mamba Transformer 德英基础模型，在作者的对比中超越了所有欧洲主权基线，包括活跃参数量远超自身的模型。Soofi S 30B-A3B is a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English that outperforms every European sovereign baseline in the authors' comparison, including ones far larger in active parameters.

GigaWorld-Policy-0.5 在保留未来视觉动力学训练收益的同时提升了机器人控制的推理效率，并引入 Mixture-of-Transformers 架构，将视觉动力学建模与动作生成分离到专门的专家模块中。GigaWorld-Policy-0.5 preserves the training benefits of future visual dynamics while improving inference efficiency for robot control, and introduces a Mixture-of-Transformers architecture that separates visual dynamics modeling and action generation into specialized experts.

研究表明模型具备相关子群体知识，但难以稳定传递到聚合估计中，这一差距使统计自一致性成为评估 LLM 的尚未饱和、无需参考的准则。It is suggested that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates, and this gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

本文旨在为研究者与从业者提供一份快速、全面的参考，通过对现有工作的广泛、信息密集型总结来汲取洞见，以推动 LLM 研究的发展。This review article is intended to provide a quick, comprehensive reference for the researchers and practitioners to draw insights from extensive, informative summaries of the existing works to advance the LLM research.

提出 BloombergGPT，一个 500 亿参数的语言模型，在广泛的金融数据上训练而成，并基于 Bloomberg 丰富的数据源构建了包含 3630 亿 token 的数据集，可能是迄今最大的领域专用数据集。This work presents BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data, and constructs a 363 billion token dataset based on Bloomberg's extensive data sources, perhaps the largest domain-specific dataset yet.

本文试图以简洁的方式描述受生成式 AI 影响的主要行业与模型，并给出近期主要生成式模型的分类体系。This work consists on an attempt to describe in a concise way the main models are sectors that are affected by generative AI and to provide a taxonomy of the main generative models published recently.

综合评估表明，DeepSeek-V3 优于其他开源模型，并达到与领先闭源模型相当的性能。Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models.

本文综述了一些最具代表性的 LLM，包括三大主流 LLM 家族（GPT、LLaMA、PaLM），讨论其特性、贡献与局限性，并概述了构建与增强 LLM 的相关技术。This paper reviews some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discusses their characteristics, contributions and limitations, and gives an overview of techniques developed to build, and augment LLMs.

这些发现表明，即便长上下文评估正从简单检索转向复杂推理，对相关证据的准确 grounding 仍是一项不可或缺且仍有大幅提升空间的能力。These findings suggest that, even as long-context evaluation shifts from simple retrieval toward complex reasoning, accurate grounding in relevant evidence remains an indispensable capability with substantial room for improvement.

本文描述了 TensorFlow 接口及 Google 构建的该接口实现，已被用于开展研究，并在计算机科学及其他十余个领域中将机器学习系统部署至生产环境。The TensorFlow interface and an implementation of that interface that is built at Google are described, which has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields.

本文推出参数规模从 7B 到 65B 的基础语言模型集合 LLaMA，并证明完全使用公开数据集即可训练出 SOTA 模型，无需依赖专有或不可获取的数据。LLaMA, a collection of foundation language models ranging from 7B to 65B parameters, is introduced and it is shown that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets.

通过从概率分布的动态 nucleus 中采样文本，可在有效截断不可靠分布尾部的同时保持多样性，使生成文本更接近人类文本质量，在不牺牲流畅性与连贯性的前提下提升多样性。By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

本文指出基于 Transformer 的次二次时间模型的关键缺陷在于无法执行基于内容的推理，并将选择性 SSM 集成到不包含注意力乃至 MLP 块的简化端到端神经网络架构（Mamba）中。This work identifies that a key weakness of subquadratic-time models based on Transformer architecture is their inability to perform content-based reasoning, and integrates selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba).

该设计将 hypernetwork 的注入能力与目标模型的通用能力解耦，首次实现了对 hypernetwork 架构 scaling law 的严格研究，并提供了首个基于实证的 scaling law，用以指导大语言模型中面向事实推理的 hypernetwork 设计。The design decouples the hypernetwork's injection capacity from the target model's general capability, enabling, for the first time, a rigorous study of scaling laws for hypernetwork architectures, and provides the first empirically grounded scaling laws to guide hypernetworks for factual reasoning in large language models.

GPT-3 在多个 NLP 数据集上取得了强劲表现，包括翻译、问答和完形填空任务，以及若干需要即时推理或领域适应的任务，例如乱序词重组、在句子中使用新词、或执行三位数算术运算。GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.

本文提出了一种新的解码策略——self-consistency，用于替代思维链 prompt 中使用的朴素贪心解码：首先采样一组多样化的推理路径，而非仅取贪心路径，然后通过对采样路径进行边缘化来选择最一致的答案。This paper proposes a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting that first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths.

本文进行了迄今为止对 Code LLMs 最全面的评估，结果显示 StarCoderBase 在支持多编程语言的开放 Code LLMs 中表现最优，并且能够匹敌或超越 OpenAI code-cushman-001 模型。This work performs the most comprehensive evaluation of Code LLMs to date and shows that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model.

本文旨在建立一套系统化的开放问题与应用成果清单，以便机器学习研究者更快地理解该领域的现状并开展有效工作。This paper aims to establish a systematic set of open problems and application successes so that ML researchers can comprehend the field's current state more quickly and become productive.

PaLM 2 是一个新的 SOTA 语言模型，相比其前身 PaLM 具有更强的多语言和推理能力，并具备更高的计算效率，能够在不增加额外开销或影响其他能力的前提下在推理时控制输出毒性。PaLM 2 is a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM and enables inference-time control over toxicity without additional overhead or impact on other capabilities.

本文在使用 SOTA CNN 的情况下，实现了 16–24 倍的网络压缩，仅带来 1% 的分类准确率损失，并发现针对存储开销最大的全连接层进行压缩时，向量量化方法相比现有矩阵分解方法具有明显优势。This paper is able to achieve 16-24 times compression of the network with only 1% loss of classification accuracy using the state-of-the-art CNN, and finds in terms of compressing the most storage demanding dense connected layers, vector quantization methods have a clear gain over existing matrix factorization methods.

本文介绍 Gemma，一族基于 Gemini 模型所使用的研究与技术构建的轻量级 SOTA 开源模型，并全面评估模型的安全性与责任性，同时详细描述模型开发过程。This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models, and presents comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development.

本文提出 IndicTalk，目前最大的多语言印度语码混合（code-mixed）会话语料库，包含超过 13,28,604 段事件驱动的多轮对话，覆盖 9 种印度语言的 18 种语言变体，将公开发布以支持低资源印度语多语言会话 AI 的开发与评估。IndicTalk is presented, one of the largest multilingual Indic code-mixed conversational corpora, comprising over 13,28,604 event-grounded multi-turn conversations across 18 language varieties covering 9 Indic languages and will be released to support the development and evaluation of multilingual conversational AI for underrepresented Indic languages.

基于 ONNX Runtime 的原生 C++ 推理引擎击败了所有 baseline，置信区间甚至不重叠；batch 策略的影响比语言或运行时的选择更关键，超出作者预期。A native C++ inference engine on ONNX Runtime that beat every baseline, confidence intervals didn't even overlap, and batching strategy mattered more than either the language or the runtime choice, more than the authors expected.

提出一个系统性的、以架构为中心的 LLM 记忆分类法，沿三个正交轴刻画记忆：表示、更新动态与持久性，有效桥接不同的架构范式。A systematic, architecture-centric taxonomy of memory in LLMs is presented, characterizes memory along three orthogonal axes: representation, update dynamics, and persistence, effectively bridging disparate architectural paradigms.

提出 CADENCE，一个统一框架，对每种叠加式失败模式给出针对性修复，证明通过原则化的蒸馏即可在不依赖数据中心级硬件的条件下获得强推理质量。CADENCE, a unified framework with a targeted fix for each compounding failure modes, is presented, showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.

对有损验证方法所诱导的分布进行原则化分析，揭示一项根本缺陷：由于分布失真，其性能可能相对于真实截断采样基线显著退化。This work presents a principled analysis of the distributions induced by lossy verification methods, and identifies a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion.

Meshy T2在几何保真度上达到SOTA，端到端图像到网格生成中位耗时6秒，比自回归基线快一个数量级以上。Meshy T2 achieves state-of-the-art geometric fidelity and completes end-to-end image-to-mesh generation within a median of 6 seconds, over an order of magnitude faster than autoregressive baselines.

提出Criterion-Distilled Policy Optimization (CriPO)，通过on-policy自蒸馏增强基于rubric的RL，并采用反事实自教师定位负优势rollout中与准则相关的token，将其token级优势翻转为正值，保留本将被抑制的有用模式。Criterion-Distilled Policy Optimization (CriPO) is proposed, which enhances rubric-based RL via on-policy self-distillation and employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed.