论文

本文对 softmax 注意力与四种近期的循环线性注意力架构（DeltaNet、Gated DeltaNet、Kimi Delta Attention 与 Gated DeltaNet-2）进行对比研究，明确阐述它们在表达能力、记忆衰减、擦写控制、训练吞吐量与实现复杂度上的差异。A comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 is presented, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity.

研究表明模型具备相关子群体知识，但难以稳定传递到聚合估计中，这一差距使统计自一致性成为评估 LLM 的尚未饱和、无需参考的准则。It is suggested that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates, and this gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

提出 BloombergGPT，一个 500 亿参数的语言模型，在广泛的金融数据上训练而成，并基于 Bloomberg 丰富的数据源构建了包含 3630 亿 token 的数据集，可能是迄今最大的领域专用数据集。This work presents BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data, and constructs a 363 billion token dataset based on Bloomberg's extensive data sources, perhaps the largest domain-specific dataset yet.

综合评估表明，DeepSeek-V3 优于其他开源模型，并达到与领先闭源模型相当的性能。Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models.

这些发现表明，即便长上下文评估正从简单检索转向复杂推理，对相关证据的准确 grounding 仍是一项不可或缺且仍有大幅提升空间的能力。These findings suggest that, even as long-context evaluation shifts from simple retrieval toward complex reasoning, accurate grounding in relevant evidence remains an indispensable capability with substantial room for improvement.

本文描述了 TensorFlow 接口及 Google 构建的该接口实现，已被用于开展研究，并在计算机科学及其他十余个领域中将机器学习系统部署至生产环境。The TensorFlow interface and an implementation of that interface that is built at Google are described, which has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields.

本文推出参数规模从 7B 到 65B 的基础语言模型集合 LLaMA，并证明完全使用公开数据集即可训练出 SOTA 模型，无需依赖专有或不可获取的数据。LLaMA, a collection of foundation language models ranging from 7B to 65B parameters, is introduced and it is shown that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets.

通过从概率分布的动态 nucleus 中采样文本，可在有效截断不可靠分布尾部的同时保持多样性，使生成文本更接近人类文本质量，在不牺牲流畅性与连贯性的前提下提升多样性。By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

本文指出基于 Transformer 的次二次时间模型的关键缺陷在于无法执行基于内容的推理，并将选择性 SSM 集成到不包含注意力乃至 MLP 块的简化端到端神经网络架构（Mamba）中。This work identifies that a key weakness of subquadratic-time models based on Transformer architecture is their inability to perform content-based reasoning, and integrates selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba).

该设计将 hypernetwork 的注入能力与目标模型的通用能力解耦，首次实现了对 hypernetwork 架构 scaling law 的严格研究，并提供了首个基于实证的 scaling law，用以指导大语言模型中面向事实推理的 hypernetwork 设计。The design decouples the hypernetwork's injection capacity from the target model's general capability, enabling, for the first time, a rigorous study of scaling laws for hypernetwork architectures, and provides the first empirically grounded scaling laws to guide hypernetworks for factual reasoning in large language models.

GPT-3 在多个 NLP 数据集上取得了强劲表现，包括翻译、问答和完形填空任务，以及若干需要即时推理或领域适应的任务，例如乱序词重组、在句子中使用新词、或执行三位数算术运算。GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.

本文提出了一种新的解码策略——self-consistency，用于替代思维链 prompt 中使用的朴素贪心解码：首先采样一组多样化的推理路径，而非仅取贪心路径，然后通过对采样路径进行边缘化来选择最一致的答案。This paper proposes a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting that first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths.

本文进行了迄今为止对 Code LLMs 最全面的评估，结果显示 StarCoderBase 在支持多编程语言的开放 Code LLMs 中表现最优，并且能够匹敌或超越 OpenAI code-cushman-001 模型。This work performs the most comprehensive evaluation of Code LLMs to date and shows that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model.

PaLM 2 是一个新的 SOTA 语言模型，相比其前身 PaLM 具有更强的多语言和推理能力，并具备更高的计算效率，能够在不增加额外开销或影响其他能力的前提下在推理时控制输出毒性。PaLM 2 is a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM and enables inference-time control over toxicity without additional overhead or impact on other capabilities.

本文在使用 SOTA CNN 的情况下，实现了 16–24 倍的网络压缩，仅带来 1% 的分类准确率损失，并发现针对存储开销最大的全连接层进行压缩时，向量量化方法相比现有矩阵分解方法具有明显优势。This paper is able to achieve 16-24 times compression of the network with only 1% loss of classification accuracy using the state-of-the-art CNN, and finds in terms of compressing the most storage demanding dense connected layers, vector quantization methods have a clear gain over existing matrix factorization methods.

本文介绍 Gemma，一族基于 Gemini 模型所使用的研究与技术构建的轻量级 SOTA 开源模型，并全面评估模型的安全性与责任性，同时详细描述模型开发过程。This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models, and presents comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development.

本文提出一个半符号化框架，将用于 OCR 后校正的 word-spotting 技术与知识图谱表示相结合，使 agent 可通过合成查询访问信息，从而对误读和幻觉具有鲁棒性。This paper introduces a semi-symbolic framework that integrates word-spotting techniques for post-OCR correction with a knowledge graph representation that enables the agent to access information through synthesized queries that are robust to misinterpretation and hallucination.

基于 ONNX Runtime 的原生 C++ 推理引擎击败了所有 baseline，置信区间甚至不重叠；batch 策略的影响比语言或运行时的选择更关键，超出作者预期。A native C++ inference engine on ONNX Runtime that beat every baseline, confidence intervals didn't even overlap, and batching strategy mattered more than either the language or the runtime choice, more than the authors expected.

提出 CADENCE，一个统一框架，对每种叠加式失败模式给出针对性修复，证明通过原则化的蒸馏即可在不依赖数据中心级硬件的条件下获得强推理质量。CADENCE, a unified framework with a targeted fix for each compounding failure modes, is presented, showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.

对有损验证方法所诱导的分布进行原则化分析，揭示一项根本缺陷：由于分布失真，其性能可能相对于真实截断采样基线显著退化。This work presents a principled analysis of the distributions induced by lossy verification methods, and identifies a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion.

提出Criterion-Distilled Policy Optimization (CriPO)，通过on-policy自蒸馏增强基于rubric的RL，并采用反事实自教师定位负优势rollout中与准则相关的token，将其token级优势翻转为正值，保留本将被抑制的有用模式。Criterion-Distilled Policy Optimization (CriPO) is proposed, which enhances rubric-based RL via on-policy self-distillation and employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed.

提出SA，一种Stable Advantage Fusion框架，避免熵崩塌，在所有六个模型-领域设定上持续优于固定系数的GRPO+OPD融合，聚合得分提升0.70%，训练更稳定。SA is proposed, a Stable Advantage Fusion framework that avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.70% across all six model-domain settings while achieving more stable training.

本文提出 DAPD，一种具有两级锚定的统一框架，可显著缓解特权错觉，在 Qwen3-4B 上以平均 +2.00 分优于 OPSD。DAPD is proposed, a unified framework with two levels of anchoring that significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks.

GradCuit（梯度穿越电路）在所选 Transformer 层、提示隐藏表示与生成续写之间插入可优化的潜变量，开启了鲁棒且可解释的测试时缩放新维度，使 LLM 调整其推理方式，而不仅仅是重新生成、采样或重排输出。GradCuit (gradient through circuit), which inserts optimizable latent states at a selected Transformer layer between the hidden representations of the prompt and the generated continuation, opens a new axis of robust and interpretable test-time scaling, where LLMs adapt how they reason rather than merely regenerate, sample, or rerank outputs.

本文提出 GPTQ-2D，以三次时间复杂度生成相同的取整矩阵，并研究该任务的双侧版本——固定非奇异基矩阵同时作用于残差的左右两侧This work presents GPTQ-2D, which produces the identical rounded matrix in cubic time, and studies the two-sided version of this task, in which fixed nonsingular basis matrices act on both the left and the right of the residual.

简化了 MoE 路由算法，设计出通信与计算成本更低的直观改进模型，并首次证明大型稀疏模型可以使用更低精度格式进行训练This work simplifies the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs and shows large sparse models may be trained, for the first time, with lower precision formats.

Pointer sentinel-LSTM 模型在 Penn Treebank 上以远少于标准 softmax LSTM 的参数量达到 SOTA 语言建模性能，并开源了 WikiText 语料库The pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank while using far fewer parameters than a standard softmax LSTM and the freely available WikiText corpus is introduced.

DASH (Drift Aware advantage SHaping) 根据每个推理片段是朝向还是偏离正确性来分配片段级 credit，在 overthinking 普遍的场景下取得最高准确率，同时减少 overthinking 行为，比基线实现更高效的自纠错。DASH (Drift Aware advantage SHaping), which assigns segment-level credit based on whether each reasoning segment leads toward or away from correctness, achieves the highest accuracy where overthinking is prevalent while reducing overthinking behaviors and achieving more productive self-correction than baselines.

提出 ARCHead，一种紧凑型 LM-head 压缩器，结合量化低秩核心、组级 INT4 残差以及基于激活导出度量拟合的低秩校正，在 block 量化器无法触及的大输出投影上实现互补压缩。ARCHead is presented, a packed LM-head compressor that combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted in an activation-derived metric to complements block quantizers by compressing the large output projection they can leave untouched.

提出 RestoreKV，在相同 KV 总预算下以可学习的恢复机制补充基于选择的 KV 压缩方案，显著降低压缩带来的性能损失。RestoreKV is introduced, which complements this selection-based formulation with learned restoration with learned restoration under the same total KV budget, and substantially reduces compression-induced degradation.

在三种大语言模型上的实验表明，思维链提示能够在一系列算术、常识和符号推理任务上提升性能。Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks.

本工作提出两种参数削减技术以降低 BERT 的内存占用并提升训练速度，并采用一种聚焦于建模句子间连贯性的自监督损失。This work presents two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT, and uses a self-supervised loss that focuses on modeling inter-sentence coherence.

研究表明，利用变分方法最新进展的深度生成模型与近似贝叶斯推断能够带来显著提升，使生成式方法在半监督学习上极具竞争力。It is shown that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.

这项系统性研究在数十项语言理解任务上比较了预训练目标、架构、无标注数据集、迁移方法及其他因素，并在涵盖摘要、问答、文本分类等的许多基准上取得了 SOTA 结果。This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.

本文提出了 mixup，一种通过对样本对及其标签的凸组合来训练神经网络的简单学习原则，提升了 SOTA 神经网络架构的泛化能力。This work proposes mixup, a simple learning principle that trains a neural network on convex combinations of pairs of examples and their labels, which improves the generalization of state-of-the-art neural network architectures.