论文

SwiftCache 是一个协同推理系统，使异构模型可在同一服务器内共享未充分利用的 GPU 内存与 NVLink 带宽，支持跨模型通过 NVLink 共享 KV cache，避免使用慢速 PCIe 传输。SwiftCache is a collaborative inference system that enables heterogeneous models to share underutilized GPU memory and NVLink bandwidth within a server, allowing cross-model KV cache sharing over NVLink and avoiding slow PCIe transfers.

提出 OScaR（Omni-Scaled Canalized Rotation），一个面向 X-LLMs 的精确且轻量的 KV cache 压缩框架，作为一个鲁棒、低复杂度、通用的框架确立了新的 Pareto 前沿。OScaR (Omni-Scaled Canalized Rotation), an accurate and lightweight KV cache compression framework for X-LLMs, is proposed, establishing it as a robust, low-complexity, and universal framework that defines a new Pareto front.

WAIT (Waiting for Accumulated Inference Threshold) 是一种基于阈值的准入规则，适用于已知输出长度；Nested WAIT 通过调控请求在 decode 阶段各分段间的推进方式，将该规则扩展到未知输出长度场景。WAIT (Waiting for Accumulated Inference Threshold), a threshold-based admission rule for known output lengths, and Nested WAIT, which extends the rule to unknown output lengths by regulating how requests advance across decode-stage segments are designed.

本文提出 AIConfigurator，一个统一的性能建模系统，能够在不依赖 GPU profiling 的前提下进行快速、与框架无关的推理配置搜索；并提供一个抽象层，自动为目标后端解析最优启动参数，无缝集成到生产级编排系统中。AIConfigurator is presented, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling, and an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems.

研究发现 RLMs 能够成功处理超出模型上下文窗口长达两个数量级的输入，即便在较短 prompt 下，其质量也显著优于原生前沿 LLM 以及常见的长上下文与编程脚手架。It is found that RLMs can successfully process inputs up to two orders of magnitude beyond model context windows and, even for shorter prompts, dramatically outperform the quality of vanilla frontier LLMs and common long-context and coding scaffolds.

本文提出了一种实证方法，用于研究基于 GPU 的 LLM 推理服务系统中的 software aging 问题，并提供了一个可复现的框架，开辟了 software aging 与 software rejuvenation 与 LLM serving 交叉方向的研究。This paper proposes an empirical methodology to study software aging in GPU-based LLM serving systems and provides a reproducible framework that opens a research direction at the intersection of the software aging and rejuvenation and LLM serving communities.

本文提出首个将计算与 GPU 显存约束显式纳入 LLM 推理分析的排队论框架，并推导了严格的稳定性与不稳定性条件，用以判定 LLM 推理服务能否在持续到达的请求下避免队列无界增长。This paper introduces the first queueing-theoretic framework that explicitly incorporates both computation and GPU memory constraints into the analysis of LLM inference, and derives rigorous stability and instability conditions that determine whether an LLM inference service can sustain incoming demand without unbounded queue growth.

介绍 Albireo，一种并行推理系统：通过调度与 I/O 与计算及序列并行采样的重叠来压缩不可扩展部分，从而提高可达到的 TP 度 t，且不改变模型架构。Albireo is presented, a parallel inference system that raises the attainable TP degree t by shrinking the non-scalable portion via overlap of scheduling and I/O with compute and sequence-parallel sampling, without changing model architectures.

本工作开发了一个 SOTA 的 paged attention kernel，完全基于领域特定即时编译语言 Triton 构建，在 NVIDIA 与 AMD GPU 上均达到 SOTA 性能。This work develops a state-of-the-art paged attention kernel that builds exclusively on the domain-specific just-in-time compiled language Triton to achieve state-of-the-art performance on both NVIDIA and AMD GPUs.

本工作提出一种新颖的混合精度 PTQ 策略，直接最小化整个模型的全局误差传播，而非孤立地处理逐层误差，并开发了一种新颖的联合优化方法，在统一搜索空间中同时学习结构化剪枝决策与混合精度量化策略。This work proposes a novel mixed-precision PTQ strategy that directly minimizes global error propagation across the entire model, rather than isolating layer-wise errors, and develops a novel joint optimization approach that simultaneously learns structural pruning decisions and mixed-precision quantization policies within a unified search space.

本工作借鉴线性模型的 state space model（SSM）视角，提出三项核心方法改进并组合形成更具表达力的递推结构：源自 SSM 离散化的递推式、用于更丰富状态追踪的复数值状态更新规则，以及在不增加 decode 延迟前提下提升模型性能的多输入多输出（MIMO）建模。This work introduces three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models that combine to form a more expressive recurrence derived from SSM discretization, a complex-valued state update rule that enables richer state tracking, and a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency.

Nemotron 3 Super 是 Nemotron 3 系列中首个采用 NVFP4 进行预训练的模型，借助 LatentMoE（一种同时优化精度 per FLOP 与精度 per parameter 的新型 Mixture-of-Experts 架构），并集成 MTP 层以通过原生 speculative decoding 加速推理。Nemotron 3 Super is the first model in the Nemotron 3 family to be pre-trained in NVFP4, leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and include MTP layers for inference acceleration through native speculative decoding.

Tangram 是一种 serving 框架，将先前系统动态处理的内容静态解析，可作为现有 non-uniform 压缩方法的即插即用底座，在匹配其精度的同时，端到端吞吐量较 full-KV 基线最高提升 2.6×。Tangram is a serving framework that statically resolves what prior systems handle dynamically, and serves as a drop-in substrate for existing non-uniform compression methods, matching their accuracy while improving end-to-end throughput by up to $2.6\times over the full-KV baseline.

本文为 LLM serving 中的 SD（投机解码）提出了一种简单且可解释的 latency 模型，能够准确刻画实际观测到的 latency，解释为何加速比常随服务器负载上升而下降，并系统刻画了 draft length、acceptance rate 以及 verifier 与 drafter 规模在不同 serving 条件下对 latency 的影响。A simple and interpretable latency model for SD in LLM serving is developed that accurately describes observed latency, explains why speedups often diminish as server load increases, and characterizes how draft length, acceptance rate, and verifier-drafter size shape latency across serving conditions are characterized.

Tutti 是一种高效的 SSD-backed KV caching 方案，将 CPU 从 HBM 与 SSD 之间的关键数据与 I/O 控制路径中彻底移除，在提供近乎无限容量的同时，实现了与 DRAM-backed LMCache 几乎相当的 inference 性能。Tutti is an efficient SSD-backed KV caching solution that eliminates CPU intervention from the critical data and I/O control paths between HBM and SSDs, and achieves nearly the same inference performance as DRAM-backed LMCache, while providing almost infinite capacity.

本文提出了一种简单的 flow-control 框架，通过控制 prompt 加入 LLM 活跃集合的速率，实现更高的 token 与 request 吞吐量、更低的平均与尾部 latency，以及更稳定的 KV cache 利用率。A simple flow-control framework is proposed that controls the rate at which prompts join the active set in large language models and achieves higher token and request throughput, lower average and tail latency, and more stable KV cache utilization.

本文提出 CATS，一种 self-speculative decoding 框架，在内存受限设备上结合 memory budget 与 parameter offloading pattern 进行级联式 verify 与 correction，在设备峰值显存与单独运行 target model 相当的前提下，最大化 token acceptance rate 与端到端加速比。CATS, a self-speculative decoding framework that conducts cascaded verification and correction based on the memory budget and parameter offloading patterns on memory-limited devices, is proposed, which maximizes token acceptance rate and end-to-end speedup while keeping the peak memory footprint on the device equal to that of the target model alone.

本文提出 DualPath，一种 inference 系统，通过引入 dual-path KV-Cache loading 打破瓶颈，并实现一条新的 storage-to-decode 路径：KV-Cache 先加载到 decode engine，再通过 compute network 上的 RDMA 高效转发至 prefill engine。DualPath is presented, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading and enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network.

本文提出 TTKV，一种 KV cache 管理框架，将人类记忆系统映射到具备异构容量与精度的 KV cache 上，在 128K 上下文任务上将跨层流量降低 5.94 倍。TTKV is proposed, a KV cache management framework that maps the human memory system onto the KV cache with heterogeneous capacity and precision, and reduces cross-tier traffic by 5.94x on 128K-context tasks.

本文提出 KV Policy (KVP)，一种仅基于 key 与 value 向量、在预计算生成轨迹上训练的轻量级 per-head RL Agent 框架，证明学习预测未来 token 效用是自适应 KV cache 管理中强大且可扩展的范式。KV Policy (KVP), a framework of lightweight per-head RL agents trained on pre-computed generation traces using only key and value vectors, is introduced, demonstrating that learning to predict future token utility is a powerful and scalable paradigm for adaptive KV cache management.

TritonForge 是一个面向自动化 Triton kernel 优化的 profiling 引导框架，融合 kernel 分析、运行时 profiling 与迭代式代码转换以简化优化流程，并为自动化 GPU 性能优化领域的未来研究奠定基础。TritonForge, a profiling-guided framework for automated Triton kernel optimization that integrates kernel analysis, runtime profiling, and iterative code transformation to streamline the optimization process and provides a foundation for future research in automated GPU performance optimization.

本文探讨容器化、微服务、动态调度等 Cloud Native 技术如何从根本上提升 LLM 推理服务，并展示 Cloud Native 系统在高需求场景下实现更高效资源分配、降低延迟与提升吞吐的能力。This article explores how Cloud Native technologies, such as containerization, microservices, and dynamic scheduling, can fundamentally improve LLM inference serving and demonstrates how a Cloud Native system enables more efficient resource allocation, reduces latency, and enhances throughput in high-demand scenarios.

NetKV，一种使用该 oracle 信息的 O(|D|) 每请求贪心策略，其层级排序被证明对过时遥测数据具有鲁棒性；并证明随着上下文长度增长，忽略网络项会使仅缓存感知的调度任意次优。NetKV, the O(|D|) per-request greedy that consumes this oracle, has tier rankings that are provably robust to stale telemetry, and it is proved that ignoring the network term renders cache-aware-only scheduling arbitrarily suboptimal as context length grows.

本工作提出 LMCACHE，首个也是目前最高效的开源 KV 缓存方案，可将现代 LLM 引擎生成的 KV 缓存从 GPU 显存中提取并存储，并跨引擎和查询共享。This work presents LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines out of the GPU memory and shares them across engines and queries.

简化了 MoE 路由算法，设计出通信与计算成本更低的直观改进模型，并首次证明大型稀疏模型可以使用更低精度格式进行训练This work simplifies the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs and shows large sparse models may be trained, for the first time, with lower precision formats.

Pointer sentinel-LSTM 模型在 Penn Treebank 上以远少于标准 softmax LSTM 的参数量达到 SOTA 语言建模性能，并开源了 WikiText 语料库The pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank while using far fewer parameters than a standard softmax LSTM and the freely available WikiText corpus is introduced.

论文证明，使用标注数据进行微调，并允许模型查询外部知识源，能够在安全性和事实性这两个关键挑战上带来显著提升。It is demonstrated that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.

一个仅依赖字符级输入的简单神经语言模型，仅从字符即可编码语义和正字法信息，表明在许多语言中，字符输入足以完成语言建模。A simple neural language model that relies only on character-level inputs that is able to encode, from characters only, both semantic and orthographic information and suggests that on many languages, character inputs are sufficient for language modeling.