本文提出 Color Pass-Through,一个端到端可学习的框架,直接在采集图像上运行,将相机和显示器视为耦合系统进行联合处理,而非单独校准。This work proposes Color Pass-Through, an end-to-end learned framework that operates directly on captured images, to treat the camera and display as a coupled system rather than calibrating them in isolation.
论文
189 张论文卡片 · 多模态 · 方法
本文提出 SANA-Video 2.0,一个混合视频扩散 Transformer,在统一架构下实例化 5B 和 14B 两种规模,以显著降低的计算成本恢复了 softmax 级别的表达能力,解锁了可扩展的长时长、高分辨率视频生成。This work introduces SANA-Video 2.0, a hybrid video diffusion transformer instantiated at 5B and 14B scales under a unified architecture that recovers softmax-level expressiveness at substantially reduced cost, unlocking scalable long, high resolution video generation.
本文显著推进了针对已对齐语言模型的对抗攻击 SOTA,并提出了关于如何防止此类系统生成不良信息的重要问题。This work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information.
本实证研究通过建模数据组成对计算定律及分配指数的影响,推导出指定模型规模、token 数和数据混合精确配置的效率前沿,为可预测地扩展多模态基础模型奠定了必要基础。This empirical research establishes the essential groundwork for predictably scaling multimodal foundation models by modeling the influence of data composition on compute laws and allocation exponents and derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture.
VisCo 是一个训练高效的自压缩框架,复用预训练 VLM 本身作为内在压缩器,使用少量 memory token 压缩视觉信息,并将层次化信息从编码传递到解码。VisCo is a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding.
提出 Spectral Alignment,一种基于 guidance 的轻量级方法,将中间预测的功率谱校准到预先计算的先验,并与 Classifier-Free Guidance (CFG) 互补。Spectral Alignment is proposed, a lightweight, guidance-based method that calibrates the power spectrum of intermediate predictions to a pre-computed prior and is complementary to Classifier-Free Guidance (CFG).
该工作在多语句、多模态设定下研究 ASV,考察跨匿名语音聚合信息是否影响隐私,并发现帧级聚合得到的 EER 最低。This work investigates ASV in a multi-utterance, multimodal setting and examines whether aggregating information across anonymized speech impacts privacy, and finds that frame-level aggregation yields the lowest EERs.
这些结果将 tracked scattering 确立为通往高维 one-step generation 的路径,并给出一张设计图,将 diffusion 相关监督、Drift-like 动力学与 GAN-like 目标联系起来。These results establish tracked scattering as a route to high-dimensional one-step generation and provide a design map relating diffusion-related supervision, Drift-like dynamics, and GAN-like objectives.
该工作提出一个面向异常检测的免训练 agentic 框架,无需领域特定知识,旨在追踪被检测对象随时间变化的空间-时间动态与底层变换,然后基于逐对象的时间状态轨迹进行推理,在 grounding 帧中识别异常对象。This work introduces a training-free agentic framework for anomaly detection free of domain-specific knowledge, designed to track spatial-temporal dynamics and underlying transformations of detected objects over time, and then reason over the object-wise temporal state trajectories to identify abnormal objects in grounded frames.
本文提出无需训练的 Sol-Attn(Sparsifying online attention),在单次 online-softmax pass 中统一动态路由、稀疏计算与近似修正,在稀疏注意力中取得更好的精度–效率权衡。This paper introduces training-free Sol-Attn (Sparsifying online attention), which unifies dynamic routing, sparse computation, and approximation correction in a single online-softmax pass, achieving a better accuracy-efficiency trade-off in sparse attention.
本文提出 Chamaileon,通过将问题建模为跨上下文结合景观(cross-context binding landscape modeling),统一多目标与多态 binder 设计,有效生成可适配多样构象景观与多目标需求的序列。Chamaileon is introduced, which unifies multi-target and multi-state binder design by formulating the problem as cross-context binding landscape modeling and effectively generates sequences adaptable to diverse conformational landscapes and multi-target requirements.
WorldDiT 是一种统一的 diffusion Transformer 架构,将动作生成与视觉世界建模耦合,无需大型预训练 VLM 动作主干即取得强性能,在报告全部四个 suite 的方法中,其总模型参数量与平均成功率处于已报告的 Pareto 前沿上。WorldDiT, a unified diffusion transformer architecture that couples action generation with visual world modeling and achieves strong performance without a large pretrained VLM action backbone, lies on the reported Pareto frontier for total model parameters and mean success among methods reporting all four suites.
本文发现视觉提示工程(visual prompt engineering,简称 VIPE)能在多项任务上提升视频推理性能,甚至比经典的文本提示工程或 test-time scaling 更有效。It is found that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks and can be even more effective than classic text-based prompt engineering or test-time scaling.
本文提出 Mage-VL,一种面向实时多模态理解与交互的高效 codec-native 流式基础模型,并构建了 AI4AI 数据流水线,涵盖面向多模态 captioning 的 prompt-code 联合优化与以 AI 驱动的性能诊断,以指导训练方案。Mage-VL is presented, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction and establishes AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes.
介绍 Parallel Decoding Distillation,一种简化且可扩展的基于轨迹的蒸馏方法,用于 diffusion 和 flow matching 模型的快速推理,并显著提升生成视频的多样性。Parallel Decoding Distillation is introduced, a simplified and scalable trajectory-based distillation method for fast inference of diffusion and flow matching models and presents a significant improvement in generated video diversity.
本文提出基于双向 Mamba 块(Vim)的通用视觉 backbone,通过位置嵌入标记图像序列,并利用双向 state space model 压缩视觉表征,具有成为下一代视觉基础模型 backbone 的巨大潜力。This paper proposes a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models and has great potential to be the next-generation backbone for vision foundation models.
本文提出 embodied language model,将真实世界连续传感器模态直接融入语言模型,从而建立词语与感知之间的联系,实现真实世界中的通用推理。This work proposes embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts to enable general inference in the real world.
本文提出极简的预训练框架 SimVLM,在广泛的判别式与生成式视觉-语言基准上显著超越既往预训练方法并取得新 SOTA,包括 VQA、NLVR2 以及图像描述任务。This work presents a minimalist pretraining framework, named SimVLM, which significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA, NLVR2, and image captioning tasks.
结果表明,长上下文 memory 可沿 layer 轴(而非仅沿 token 轴)进行组织,并揭示了有界检索的优势及其在窗口内的压缩代价。These results show that long-context memory can be organized along the layer axis, not only the token axis, and expose both the benefits of bounded retrieval and its in-window compression tax.
ShadowDancer 引入 shadow pair,即在同一动力学下对外观做独立重采样的成对视频,并由 Shadow Library 大规模构建;一个 dynamics family 可控,当且仅当能为其构造出这样的 pair。ShadowDancer introduces shadow pairs, video pairs that replay the same dynamics under independently resampled appearance, constructed at scale by the Shadow Library, so that a dynamics family becomes controllable exactly when such pairs can be constructed for it.
提出 OmniScope,一个无需训练的 token 压缩框架,以 query 作为跨模态共享的语义锚点,并对音频与视频分别估计相关性;由此给出 OmniLLM 推理的简单设计原则:跨模态共享 query,但不共享显著性估计。This work proposes OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video, and suggests a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates.
研究发现收敛后的扩散损失与提示中结构化语言量成比例关系,并采用两个互补度量:白盒似然指标(GPG)与黑盒属性指标(ED)。Surprisingly, it is found that the converged diffusion loss scales with the amount of structured language in the prompt, and two complementary measures are adapted: a white-box likelihood metric (GPG) and a black-box attribute metric (ED).
本文为RGB-NIR低光成像提供新视角:通过引入3D感知神经建模,无需采集干净RGB数据,并能跨不同噪声水平泛化。This paper offers a new perspective for RGB-NIR low-light imaging by incorporating 3D-aware neural modeling, which obviates the requirement for clean RGB data collection, and generalizes across different noise levels.
研究发现privileged shifts无法给出可靠的答案对齐方向,其幅度主要反映反事实敏感性而非token级学习价值;提出Counterfactual Sensitivity Credit Reallocation (CSCR),作为GRPO的简单扩展,降低高敏感token的credit并对token级优势重新归一化,同时保留原始credit预算与verifier确定的方向。These findings show that privileged shifts fail to provide reliable answer-aligned directions, while their magnitudes primarily reflect counterfactual sensitivity rather than token-level learning value, and propose Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens and renormalizes token-level advantages to preserve both the original credit budget and verifier-determined direction.
本文提出 Motion Beyond Morphology(超越形态的运动)这一视角,旨在跨固定结构对应迁移运动,通过两阶段框架保留在不同目标形态间仍具有意义的动力学。This work introduces Motion Beyond Morphology, a perspective that seeks to transfer motion beyond fixed structural correspondence, by preserving dynamics that remain meaningful across different target morphologies by proposing a two-stage framework.
本文提出 3DZip,一种三阶段 token 压缩框架:首先采用粗粒度体素化去除点级冗余,再通过 Determinantal Point Process 基于特征空间多样性选取锚点 token,最后在空间约束下融合剩余 token 以保持几何一致性。3DZip is proposed, a three-stage token compression framework that first applies coarse voxelization to remove point-level redundancy, then selects anchor tokens based on feature-space diversity via a Determinantal Point Process, and finally merges remaining tokens under spatial constraints to preserve geometric coherence.
本文提出 LeapTalk,一种通过单次前向实现稳定且实时说话头生成、可扩展至任意长视频的新颖框架,并引入音频驱动的无分类器引导机制,在极端步数缩减下保持细粒度唇形同步。LeapTalk is proposed, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos, and an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction.
本文提出 DeepVoyager-VL,一种面向视觉在环搜索的长程多模态深度搜索框架,通过构建多模态事件图驱动数据合成,从而产出具有中间视觉依赖与长推理链的问题。DeepVoyager-VL is proposed, a long-horizon multimodal deep-search framework for vision-in-the-loop search that constructs a multimodal event graph to drive data synthesis, yielding problems with intermediate visual dependencies and long reasoning chains.
本文提出 DreamTraj,从单张 RGB 图像和任务指令预测物体 6-DoF 轨迹,推理时无需视频、深度或 CAD 模型,是首个直接从中间视频扩散表征(而非生成像素)解码物体 6-DoF 轨迹的方法This work proposes DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference, and is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels.
在四种拆分 backbone 上,ReBA 降低所有报告 benchmark 输入的负载,同时保持与 Std-Aux 相当的平均任务准确率,并在分辨率与分块变化下降低测试范围内的平均负载与最差物理负载Across four split backbones, ReBA lowers load on every reported benchmark input while keeping mean task accuracy comparable to Std-Aux, and lowers average load over the tested range and worst physical load under resolution and tiling shifts.
本文在中间层附加轻量预测头,保持 backbone 冻结,并利用中间预测与最终预测的差异作为采样时的自引导方向,训练一个能够自引导的冻结预训练像素扩散模型This work attaches a lightweight prediction head to an intermediate layer, keep the backbone frozen, and use the discrepancy between the intermediate and final predictions as a self-guidance direction during sampling to train a frozen, pretrained pixel diffusion model that can guide itself.
在作者研究的粗粒度属性上,MLLM 编码了视觉证据但无法可靠控制对其的依赖For the coarse attributes the authors study, MLLMs encode the visual evidence but cannot reliably control their reliance on it, indicating that for the coarse attributes they study, MLLMs cannot reliably control their reliance on it.
提出 Video-DR,采用解耦的感知-探索流水线与分阶段工具解锁,强制在 web 检索前进行充分的跨帧视觉定位,实现突破模仿学习上限的自主探索。Video-DR is introduced, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web retrieval, enabling autonomous exploration that breaks the imitation-learning ceiling.
提出 V2N (Video to Notes),首个完整的 VPT 系统:共享时序 backbone 驱动 onset、offset、key hold、velocity 等任务专属 head,采用逐帧监督而非仅在窗口中心进行联合训练。This work presents V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center.
提出 MiniWorld,一个从零训练流式视频世界模型的可复现框架,采用 chunk-wise 非递减噪声调度与两阶段继续训练,提升时间建模与稳定性,将促进未来视频世界模型的研究。MiniWorld is presented, a reproducible framework for training streaming video world models from scratch that adopts a chunk-wise non-decreasing noise schedule and two-stage continued training to improve temporal modeling and stability and will facilitate future research on video world modeling.
为人体动作识别引入新问题域:基于标准 RGB 视频对儿童步态行为进行细粒度分析,并描述一个统一的端到端框架用于解码儿科步态的基本组成。This work introduces a new problem domain for human action recognition: the fine-grained analysis of children's gait behaviors from standard RGB video, and describes a unified end-to-end framework for decoding fundamental components of pediatric gait.