研究发现,recall@k 并非已部署 KB-VQA 的正确评价指标,且弥合差距需要 reader 侧介入;本文首次对多模态 KB-VQA 中 reader 侧位置依赖性进行了受控探查,设计了一种 gold-position 协议——在问题提示中仅改变 gold passage 所在的槽位。The findings indicate that recall@k is the wrong metric for deployed KB-VQA and that closing the gap requires reader-side intervention; the first controlled probe of reader-side position dependence in multimodal KB-VQA is designed, a gold-position protocol in which only the gold passage's prompt slot varies within question.
论文
258 张论文卡片 · 多模态
本文提出 Vision-aligned Latent Reasoning(VaLR),一种简洁而有效的推理框架,在每个 Chain of Thought 推理步骤之前动态生成视觉对齐的 latent token,引导模型在 latent space 中基于感知线索进行推理。Vision-aligned Latent Reasoning (VaLR) is introduced, a simple, yet effective reasoning framework that dynamically generates vision-aligned latent tokens before each Chain of Thought reasoning step, guiding the model to reason based on perceptual cues in the latent space.
本文提出 ContextRL,一种上下文感知的强化学习方法,通过间接辅助目标提升长周期推理与多模态性能,并与将相同对比上下文复用作标准 query–context–answer 样本的数据增强基线进行对比。This work proposes ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an indirect auxiliary objective, and compares against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples.
本研究建立了一套实用的 LongPT 方案,为推进长上下文 vision-language 模型奠定了经验基础,并提出 MMProLong,无需任务专属监督即可泛化至基于网页的多模态 needle 检索、长上下文图文压缩以及长视频理解等任务。This study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models, and introduces MMProLong, which generalizes to webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding without task-specific supervision.
本文提出 LLaDA-V,一种完全基于扩散范式的多模态大语言模型 (MLLM),将视觉指令微调与 masked diffusion 模型相结合,脱离了当前多模态方法中主流的自回归范式。LLaDA-V is introduced, a purely diffusion-based Multimodal Large Language Model (MLLM) that integrates visual instruction tuning with masked diffusion models, representing a departure from the autoregressive paradigms dominant in current multimodal approaches.
本文推出 STEP3-VL-10B,一个面向"紧凑效率与前沿级多模态智能"权衡的轻量级开源基础模型,并发布完整模型套件,为社区提供强大、高效且可复现的 baseline。STEP3-VL-10B is presented, a lightweight open-source foundation model designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence, and the full model suite is released to provide the community with a powerful, efficient, and reproducible baseline.
本文概述了第二届 MAGMaR(Multimodal Retrieval 驱动的多模态增强生成)研讨会共享任务的成果,参赛系统聚焦于视频检索,或在给定检索视频的基础上进行有依据的文章生成。This overview paper presents the results of the shared task for the second workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR), where participants submitted systems focused on either video retrieval or grounded generation of articles given retrieved videos.
我们提出 MOSS-VL,一个开源视觉-语言模型系列,将实时交互(边说边看)视为一等能力。它贯穿整个栈进行协同设计:语言解码器仅通过门控交叉注意力访问视觉,因此模型在生成过程中可以自然地感知新输入帧;合成的交互语料用于监督何时说话、何时沉默、何时修正;分阶段课程将所有实时相关训练集中在一个轻量的最终阶段,基于强大的离线基础模型。在离线场景下,MOSS-VL-Instruct 在We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at
性能评估显示,新训练的深度 CNN 模型 SentiBank 2.0(即 DeepSentiBank)在标注准确率与检索性能上较此前主要采用二分类 SVM 的版本有显著提升。Performance evaluation shows the newly trained deep CNNs model SentiBank 2.0 (or called DeepSentiBank) is significantly improved in both annotation accuracy and retrieval performance, compared to its predecessors which mainly use binary SVM classification models.
提出 Internalized Visual Thinking (IVT),一种在无标签视频上联合优化文本预测与下一 embedding 预测的后训练框架,表明在推理时显式的像素级生成对有效的主动视频推理并非必要。Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and next-embedding prediction over unlabeled videos, is introduced, suggesting that explicit pixel-space generation at inference time may not be necessary for effective proactive video reasoning.
该工作提出 TAMP-Nav,一个用于高效具身导航的统一框架,可在关键节点动态触发 Chain-of-Thought 并仅在关键节点保留高保真记忆,将冗余轨迹压缩为轻量级时空指示器,从而保留关键历史信息并增强时空感知。This work proposes TAMP-Nav, a unified framework for efficient embodied navigation that dynamically triggers Chain-of-Thought and retains high-fidelity memory only at critical nodes, compressing redundant trajectories into lightweight Space-Time Indicators, thereby preserving critical historical information and enhancing spatio-temporal perception.
提出 EditBridge,一个用于高效超高分辨率编辑的扩散桥框架,可在最高 4K 分辨率下实现高保真编辑与卓越感知质量,在 2K 分辨率下带来 3.6–8.4 倍加速,并能在 61 秒内完成实用 4K 编辑。This work proposes EditBridge, a diffusion bridge framework for efficient ultra high-resolution editing that achieves high-fidelity editing with superior perceptual quality at resolutions up to 4K, delivering 3.6--8.4$\times$ speedup at 2K and enabling practical 4K editing in 61 seconds.
提出 DiSCO,一种零样本、严格黑盒的防御方法,完全在提示层面以即插即用模块的形式运行,无需模型重训练、微调或访问模型内部,可直接应用于任何文本到图像系统,无需对模型本身进行任何修改。DiSCO is proposed, a zero-shot, strictly black-box defense that operates entirely at the prompt level as a plug-and-play module, requiring no model retraining, fine-tuning, or access to model internals, and can be readily applied to any text-to-image system without necessitating any changes to the model itself.
本文系统性地研究了视觉编码器扩展中的 MoE 设计,发现细粒度 MoE 拓扑相较于稠密与标准 MoE 基线均带来显著提升;提出了一种无辅助损失的均衡变体以改善专家利用率,并设计了专用 MoE kernel 以缓解推理时延开销。This work systematically study MoE designs for vision encoder scaling and finds that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts, and proposes an auxiliary-loss-free balancing variant for better expert utilization, and designs a specialized MoE kernel to mitigate inference latency overhead.
在代表性视频生成模型上的实验表明,在保持参考图像中任务相关语义一致性的同时实现预期结果仍然具有挑战性Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.
本文提出 OmniScientist,一种端到端、全模态 AI 科学家,可直接基于异构原始证据开展跨学科研究,表明全生命周期感知对于基于证据的科学发现至关重要,并为构建广泛适用的 AI 科学家提供了一条切实可行的路径OmniScientist is introduced, an end-to-end, omni-modal AI scientist that conducts multidisciplinary research directly from heterogeneous raw evidence and demonstrates that lifecycle-wide perception is essential for evidence-grounded scientific discovery and provides a practical path toward broadly capable AI scientists.
VA-Judger:用于联合视频-音频生成的思维链全模态奖励模型。它首先从具有明显质量差距的数据对中学习,以建立结构化输出与粗粒度偏好判别;然后通过对照人工标注进行拒绝采样,蒸馏得到针对更难近质量对比的可靠偏好解释;最终执行按维度分解的强化学习,将人类反馈分解为各个质量维度以获得更密集的奖励信号。VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation that first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals.
提出 RapidLiDAR,一种将初始化本身视为可学习、数据驱动组件的 LiDAR 场景补全方法,在与 SOTA 相当的补全性能下,0.1 秒完成整个场景,比此前最快方法快 2.3 倍。RapidLiDAR is presented, a LiDAR scene completion method that treats the initialization itself as a learned, data-driven component and achieves completion performance on par with the state of the art while completing a full scene in 0.1 seconds, which is 2.3 times faster than the fastest prior method.
4DAnyone 在新视角视频质量和下游 4DGS 重建上均优于先前方法,并具有稳健的野外泛化能力。4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization.
长周期机器人操控要求机器人既能可靠执行各项技能,又能在一系列长时间任务中将其连贯编排。多数分层视觉-语言-动作(VLA)模型仅通过单次前向过程做出每个决策,缺乏将额外算力分配给困难或关键抉择的机制。我们提出 τ_0-VLA,一种分层机器人基础模型,将高层子任务生成建模为可通过世界模型引导的测试时计算来扩展算力的推理问题。在每次推理时,高层策略借助执行……Long-horizon robot manipulation requires a robot to both execute individual skills reliably and sequence them coherently over extended tasks. Most hierarchical vision-language-action (VLA) models make each such decision with a single forward pass, leaving no mechanism to allocate additional computation to difficult or consequential choices. We introduce τ_0-VLA, a hierarchical robot foundation model that formulates high-level subtask generation as a compute-scalable inference problem through world-model-guided test-time computation. At each inference step, the high-level policy uses execution
免训练的块稀疏注意力可加速视频 Transformer,但仅凭逐行注意力集中度本身无法确定一个可执行的稀疏算子。共享同一块路由路径的查询其支撑集可能重叠很差,而仅保留的注意力质量并不足以决定由跳过的交互所产生的 softmax 后误差。我们证明划分几何同时影响池化支撑集与从稀疏输出预测剩余残差的能力。我们提出 SparsePR,将响应耦合划分与探测拟合残差重构相结合。采样查询键Training-free block-sparse attention can accelerate video transformers, but row-wise attention concentration does not by itself specify an executable sparse operator. Queries sharing a block route may have poorly overlapping supports, while retained attention mass alone does not determine the post-softmax error from skipped interactions. We show that partition geometry affects both pooled support and the predictability of the remaining residual from the sparse output. We introduce SparsePR, which combines Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction. Sampled-query ke
我们提出 Hydra-0,一种以动作流为条件的通用世界模型,将机器人动作表示为像素运动。这种共享的视觉接口使得跨具身、任务、环境和视频生成 backbone 的通用世界建模与控制成为可能,学习动作在不同场景下的后果。我们的最佳配置相比动作条件 baseline,机器人运动误差降低 90.4%,物体运动误差降低 60.2%,同时支持零样本组合与数据高效适配。在 RoboLab 基准上,Hydra-0 在 replayeWe introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replaye
语义视觉编码器已成为多模态理解与图像生成中语义条件的关键视觉接口。然而其最终 token 丢弃了细粒度视觉细节,导致像素重建质量较差,限制了其在图像生成与编辑等对重建敏感的任务中的应用。本工作探讨理解、生成与编辑能否在由预训练语义 ViT 构建的单一视觉表征空间中建模。我们证明,语义 ViT 的冻结 Transformer 块本身并非无法保留Semantic vision encoders have become a central visual interface for multimodal understanding and semantic conditioning in image generation. However, their final tokens discard fine-grained visual details, leading to poor pixel reconstruction and limiting their use in reconstruction-sensitive tasks such as image generation and editing. In this work, we ask whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrained semantic ViT. We show that the frozen Transformer blocks of a semantic ViT are not intrinsically unable to preserve
尽管文本到 3D 生成进展迅速,在低推理成本下实现高几何保真度仍具挑战。现有文本到 3D 方法要么自回归地解码离散形状 token,要么通过扩散或流模型迭代优化全局 3D 表示。然而,自回归解码是顺序执行的且无法修正错误,而扩散与流匹配模型反复处理完整表示,使高质量生成成本日益高昂。本文提出 Block3D,一种块级扩散框架,将离散 sWhile text-to-3D generation has advanced rapidly, achieving high geometric fidelity at low inference cost remains challenging. Existing text-to-3D methods either decode discrete shape tokens autoregressively or iteratively refine global 3D representations with diffusion or flow models. However, autoregressive decoding is sequential and cannot revise errors, whereas diffusion and flow-matching models repeatedly process the full representation, making high-quality generation increasingly expensive. In this paper, we propose Block3D, a block-wise diffusion framework that partitions the discrete s
现有图像编辑框架主要沿用 text-to-image 扩散模型的训练范式。然而将该范式扩展到图像编辑时暴露出两个固有差异:一是对编辑概念粒度的关注不足,二是稀疏监督信号导致的训练低效。为应对这些问题,我们建立了一个包含超过 1,000 种细粒度编辑概念的综合分层分类体系,并构建了 ConceptEdit-12M——通过改进的合成框架生成的包含 1,200 万高质量编辑对的超大规模数据集。该 library-drExisting image editing frameworks predominantly follow the training paradigm of text-to-image diffusion models. However, extending this paradigm to image editing highlights two inherent discrepancies, specifically, the insufficient attention to edit concept granularity and the training inefficiency caused by sparse supervision signals. To address these issues, we establish a comprehensive hierarchical taxonomy featuring over 1,000 fine-grained edit concepts and build ConceptEdit-12M, a massive dataset of 12 million high-quality editing pairs via an improved synthesis framework. This library-dr
深度人脸识别 (FR) 模型已达到近乎饱和的精度,但仍缺乏透明度:从业者无法追问某个相似度分数究竟依赖了哪些语义属性。EXPL-FR 在 FR 模型自身的嵌入空间内给出答案。一个轻量适配器将 vision-language model (VLM) 的图像编码器与冻结的 FR 空间对齐,仅基于人脸图像训练,从不基于文本。由于 VLM 的编码器共享同一空间,同一适配器同样适用于文本编码器,从而无需额外成本即可将 22 个类别中的 978 条属性提示(也可扩展)转化为 FR 空间锚点。我们并未假设这种迁移Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this tra
ViT-5 的设计与当代基础模型实践保持一致,可作为对 vanilla ViT 的直接替换升级方案,适用于 2020 年代中期的视觉骨干网络,并为生成建模提供更强大的骨干。With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones and serves as a stronger backbone for generative modeling.
构建了 RoboGenesis,一个基于仿真的工作流与数据引擎,可从原子技能组合配置好的实验工作流,对 rollout 进行验证与过滤,并跨支持的机器人配置导出结构化演示数据。RoboGenesis is built, a simulation-based workflow and data engine that composes configured laboratory workflows from atomic skills, validates and filters rollouts, and exports structured demonstrations across supported robot profiles.
本文提出 DashboardQA,这是首个明确设计用于评估视觉-语言 GUI Agent 对真实世界仪表板理解与交互能力的基准,结果表明交互式仪表板推理对所有受评估的 VLM 而言都是一项具有挑战性的任务。DashboardQA is introduced, the first benchmark explicitly designed to assess how vision-language GUI agents comprehend and interact with real-world dashboards, and indicates that interactive dashboard reasoning is a challenging task overall for all the VLMs evaluated.
提出 LingBot-Video,一种专为具身智能设计的基于 DiT 的视频预训练范式,并将其作为首个大规模开源 MoE 视频基础模型贡献给社区,致力于在数字创意与物理执行之间搭建桥梁。LingBot-Video is presented, a DiT-based video pretraining paradigm specifically tailored for embodied intelligence, and is contributed as the inaugural large-scale, open-source MoE video foundation model to the community, in a pioneering effort to bridge digital creativity and physical actuation.
首次在 World Modeling 领域引入 Agentic Harness 框架:由 pilot agent 负责规划并执行角色行为,director agent 负责随着场景推进合成新的环境要素。The integration of an agentic harness within the domain of world modeling is pioneered, wherein a pilot agent is tasked with planning and executing character behaviors, while a director agent is responsible for synthesizing novel environmental elements as the scene progresses.
提出 LaMem-VLA,一种以潜在记忆为核心框架的方法,将历史经验重建为潜在记忆 token,并直接与 VLA 推理交织,使记忆能够在有界上下文下直接参与 VLA 推理并引导动作生成。LaMem-VLA is introduced, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning, and enables memory to directly participate in VLA reasoning and guide action generation under a bounded context.
WildCity 是一个由自动驾驶车队在复杂城市环境中采集的真实多模态数据集,旨在推动城市级渲染的进展,并更广泛地推动 AI 在空间感知、记忆与推理方面达到与人类认知相当规模的能力。WildCity, a real-world multimodal dataset collected by autonomous fleets traversing complex urban environments, aims to catalyze progress not only in city-scale rendering, but more broadly in the pursuit of AI that can perceive, remember, and reason across space at a scale comparable to human cognition.