论文

提出 Flamingo，一个 VLM 系列，能够桥接强大的纯视觉与纯语言预训练模型，处理任意交错排列的图文序列，并无缝接收图像或视频作为输入。This work introduces Flamingo, a family of Visual Language Models (VLM) with this ability to bridge powerful pretrained vision-only and language-only models, handle sequences of arbitrarily interleaved visual and textual data, and seamlessly ingest images or videos as inputs.

Gemini 1.5 在跨模态长上下文检索任务上取得近乎完美的召回率，在长文档 QA、长视频 QA 与长上下文 ASR 上刷新 SOTA，并在广泛基准上达到或超越 Gemini 1.0 Ultra 的 SOTA 表现。Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks.

实验表明，与语言模型类似，视觉模型也会学习利用全局捷径，从而无法在任务长度或复杂度上泛化，但本文证明基于严格局部感知的循环视觉策略可以缓解这些失败，从而使模型在这些任务上具备泛化能力。The experiments reveal that, similar to language models, vision models can learn to exploit global shortcuts and thereby fail to generalize over task length or complexity, but it is shown that recurrent vision policies based on strictly local perception can mitigate these failures, thereby allowing models to generalize on these tasks.

该模型在效果上匹配基于优化的方法，同时实现近 800 倍加速，并以明显更少的参数量和更低的训练/推理开销超越 SOTA 可泛化模型的零样本性能，整体效率显著提升。This model matches optimization-based methods while delivering nearly 800x speedup, and surpasses the zero-shot performance of state-of-the-art generalizable models with markedly fewer parameters and reduced training/inference overhead, achieving an overall efficiency improvement.

本文提出 token 时间连续扩散 (TTCD)，一种新的扩散语言模型，运行于连续空间，将高斯噪声确定性地映射到最终的 token canvas 而无需额外采样，并引入每个 token 时间的新概念。This paper introduces token time continuous diffusion (TTCD), a new diffusion language model which operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and incorporates a new notion of per-token times.

S1-Omni 基于 S1-Omni-Corpus 训练，覆盖 200 个科学任务并包含数百万推理样本，在 60 余个科学基准上进行了评估，为统一的科学建模提供了一条可行路径。S1-Omni is trained on S1-Omni-Corpus, which covers 200 scientific tasks and contains millions of reasoning samples, and is evaluated on over 60 scientific benchmarks, providing a practical path toward unified scientific modeling.

表明差值信号显著提升在策略蒸馏效果，提出新方法称为在策略差值蒸馏（OPD），使推理LLM仅需短暂的后训练即可获得强性能。It is shown that the delta signal substantially improves on-policy distillation and the new distillation method is referred to as On-Policy Delta Distillation (OPD), enabling reasoning LLMs to achieve strong performance with only a short post-training period.

VideoRAE是一种表征自编码器，利用冻结视频基础编码器的多尺度分层特征，并通过轻量级1D自注意力投影器进行压缩，验证了冻结VFM表征可作为通用且利于生成的视频潜变量。VideoRAE is a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector to validate frozen VFM representations as versatile and generation-friendly video latents.

本文回顾现有方法，并融合多种技术从数据与模型规模两方面扩展预训练，提出一条自动化流水线以构建专用、多样且经过筛选的图像数据集，替代自监督文献中常用的未筛选数据。This work revisits existing approaches and combines different techniques to scale the pretraining in terms of data and model size, and proposes an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature.

Pointmaps 在保留预训练 2D VLA 所需 H × W 稠密网格的同时，提供机器人坐标系下的 3D 几何信息，能以极小的架构改动集成到现有 VLA 中，并提升 pi0.5 与 SmolVLA 的性能，优于代表性的相机视点和 3D 感知基线。Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change and improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines.

HOMIE 提出了一种更优的 MLLM 集成策略，可在不损害文本编码器可控性或引入昂贵重新对齐的前提下，提取参考级关系知识，并在 self-attention 中引入全局多模态引导，使 MLLM 派生的语义特征与 VAE token 更好对齐。HOMIE proposes a better MLLM integration strategy to extract knowledge of reference-level relationships without compromising the controllability of text encoders or incurring costly re-alignment, and introduces global multimodal guidance within self-attention to better align MLLM-derived semantic features with VAE tokens.

在七个基准上，TimeLens2-2B 在所有基准上均优于规模相当的所有基线，4B 和 8B 变体则取得了 SOTA 性能，超越了参数量高达 397B 的开源模型。Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters.

提出了ReflectWorld-MM，一个面向开放视频流的以实体为中心的多模态记忆系统，在六个长视频和终身记忆基准测试上均达到最优准确率，超越了强记忆Agent和前沿模型。ReflectWorld-MM is proposed, an entity-oriented multimodal memory system for open-ended video streams that achieves the best accuracy on all six long-video and lifelong-memory benchmarks, outperforming strong memory agents and a frontier model.

本文研究了在采用极小卷积滤波器的架构下，卷积网络深度对大规模图像识别精度的影响，并表明将深度推进至 16-19 个权重层，可在先前 SOTA 配置基础上取得显著提升。This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.

在 YouCook2、CrossTask 等教学视频数据集上，基于该数据训练的文本-视频 embedding 在文本到视频检索与动作定位任务上达到了 SOTA 结果。It is demonstrated that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask.

提出在通过跨模态注意力融合之前对齐图像与文本表示的对比损失（ALBEF），可实现更扎实的视觉-语言表征学习；并提出动量蒸馏，一种利用动量模型生成伪目标进行自训练的方法。A contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning and proposes momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model.

本文提出 LLaVA：Large Language and Vision Assistant，一个端到端训练的大型多模态模型，将视觉编码器与 LLM 相结合用于通用视觉和语言理解；并引入 GPT-4 生成的视觉指令微调数据，模型与代码库已开源。This paper presents LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding and introduces GPT-4 generated visual instruction tuning data, the model and code base publicly available.

m-RNN 模型直接对给定先前词语和图像条件下生成下一个词的概率分布建模，相较于直接优化排序目标函数进行检索的 SOTA 方法，取得了显著的性能提升。The m-RNN model directly models the probability distribution of generating a word given previous words and an image, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.

提出极简的 VLP 模型 Vision-and-Language Transformer (ViLT)，其一体化设计将视觉输入处理大幅简化为与文本输入相同的无卷积方式；ViLT 比此前的 VLP 模型快达数十倍，同时下游任务性能具有竞争力甚至更优。A minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that the authors process textual inputs, showing that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance.

Contrastive Captioner (CoCa) 采用极简设计，对图文编码器-解码器基础模型联合使用对比损失与字幕损失进行预训练，从而兼具 CLIP 等对比方法与 SimVLM 等生成方法的能力。Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM.

本文提出 MiniGPT-4，通过一个投影层将冻结的视觉编码器与冻结的先进 LLM Vicuna 对齐，发现将视觉特征与先进大语言模型恰当对齐可获得类似 GPT-4 所展现的多种先进多模态能力。MiniGPT-4 is presented, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer to uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by G PT-4.

本文表明视频掩码自编码器（VideoMAE）是自监督视频预训练（SSVP）的数据高效学习器，并受近期 ImageMAE 启发，提出采用极高掩码比例的定制化视频管状掩码策略。This paper shows that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP), and proposes customized video tube masking with an extremely high ratio, inspired by the recent ImageMAE.

本文基于预训练 BLIP-2 模型，对视觉-语言指令微调展开系统全面研究，并提出指令感知的 Query Transformer，用于提取针对给定指令的信息丰富特征。This paper conducts a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models, and introduces an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction.

ReViV 是首个用于整体第一人称 4D 重建的统一框架，能够从单个单目 RGB 视频中同时提取观察者与视角动态，在整体自身体、手部与注视重建以及相机跟踪方面达到 SOTA 精度与效率，同时保持极具竞争力的第一人称深度估计能力。ReViV is the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video and achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation.

本工作通过 teacher forcing 与 ODE 蒸馏，将一个双向动作条件教师模型逐步蒸馏为因果学生模型，并提出 LongForcing，将学生模型的长时间自展开与扩展时域教师模型对齐，从而缓解累积的分布漂移与自回归漂移。This work progressively distill a bidirectional action-conditioned teacher into a causal student through teacher forcing and ODE distillation, and introduces LongForcing to align long student self-rollouts with an extended-horizon teacher, mitigating accumulated distribution shift and autoregressive drift.

Self Gradient Forcing（SGF）是一种两阶段训练策略，在原生自回归训练目标内恢复缺失的"记忆写入"监督信号，通过对未来视频 latent 的损失来训练模型将上下文编码为更有效的因果记忆。Self Gradient Forcing (SGF), a two-pass training strategy that restores this missing memory-writing supervision within the native autoregressive training objective, using losses on future video latents to train the model to encode context into more effective causal memory.

本文提出 Trace，一个面向多领域视觉推理的、由分类体系引导的环境，其将任务构建分解为场景语法与可执行任务程序，将视觉呈现与答案计算解耦，并提供了广泛的程序化训练可迁移到生成任务分布之外的证据。Trace is introduced, a taxonomy-guided environment for multidomain visual reasoning that factorizes task construction into a scene grammar and an executable task program, separating visual realization from answer computation, providing evidence that broad procedural training can transfer beyond the generated task distributions.

本文提出 \method，一种无需训练的稀疏注意力系统，可在多 GPU 序列并行下提升自适应稀疏注意力的分布式执行效率。This work presents \method, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism.

提出两项审计协议——grounding and truth 对比以及 swap to an independent evaluator，以及 RECAP（Readable Encodings via Co-trained Auxiliary Predictors），即与目标模型联合训练的线性头，用于保持指定内容的可解码性。Two audit protocols, the comparison of grounding and truth and the swap to an independent evaluator, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors), linear heads trained alongside the target model to keep designated content decodable are contributed.

本文设计了一种直接处理点云的新型神经网络，较好地尊重了输入点的置换不变性，并为从物体分类、部件分割到场景语义解析等应用提供了统一架构。This paper designs a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input and provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing.

本文提出了一个自然场景文本识别框架，无需任何人工标注数据，并以整体方式对整幅图像进行单词识别，区别于过去基于字符的识别系统。This work presents a framework for the recognition of natural scene text that does not require any human-labelled data, and performs word recognition on the whole image holistically, departing from the character based recognition systems of the past.

研究揭示，正弦激活会诱发谐波线谱，为循环展开如何丰富隐式神经表示（INR）的有效频谱支撑提供了频谱层面的解释。It is revealed that sinusoidal activations induce a harmonic line spectrum, providing a spectral account of how recurrent unrolling enriches the effective spectral support in implicit neural representations (INRs).

提出 ReferTrack，一种"先指代后跟踪"的范式，仅使用单个前向摄像头完成 EVT grounding，在四足机器人和人形机器人上的真实部署验证了其稳健的 sim-to-real 迁移能力。ReferTrack is introduced, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera, and real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities.