Papers · organized/paper_cards

论文

162 张论文卡片 · 多模态 · OA 绿色

开放获取 全部 绿色 · 724
Object Detection in 20 Years: A Survey
目标检测二十年:综述
arXiv:1905.05055 多模态 综述 OA · 绿色 被引 3564 · S2

本文从技术演进的角度,对这一快速发展的研究领域进行了广泛综述,跨越超过四分之一世纪的时间跨度(从 1990 年代到 2022 年)。This article extensively reviews this fast-moving research field in the light of technical evolution, spanning over a quarter-century’s time (from the 1990s to 2022).

Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
多模态 LLM 的计算幽默:方法、数据集、评估与挑战
arXiv:2607.19011 多模态 综述 OA · 绿色 被引 0 · S2 + OpenAlex

本综述聚焦于单图与多格视觉作品中的幽默理解,同时将幽默生成视为新兴的下游前沿方向,并围绕多模态对齐、证据 grounded 推理与可控生成,对基准设计、评估协议与建模范式进行系统综述。This survey focuses on visual humor understanding in single-image and multi-panel artifacts, while treating humor generation as an emerging downstream frontier, and synthesizes benchmark design, evaluation protocols, and modeling paradigms based on multimodal alignment, evidence-grounded reasoning, and controlled generation.

Self Gradient Forcing: Native Long Video Extrapolation
Self Gradient Forcing:原生长视频外推
arXiv:2607.20368 多模态 方法 OA · 绿色 被引 1 · S2

Self Gradient Forcing(SGF)是一种两阶段训练策略,在原生自回归训练目标内恢复缺失的"记忆写入"监督信号,通过对未来视频 latent 的损失来训练模型将上下文编码为更有效的因果记忆。Self Gradient Forcing (SGF), a two-pass training strategy that restores this missing memory-writing supervision within the native autoregressive training objective, using losses on future video latents to train the model to encode context into more effective causal memory.

Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning
Trace:面向多领域视觉推理的 Taxonomy 引导环境
arXiv:2607.19790 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文提出 Trace,一个面向多领域视觉推理的、由分类体系引导的环境,其将任务构建分解为场景语法与可执行任务程序,将视觉呈现与答案计算解耦,并提供了广泛的程序化训练可迁移到生成任务分布之外的证据。Trace is introduced, a taxonomy-guided environment for multidomain visual reasoning that factorizes task construction into a scene grammar and an executable task program, separating visual realization from answer computation, providing evidence that broad procedural training can transfer beyond the generated task distributions.

An Exam for Active Observers
面向主动观察者的评测
arXiv:2607.16165 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

人类视觉是一个闭环:注视点不断被中间假设而非单一快照持续重定向。数十年的心理物理学与认知科学研究表明,主动观察对多种任务至关重要。当代多模态大语言模型 (MLLM) 是否进行主动观察,是一个现有视觉语言基准无法回答的经验问题。我们提出 ActiveVision,一个使 MLLM 主动观察可度量的基准,包含 3 个类别共 17 个任务,任务设计强制进行重复视觉感知……Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception

FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
FVAttn:面向视频生成的自适应稀疏注意力与运行时负载均衡
arXiv:2607.16190 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文提出 \method,一种无需训练的稀疏注意力系统,可在多 GPU 序列并行下提升自适应稀疏注意力的分布式执行效率。This work presents \method, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism.

Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations
训练模型而非读者:用于可验证激活解释的可解码性监督
arXiv:2607.20379 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

提出两项审计协议——grounding and truth 对比以及 swap to an independent evaluator,以及 RECAP(Readable Encodings via Co-trained Auxiliary Predictors),即与目标模型联合训练的线性头,用于保持指定内容的可解码性。Two audit protocols, the comparison of grounding and truth and the swap to an independent evaluator, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors), linear heads trained alongside the target model to keep designated content decodable are contributed.

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
arXiv:1612.00593 多模态 方法 OA · 绿色 被引 18312 · S2

本文设计了一种直接处理点云的新型神经网络,较好地尊重了输入点的置换不变性,并为从物体分类、部件分割到场景语义解析等应用提供了统一架构。This paper designs a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input and provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing.

Gemini: A Family of Highly Capable Multimodal Models
Gemini: A Family of Highly Capable Multimodal Models
arXiv:2312.11805 多模态 方法 OA · 绿色 被引 829 · OpenAlex
Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
arXiv:1406.2227 多模态 方法 OA · 绿色 被引 1000 · S2

本文提出了一个自然场景文本识别框架,无需任何人工标注数据,并以整体方式对整幅图像进行单词识别,区别于过去基于字符的识别系统。This work presents a framework for the recognition of natural scene text that does not require any human-labelled data, and performs word recognition on the whole image holistically, departing from the character based recognition systems of the past.

ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
arXiv:2607.20092 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

提出 ENTRAP-VL(ENTRainment Assessment Probe for Vision and Language),一个由人工策展的 1,500 条数据的数据集,涵盖八个类别,按一个跨双轴的分类体系组织,并划分为文本诱发流和视觉诱发流。ENTRAP-VL (ENTRainment Assessment Probe for Vision and Language), a manually curated dataset of 1,500 items across eight categories, organized by a taxonomy that spans two axes and split into a textual-entrainment stream and a visual-entrainment stream, is introduced.

Recurrent Sinusoidal INRs for Efficient High-Fidelity Representation
循环正弦 INR 用于高效高保真表示
arXiv:2607.21485 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

研究揭示,正弦激活会诱发谐波线谱,为循环展开如何丰富隐式神经表示(INR)的有效频谱支撑提供了频谱层面的解释。It is revealed that sinusoidal activations induce a harmonic line spectrum, providing a spectral account of how recurrent unrolling enriches the effective spectral support in implicit neural representations (INRs).

ReferTrack: Referring Then Tracking for Embodied Visual Tracking
ReferTrack:先指代再跟踪的具身视觉跟踪
arXiv:2607.20061 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

提出 ReferTrack,一种"先指代后跟踪"的范式,仅使用单个前向摄像头完成 EVT grounding,在四足机器人和人形机器人上的真实部署验证了其稳健的 sim-to-real 迁移能力。ReferTrack is introduced, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera, and real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities.

Robostral Navigate
Robostral Navigate
arXiv:2607.20785 多模态 应用落地 OA · 绿色 被引 0 · S2 + OpenAlex

提出 Robostral Navigate,一个围绕该可扩展性目标构建的 8B 视觉语言模型,仅消费单目 RGB 图像流——这是机器人平台中最普及的传感器——通过在当前相机画面中指向下一目标位置来预测航点。Robostral Navigate, an 8B vision-language model built around this scalability objective, is introduced, which consumes only a stream of monocular RGB images - the most ubiquitous sensor across robotic platforms and predicts waypoints by pointing to the next target location in the current camera view.

Color Pass-Through via Camera-Display Coupling
通过相机-显示器耦合实现色彩直通
arXiv:2607.12746 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文提出 Color Pass-Through,一个端到端可学习的框架,直接在采集图像上运行,将相机和显示器视为耦合系统进行联合处理,而非单独校准。This work proposes Color Pass-Through, an end-to-end learned framework that operates directly on captured images, to treat the camera and display as a coupled system rather than calibrating them in isolation.

Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
《Sirens' Whisper:语音驱动 LLM 的不可听近超声越狱》代码
arXiv:2307.15043 多模态 方法 OA · 绿色 被引 3560 · S2

本文显著推进了针对已对齐语言模型的对抗攻击 SOTA,并提出了关于如何防止此类系统生成不良信息的重要问题。This work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information.

Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
计算机视觉中的生成对抗网络:综述与分类
arXiv:1906.01529 多模态 综述 OA · 绿色 被引 277 · OpenAlex

深入回顾文献中 GAN 相关研究,并从两个视角阐述针对三大挑战所提出的架构变体与损失变体。An in-depth review of GAN-related research in the literature is provided, and an account of the architecture-variant and loss-variants, which have been proposed to handle these three challenges from two perspectives are provided.

Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation
重新思考 On-Policy 扩散蒸馏中的无分类器引导
arXiv:2607.24731 多模态 观点 OA · 绿色 被引 0 · S2 + OpenAlex

将 Positive--Direction Matching (PDM)——一种分支感知的 OPD 目标,分别约束正预测方向与 CFG 条件方向——引入 dense-to-sparse 视频控制;由于朴素的 guided matching 对推理 guidance 尺度极为敏感,分支感知监督可实现更鲁棒、更有效的知识迁移。Positive--Direction Matching (PDM), a branch-aware OPD objective that separately constrains the positive prediction and the CFG conditional direction, is introduced to dense-to-sparse video control, where naive guided matching is highly sensitive to inference guidance scales, while branch-aware supervision enables more robust and effective knowledge transfer.

Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels
无坐标与区域标签的视觉文档理解中的证据归因
arXiv:2607.24651 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

本研究考察是否存在一条实用路径,在没有坐标界面、且无需高成本区域级监督的条件下提升归因效果,并指出了这样一条可行路径。A study investigates whether there is a practical path to improve attribution without a coordinate interface and without costly region-level supervision, and indicates a practical path to improve attribution without a coordinate interface and without costly region-level supervision.

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Sol-Attn:通过即时注意力稀疏化加速视频生成推理
arXiv:2607.24027 多模态 方法 OA · 绿色 被引 2 · S2

本文提出无需训练的 Sol-Attn(Sparsifying online attention),在单次 online-softmax pass 中统一动态路由、稀疏计算与近似修正,在稀疏注意力中取得更好的精度–效率权衡。This paper introduces training-free Sol-Attn (Sparsifying online attention), which unifies dynamic routing, sparse computation, and approximation correction in a single online-softmax pass, achieving a better accuracy-efficiency trade-off in sparse attention.

Chamaileon: Cross-Context Binder Design with Contextualized Modeling and Mixed Sampling
Chamaileon:基于上下文建模与混合采样的跨上下文结合子设计
arXiv:2607.23518 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文提出 Chamaileon,通过将问题建模为跨上下文结合景观(cross-context binding landscape modeling),统一多目标与多态 binder 设计,有效生成可适配多样构象景观与多目标需求的序列。Chamaileon is introduced, which unifies multi-target and multi-state binder design by formulating the problem as cross-context binding landscape modeling and effectively generates sequences adaptable to diverse conformational landscapes and multi-target requirements.

WorldDiT: A Unified Diffusion Architecture for World and Action Modeling
WorldDiT:面向世界建模与动作建模的统一扩散架构
arXiv:2607.23909 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

WorldDiT 是一种统一的 diffusion Transformer 架构,将动作生成与视觉世界建模耦合,无需大型预训练 VLM 动作主干即取得强性能,在报告全部四个 suite 的方法中,其总模型参数量与平均成功率处于已报告的 Pareto 前沿上。WorldDiT, a unified diffusion transformer architecture that couples action generation with visual world modeling and achieves strong performance without a large pretrained VLM action backbone, lies on the reported Pareto frontier for total model parameters and mean success among methods reporting all four suites.

UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
UltraViT:面向大视觉-语言模型的端侧延迟优化视觉编码器
arXiv:2607.23373 多模态 应用落地 OA · 绿色 被引 0 · S2 + OpenAlex

大量实验表明,结合端上延迟感知设计与定制化训练策略,建立了高效 LVLM 编码的新 SOTA,在端上以近 1.7 倍速度运行的同时显著优于现有以编码器为中心的基线。Extensive experiments demonstrate that the on-device latency-informed design combined with the tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.

Visual prompt engineering for video models
视频模型的视觉提示工程
arXiv:2607.25537 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文发现视觉提示工程(visual prompt engineering,简称 VIPE)能在多项任务上提升视频推理性能,甚至比经典的文本提示工程或 test-time scaling 更有效。It is found that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks and can be even more effective than classic text-based prompt engineering or test-time scaling.

Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
Mage-VL:面向高效编解码器原生流式多模态的基础模型
arXiv:2607.24904 多模态 方法 OA · 绿色 被引 1 · S2

本文提出 Mage-VL,一种面向实时多模态理解与交互的高效 codec-native 流式基础模型,并构建了 AI4AI 数据流水线,涵盖面向多模态 captioning 的 prompt-code 联合优化与以 AI 驱动的性能诊断,以指导训练方案。Mage-VL is presented, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction and establishes AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes.

Parallel Decoding Distillation for Fast Image and Video Generation
并行解码蒸馏:面向快速图像与视频生成
arXiv:2607.26004 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

介绍 Parallel Decoding Distillation,一种简化且可扩展的基于轨迹的蒸馏方法,用于 diffusion 和 flow matching 模型的快速推理,并显著提升生成视频的多样性。Parallel Decoding Distillation is introduced, a simplified and scalable trajectory-based distillation method for fast inference of diffusion and flow matching models and presents a significant improvement in generated video diversity.

Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
Vision Mamba:基于双向状态空间模型的高效视觉表示学习
arXiv:2401.09417 多模态 方法 OA · 绿色 被引 2083 · S2

本文提出基于双向 Mamba 块(Vim)的通用视觉 backbone,通过位置嵌入标记图像序列,并利用双向 state space model 压缩视觉表征,具有成为下一代视觉基础模型 backbone 的巨大潜力。This paper proposes a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models and has great potential to be the next-generation backbone for vision foundation models.

PaLM-E: An Embodied Multimodal Language Model
PaLM-E:一种具身多模态语言模型
arXiv:2303.03378 多模态 方法 OA · 绿色 被引 3032 · S2

本文提出 embodied language model,将真实世界连续传感器模态直接融入语言模型,从而建立词语与感知之间的联系,实现真实世界中的通用推理。This work proposes embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts to enable general inference in the real world.

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
SimVLM:基于弱监督的简单视觉语言模型预训练
arXiv:2108.10904 多模态 方法 OA · 绿色 被引 970 · S2

本文提出极简的预训练框架 SimVLM,在广泛的判别式与生成式视觉-语言基准上显著超越既往预训练方法并取得新 SOTA,包括 VQA、NLVR2 以及图像描述任务。This work presents a minimalist pretraining framework, named SimVLM, which significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA, NLVR2, and image captioning tasks.

Human-in-the-Loop Signature Bootstrapping for UAV Hyperspectral PFM-1 Mine Detection
UAV 高光谱 PFM-1 地雷检测中的人在回路签名引导
arXiv:2607.25310 多模态 应用落地 OA · 绿色 被引 1 · S2

本文研究无人机(UAV)可见光-近红外(VNIR)高光谱图像中 PFM-1 地雷的检测,使用光谱角制图(SAM)、匹配滤波器(MF)、自适应相干估计器(ACE)和约束能量最小化(CEM)。This paper studies PFM-1 landmine detection in unmanned aerial vehicle (UAV) visible and near-infrared (VNIR) HSI using spectral angle mapper (SAM), matched filter (MF), adaptive coherence estimator (ACE), and constrained energy minimization (CEM).

CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition
CLBench-V: 评估多模态上下文学习——从 grounding 到知识获取
arXiv:2607.25294 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

本文介绍 CLBench-V,一个多模态上下文学习 benchmark,围绕三个维度组织任务——上下文 grounding、新信息应用与新知识学习——以解决定位上下文使用失效位置的难题。This work introduces CLBench-V, a benchmark for multimodal context learning that addresses the difficulty of localizing where context use breaks down by organizing tasks around three dimensions: context grounding, new information application, and new knowledge learning.

Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory
理解在前部完成:大语言模型中的深度分工及其在无界上下文记忆中的应用
arXiv:2607.28263 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

结果表明,长上下文 memory 可沿 layer 轴(而非仅沿 token 轴)进行组织,并揭示了有界检索的优势及其在窗口内的压缩代价。These results show that long-context memory can be organized along the layer axis, not only the token axis, and expose both the benefits of bounded retrieval and its in-window compression tax.

ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow
ShadowDancer: 通过从视频及其阴影中学习统一动力学表示来教视频世界模型执行任意动作
arXiv:2607.28362 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

ShadowDancer 引入 shadow pair,即在同一动力学下对外观做独立重采样的成对视频,并由 Shadow Library 大规模构建;一个 dynamics family 可控,当且仅当能为其构造出这样的 pair。ShadowDancer introduces shadow pairs, video pairs that replay the same dynamics under independently resampled appearance, constructed at scale by the Shadow Library, so that a dynamics family becomes controllable exactly when such pairs can be constructed for it.

See2Think: Do Multimodal Models Really Use Intermediate Visual States?
See2Think:多模态模型真的使用了中间视觉状态吗?
arXiv:2607.26769 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

对代表性闭源与开源多模态模型的评测表明,视觉推理强依赖于模型与环境,没有任何单一设置能在所有任务上持续占优。Evaluating representative proprietary and open-source multimodal models, it is found that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks.

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
OmniScope:面向全模态大语言模型的模态解耦 token 压缩
arXiv:2607.23193 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

提出 OmniScope,一个无需训练的 token 压缩框架,以 query 作为跨模态共享的语义锚点,并对音频与视频分别估计相关性;由此给出 OmniLLM 推理的简单设计原则:跨模态共享 query,但不共享显著性估计。This work proposes OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video, and suggests a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates.

RL^2-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models
RL^2-VLA:面向 Vision-Language-Action 模型的自适应强化学习潜在组合引导与测试时缩放
arXiv:2607.26991 多模态 观点 OA · 绿色 被引 0 · S2 + OpenAlex

提出一种自适应推理时引导框架,利用VLA Latents上的强化学习,发现推理时引导在成功与失败状态下遵循根本不同的scaling laws:动作多样性在基础VLA可能失败时最为有益,但在成功可能性高时可能不必要地扰动已准确的动作。This work introduces an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents, and discovers that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely.