研究库 论文知识库
Papers · organized/paper_cards

论文

417 张论文卡片 · 多模态 · OA 绿色

开放获取 全部 绿色 · 1640
Vision-Language Grounding as Bidirectional Concept Correspondence
视觉-语言接地作为双向概念对应
arXiv:2608.07886 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

该形式化方法将短语定位、指代表达定位以及开放词汇检测等常见 grounding 任务统一起来,将文本分割、图像分割以及跨模态对齐视为单一的对应预测问题。This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem.

Domain Adaptation for Visual Applications: A Comprehensive Survey
视觉应用中的域适应:综合综述
arXiv:1702.05374 多模态 综述 OA · 绿色 被引 558 · S2

综述领域自适应与迁移学习,重点关注视觉应用及超越图像分类的方法,如目标检测、图像分割、视频分析或视觉属性学习。An overview of domain adaptation and transfer learning with a specific view on visual applications and the methods that go beyond image categorization, such as object detection or image segmentation, video analyses or learning visual attributes are overviewed.

Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation
Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation
arXiv:2608.06751 多模态 方法 OA · 绿色 被引 1 · S2

Atelier 是一种面向艺术家风格图像生成的捷径感知控制状态规划框架,可提升艺术家级风格保真度,更忠实地保持源结构,并相较提示工程、检索增强与通用 agent 基线大幅减少捷径替换。Atelier, a shortcut-aware control-state planning framework for artist-grounded image generation, improves artist-level style fidelity, preserves source structure more faithfully, and substantially reduces shortcut substitution compared with prompt-engineered, retrieval-augmented, and general-purpose agent baselines.

Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands
手部可见性检测器:基于关键点的手部可见性估计
arXiv:2608.11574 多模态 方法 OA · 绿色 被引 1 · S2

本文表明,利用在大规模数据上预训练的 HPE 模型先验知识作为骨干网络可在该任务上取得高性能,并验证了手部可见性检测器在通过 2D 关键点多视角三角化进行 3D 手部姿态标注的下游任务中的有效性。It is shown that leveraging the prior knowledge of HPE models pretrained on large-scale data as a backbone yields high performance in this task, and the utility of Hand Visibility Detector on a downstream task of 3D hand pose annotation via multi-view triangulation of 2D keypoints.

AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Models
AtlasVLA:面向视觉—语言—动作模型的持久世界—自我状态建模
arXiv:2608.06729 多模态 方法 OA · 绿色 被引 1 · S2

AtlasVLA 是一种新框架,通过持久化的世界-自我状态从直接反应式操作转向主动推理,显著优于多视角基线,在 LIBERO-Long 上取得 9.4% 的绝对成功率提升,在真实世界长周期任务中取得 17.5% 的提升。AtlasVLA is a novel framework that transitions from direct reactive manipulation to proactive reasoning through a persistent world-ego state and decisively outperforms multi-view baselines, yielding absolute success rate improvements of 9.4% on LIBERO-Long and 17.5% in real-world long-horizon tasks.

Simplex Relaxation for Discrete Diffusion
离散扩散的单纯形松弛
arXiv:2608.10615 多模态 方法 OA · 绿色 被引 1 · S2

提出 Simplax,一种精确的 Dirichlet–categorical 增强方法,将每个被损坏的 categorical 状态与一个辅助的单纯形值变量耦合,同时保持均匀扩散过程作为其 categorical 边际。Simplax is introduced, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable while preserving the uniform diffusion process as its categorical marginal.

Gaze Target Estimation Anywhere with Concepts
Gaze Target Estimation Anywhere with Concepts
arXiv:2608.11367 多模态 方法 OA · 绿色 被引 3 · S2

本文提出提示式注视目标估计(PGE)任务,一种用于注视分析的端到端、概念驱动新范式,并推出首个为 PGE 设计的模型 GazeAnywhere,它使用基于 Transformer 的检测器融合冻结编码器特征,同时解决主体定位、画内/画外存在性以及注视目标热图估计。The Promptable Gaze Target Estimation (PGE) task is introduced, a new end-to-end, concept-driven paradigm for gaze analysis and GazeAnywhere, the first model designed for PGE, uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation.

DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation
DreamX-Phi 1.0:面向机器人操作的动作条件视频世界模型
arXiv:2608.13489 多模态 方法 OA · 绿色 被引 3 · S2

本文提出一种用于机器人操作的动作条件视频世界模型,给定观测帧、语言指令以及由末端执行器位姿和夹爪状态组成的预定动作序列,预测对应的未来观测结果。An action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations is presented.

UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos
UniSwap:面向说话视频的流式音视频身份替换
arXiv:2608.11752 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文提出 UniSwap,这是首个用于说话视频中流式联合音视频身份替换的框架,并引入 swap-and-reconstruct 流程,从真实片段中移除视觉和声音身份,同时使用原始片段作为重建目标。This work presents UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos, and introduces a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets.

LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time
LiveAnimate:实时稳定的长视频流式人体动画
arXiv:2608.11745 多模态 应用落地 OA · 绿色 被引 1 · S2

本文提出 LiveAnimate,据作者所知是首个将实时流式生成与十亿参数规模下的稳定长视频生成相结合的系统,基于 140 亿参数的视频 Diffusion Transformer(DiT)。This work presents LiveAnimate, to their knowledge the first animation system to combine real-time streaming with stable long-form generation at billion scale, built on a 14B-parameter video Diffusion Transformer (DiT).

H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models
H2R-Bench:世界模型中人到机器人操作视频生成的基准评测
arXiv:2608.13049 多模态 评测集 OA · 绿色 被引 1 · S2

H2R-Bench 提供了一个系统性诊断框架,用于评估视频世界模型能否跨越 human-to-robot 具身差距,并将人类操作观测转化为以机器人为中心的训练资源。H2R-Bench provides a systematic diagnostic framework for evaluating whether video world models can bridge the human-to-robot embodiment gap and convert human manipulation observations into robot-centric training resources.

An AI4AI Framework for Visual Token Pruning
面向视觉 token 剪枝的 AI4AI 框架
arXiv:2608.07193 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文认为关键在于设计合适的 search-state 表示,将 LLM 的内部知识与 visual-token 剪枝的结构要求和约束相连接,并提出 AutoPrune,一种用于 LLM 驱动的 visual-token 剪枝策略设计的免训练框架。This paper argues that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning, and proposes AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design.

Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation
上下文匹配蒸馏:用于自回归视频蒸馏的教师因果性
arXiv:2608.13391 多模态 方法 OA · 绿色 被引 4 · S2

提出 Context-Matched Distillation(CMD),一种因果 DMD 框架,将 teacher 监督信号与每个 target 生成时可用信息对齐,并可自然扩展到逐帧和逐 chunk 生成、长视频蒸馏以及相机条件蒸馏。Context-Matched Distillation (CMD) is introduced, a causal DMD framework that aligns teacher supervision with the information available when each target is generated, and naturally extends to frame-wise and chunk-wise generation, long video distillation, and camera-conditioned distillation.

Robust Speech Recognition via Large-Scale Weak Supervision
Robust Speech Recognition via Large-Scale Weak Supervision
arXiv:2212.04356 多模态 方法 OA · 绿色 被引 8796 · S2

当将监督规模扩展到 680,000 小时的多语言、多任务数据时,所得到的模型在标准 benchmark 上泛化良好,在 zero-shot transfer 设置下常可与此前全监督方法的结果相当,且无需任何微调。When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning.

RibAssist 3D: Biplanar Rib-Fracture Detection, Addressing, and Selective 3D Localization from CT-Derived Projections
RibAssist 3D:基于 CT 投影的双平面肋骨骨折检测、配对与选择性 3D 定位
arXiv:2608.06914 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

建立一个可复现的选择性 3D 定位框架,并指出跨视角对应(cross-view correspondence)是主要操作瓶颈。The study establishes a reproducible framework for selective 3D localization and identifies cross-view correspondence as the dominant operational bottleneck.

Multimodal Model Diffing for Feature Discovery and Control
多模态模型 Diffing:用于特征发现与控制
arXiv:2608.09928 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

MMDiff 是一种多模态 model-diffing 框架,训练多模态 SAE 并将其转化为特征级接口,用于发现和控制多模态行为;研究表明多模态 SAE 不仅可作为可解释性工具,还可作为审计、引导和控制 MLLM 行为的机制,以实现更安全、更具能力的生成。MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior, shows that multimodal SAEs can serve not only as interpretability tools, but as mechanisms for auditing, steering, and controlling MLLMs behavior toward safer and more capable generations.

Self-Supervised Visual On-Policy Distillation
自监督视觉 On-Policy Distillation
arXiv:2608.14144 多模态 方法 OA · 绿色 被引 7 · S2

提出 Self-Supervised Visual On-Policy Distillation(S²VOPD),一种简单有效的方法,通过非对称增强视图构建 on-policy 学习信号,系统地探索了视觉增强的广阔设计空间,并发现非对称性至关重要。Self-Supervised Visual On-Policy Distillation (S$^2$VOPD), a simple yet effective method that constructs on-policy learning signals from asymmetric augmented views, systematically explores a broad design space of visual augmentations and uncover that asymmetry matters.

UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
UniProbe:一种利用多结构内部表示的可学习 token 级 VLM 幻觉检测器
arXiv:2608.10835 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

提出 UniProbe,一个轻量、统一、可学习的检测器,通过单次前向传播建模冻结 LVLM 的异构计算 trace,在 token 级和物体级幻觉检测上达到 SOTA。UniProbe is introduced, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass and achieves state-of-the-art token-level and object-hallucination detection.

GRNEdit: Efficient General Video Editing from a New Binary-Evidence Perspective in Generative Refinement Networks
GRNEdit:基于生成式 Refinement 网络的新二元证据视角下的高效通用视频编辑
arXiv:2608.16328 多模态 方法 OA · 绿色 被引 2 · S2

GRNEdit 是一个轻量级的两阶段指令驱动通用视频编辑框架,性能优于多个 14B 开源编辑器,同时其 8B 模型与领先的开源编辑器表现相当。GRNEdit, a lightweight two-stage framework for instruction-based general video editing that outperforms multiple 14B open-source editors, while the 8B model performs on par with leading open-source editors.

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation
TRACE-Bench:多参考图像生成的分解与诊断
arXiv:2608.16765 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

认识到多样化的多参考任务共享一组共同的原子操作,本文形式化了四个算子:Anchor、Disentangle、Apply 和 Compose,并构建了 TRACE-Bench,包含约 1,600 个跨 slot 数量 1–8 的评估用例。Recognizing that diverse multi-reference tasks share a common set of atomic operations, this work formalizes four operators: Anchor, Disentangle, Disentangle, Apply, and Compose, and constructs TRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1--8.

WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations
WorldRover:一个用于世界探索的、可扩展的、带有丰富标注的合成视频数据引擎
arXiv:2608.15659 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

WorldRover 将长视野世界探索转化为可扩展的数据生成问题,为需要在可探索世界中构建、维护并重访一致表征的模型提供监督信号。WorldRover turns long-horizon world exploration into a scalable data-generation problem, providing supervision for models that must build, maintain, and revisit coherent representations of an explorable world.