研究库 论文知识库
Papers · organized/paper_cards

论文

417 张论文卡片 · 多模态 · OA 绿色

开放获取 全部 绿色 · 1640
WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models
WorldCycle:面向长视野视频世界模型的自可验证强化学习
arXiv:2608.04964 多模态 方法 OA · 绿色 被引 3 · S2

提出 WorldCycle,一种自验证 RL 框架,从普通动作序列中构建闭合动作循环及其重复执行,并优化两个互补奖励:空间闭合奖励(强制镜像的前向与反向片段之间的对称性)以及时间一致性奖励(对齐多次循环执行间的状态)。WorldCycle is introduced, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions.

BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation
BridgeVLA++:用于 3D 操作的、数据高效、可泛化且具备记忆增强的 Vision-Language-Action 框架
arXiv:2608.05042 多模态 应用落地 OA · 绿色 被引 4 · S2

在 BridgeVLA 基础上开发 BridgeVLA++,引入统一的时空记忆架构,建模持久化的空间上下文与时间交互历史,使其可在保留 BridgeVLA 数据效率与泛化能力的同时对观测历史进行推理。BridgeVLA++ is developed by equipping BridgeVLA with a unified spatio-temporal memory architecture that models persistent spatial context and temporal interaction history that can reason over observation histories while preserving BridgeVLA's data efficiency and generalization capabilities.

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
Distill Where You Fail:通过自适应教师指导恢复负 RL 群体的学习信号
arXiv:2608.00782 多模态 方法 OA · 绿色 被引 8 · S2

提出 RSTG(Recovering Learning Signals via Adaptive Teacher Guidance),在最关键的位置选择性地、精确地施加蒸馏,显著优于朴素的 GRPO+OPD。RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most, and substantially outperforms naive GRPO+OPD.

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities
AVE-Compass:迈向音视频编辑能力的整体评估
arXiv:2607.24821 多模态 评测集 OA · 绿色 被引 1 · S2

提出 AVE-Agent,一种模块化 agent 框架,将复杂指令分解为相互依赖的子任务,并通过自我反思与评估器反馈迭代改进编辑结果,在联合编辑中提升指令执行、保真度保持以及音视频对齐,同时保持有竞争力的感知质量。AVE-Agent is proposed, a modular agent framework that decomposes complex instructions into dependent subtasks and iteratively improves editing results through self-reflection and evaluator feedback, and improves instruction execution, Fidelity Preserving, and audio-visual alignment in joint editing while maintaining competitive perceptual quality.

Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning
Consistency-Driven Co-Evolution:面向自监督跨表征学习的一致性驱动协同进化
arXiv:2608.04926 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

提出 CoCoEvolve,通过定义显式的一一对应关系,并利用表征间的一致性优化模型,在无需额外标注的前提下提升图表、表格与代码表征之间的跨模态一致性。This work introduces CoCoEvolve, a method to improve consistency across chart, table, and code representations by defining explicit one-to-one correspondences and optimizing models using agreement between representations, without additional annotations.

DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attack
DRIFT:利用对抗性 patch 攻击偏转流匹配 VLA 的去噪轨迹
arXiv:2608.03207 多模态 方法 OA · 绿色 被引 1 · S2

提出 DRIFT(Denoising Redirection via Input perturbation of the Flow-matching Trajectory),一种测试期通用对抗补丁,贴在机器人夹爪上,针对现成策略的去噪速度场进行攻击。This work introduces DRIFT (Denoising Redirection via Input perturbation of the Flow-matching Trajectory), a test-time universal adversarial patch placed on the robot's gripper that attacks the denoising velocity field of an off-the-shelf policy.

TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
TransUNet: Transformers 作为医学图像分割的强大编码器
arXiv:2102.04306 多模态 方法 OA · 绿色 被引 6438 · S2

文章论证了 Transformers 可作为医学图像分割任务的强大编码器,并通过与 U-Net 结合,恢复了局部空间信息以增强更精细的细节。It is argued that Transformers can serve as strong encoders for medical image segmentation tasks, with the combination of U-Net to enhance finer details by recovering localized spatial information.

SegFormer: Simple and Efficient Design for Semantic Segmentation with\n Transformers
SegFormer: 基于 Transformers 的简单高效语义分割设计
arXiv:2105.15203 多模态 方法 OA · 绿色 被引 9713 · S2

文章提出了 SegFormer,一个简单高效且强大的语义分割框架,将 Transformers 与轻量级 MLP 解码器统一,并在 Cityscapes-C 上展示了出色的零样本鲁棒性。SegFormer is presented, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders and shows excellent zero-shot robustness on Cityscapes-C.

VisualBERT: A Simple and Performant Baseline for Vision and Language
VisualBERT: 一个简单且性能优异的视觉与语言基线模型
arXiv:1908.03557 多模态 方法 OA · 绿色 被引 2413 · S2

分析表明,VisualBERT 能够在无任何显式监督的情况下将语言元素对应到图像区域,甚至对句法关系敏感,例如能够跟踪动词与其论元对应图像区域之间的关联。Analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
Swin-Unet: 类 U-Net 的纯 Transformer 医学图像分割网络
arXiv:2105.05537 多模态 方法 OA · 绿色 被引 5841 · S2

在输入和输出直接进行 4 倍下采样与上采样的设定下,实验表明,基于纯 Transformer 的 U 形编码器-解码器网络优于完全卷积或 Transformer 与卷积相结合的方法。Under the direct down-sampling and up-sampled of the inputs and outputs by 4x, experiments demonstrate that the pure Transformer-based U-shaped Encoder-Decoder network outperforms those methods with full Convolution or the combination of transformer and convolution.

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
BLIP: 面向统一视觉-语言理解与生成的引导式语言-图像预训练
arXiv:2201.12086 多模态 方法 OA · 绿色 被引 7458 · S2

BLIP 通过引导式 caption 方式有效利用含噪网络数据,由 captioner 生成合成 caption,并由 filter 去除噪声样本;在以零样本方式直接迁移到视频-语言任务时,展现出强大的泛化能力。BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones, and demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner.

A deep learning architecture for temporal sleep stage classification\n using multivariate and multimodal time series
基于多变量多模态时间序列的深度学习睡眠阶段时序分类架构
arXiv:1707.03321 多模态 方法 OA · 绿色 被引 615 · OpenAlex

本文提出了首个用于睡眠阶段分类的深度学习方法,无需计算频谱图或提取手工特征即可端到端学习,利用了全部多变量多模态 PSG 信号(EEG、EMG、EOG),并能利用每个 30 秒窗口数据的时序上下文。This work introduces here the first deep learning approach for sleep stage classification that learns end-to-end without computing spectrograms or extracting handcrafted features, that exploits all multivariate and multimodal polysomnography (PSG) signals (EEG, EMG, and EOG), and that can exploit the temporal context of each 30-s window of data.

SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models
SIGNPOST-Bench:面向多模态大语言模型中文本-视觉冲突消解的基准评测
arXiv:2608.04244 多模态 评测集 OA · 绿色 被引 1 · S2

这些结果将视觉地理定位确立为场景文本仲裁的连续诊断手段,并提供了一个受控框架,用于评估 MLLMs 如何解决冲突的多模态证据。These results establish visual geolocation as a continuous diagnostic of scene-text arbitration and provide a controlled framework for evaluating how MLLMs resolve conflicting multimodal evidence.

PaDoc: Layout-Grounded Parallel Decoding for Document Parsing
PaDoc:面向文档解析的布局引导并行解码
arXiv:2608.06146 多模态 方法 OA · 绿色 被引 1 · S2

PaDoc 是一个布局驱动的解析器,将预测布局视为共享页面表示上的分支结构,在五个并发级别下均为最快的 end-to-end 解析器。PaDoc is proposed, a layout-grounded parser that treats the predicted layout as a branching structure over a shared page representation that is the fastest end-to-end parser at five concurrency levels and is the fastest end-to-end parser at five concurrency levels.

EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal
EffectLearner:面向真实世界视频物体移除的世界感知物体-效果推理
arXiv:2608.05565 多模态 观点 OA · 绿色 被引 0 · S2 + OpenAlex

EffectLearner 是一个语义推理增强框架,结合基于 VLM 的 Object-Effect Reasoner 与基于 DiT 的 Video Eraser,在 EffectWorld-Eval 和具有挑战性的 EffectWorld-Wild 上均取得明显优势,证明其能在复杂真实场景中实现高质量的视频物体擦除。EffectLearner is proposed, a semantic-reasoning-enhanced framework that combines a VLM-based Object-Effect Reasoner with a DiT-based Video Eraser that achieves clear advantages on both EffectWorld-Eval and the challenging EffectWorld-Wild, demonstrating its ability to deliver high-quality video object removal in complex real-world scenes.

World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation
World-to-Wrist:面向细粒度机器人操作的任务条件化未来腕部建模
arXiv:2608.05369 多模态 方法 OA · 绿色 被引 1 · S2

World-to-Wrist VLA (W2-VLA) 是一个用于细粒度机器人操作的 VLA 模型,具备任务条件下的未来腕部建模;W2-CoT 是一个合成流水线,生成描述操作进度、物理过渡线索和腕部局部证据的结构化标注。World-to-Wrist VLA (W2-VLA), a VLA model for fine-grained robot manipulation with task-conditioned future wrist modeling, and W2-CoT, a synthesis pipeline that produces structured annotations describing manipulation progress, physical transition cues, and wrist-local evidence.

SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding
SmartMage:面向 3D 场景理解的动态模态编排
arXiv:2608.05137 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

SmartMage 是一个统一的 MLLM,动态调度异构模态以实现语义感知的 3D 场景理解,在五个 3D 场景理解基准上达到 SOTA,并在仅 RGB 视频理解基准上取得具有竞争力的结果。SmartMage is proposed, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding and achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks.

Invisible Shortcuts: Why Vision Encoders Know Your Camera
Invisible Shortcuts:视觉 Encoder 为何知道你的相机
arXiv:2608.05424 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

识别出像素级嵌入的不可见元数据痕迹,表明大规模语义监督(无论是类别标签还是亿级规模的 caption)在预训练中会自然引发元数据-语义相关性,导致模型将低层信号转化为预测特征。Invisible metadata traces embedded at the pixel level are identified, suggesting that large-scale semantic supervision, whether through categorical labels or billion-scale captions, naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features.

MASS: Multiplayer World Models with Authoritative Shared State
MASS:基于权威共享状态的多智能体世界模型
arXiv:2608.06257 多模态 方法 OA · 绿色 被引 5 · S2

结果表明,显式且权威的状态建模为可扩展、一致的多智能体世界仿真提供了可行基础。The results show that explicit, authoritative state modeling provides a practical foundation for scalable and consistent multi-agent world simulation.

Transformer in Transformer
Transformer in Transformer
arXiv:2103.00112 多模态 方法 OA · 绿色 被引 2308 · S2

本工作指出,这些局部 patch 内部的注意力同样是构建高性能视觉 Transformer 的关键,并探索了一种新架构,即 Transformer iN Transformer (TNT)。It is pointed out that the attention inside these local patches are also essential for building visual transformers with high performance and a new architecture, namely, Transformer iN Transformer (TNT), is explored.

KVAE: Family of Tokenizers for Multimodal Generative Models
KVAE:面向多模态生成模型的 tokenizer 家族
arXiv:2608.05798 多模态 观点 OA · 绿色 被引 0 · S2 + OpenAlex

在客观与主观指标上的重建和生成结果匹配或超越前沿开源 tokenizer,包括 Wan-2.2、HunyuanVideo-1.5、FLUX.2、MovieGen、StableAudio 和 MMAudio 的 VAE。It is demonstrated that reconstruction and generation results on objective and subjective metrics matches or surpasses frontier opensource tokenizers, such as VAEs from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio.

Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
Weights 还是 Skills?机器人学习方法综述:从预测动作的权重到自主编写技能的机器人
arXiv:2608.01851 多模态 综述 OA · 绿色 被引 0 · S2 + OpenAlex

该综述围绕"权重"与"技能"这一轴线组织领域,梳理了互补的"技能"一极——从无监督强化学习的技能发现,到大语言模型的技能库——并指出"skill"一词至少存在五种不同含义。This survey organises the field around that axis of weights versus skills, and maps the complementary"skills"pole, from unsupervised reinforcement-learning skill discovery to large-language-model skill libraries, and shows that the word "skill" is used in at least five distinct senses.

Representation Learning with Contrastive Predictive Coding
基于对比预测编码的表征学习
arXiv:1807.03748 多模态 方法 OA · 绿色 被引 14753 · S2

本文提出了一种通用的无监督学习方法——对比预测编码(Contrastive Predictive Coding),用于从高维数据中提取有用的表征,并在语音、图像、文本和 3D 环境中的强化学习四个不同领域取得了出色的性能。This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

SimWAM: A Simple World Action Model for End-to-End Autonomous Driving
SimWAM:用于端到端自动驾驶的简单 World Action Model
arXiv:2608.07468 多模态 方法 OA · 绿色 被引 5 · S2

提出 SimWAM,一种简单而有效的 WAM,仅将未来视频预测用作训练时的监督信号,并通过联合 flow matching 协同训练一个预训练视频专家与一个轻量动作专家。SimWAM is presented, a simple yet effective WAM that leverages future-video prediction solely as a training-time supervision signal, and co-trains a pretrained video expert and a lightweight action expert with joint flow matching.

Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors
往返一致性:双向扩散模型可预测自身的 rollout 误差
arXiv:2608.00675 多模态 应用落地 OA · 绿色 被引 1 · S2

往返一致性将可逆性转化为生成式模型一种实用的可信信号;双向训练带来负成本,在两个方向上均优于单向专家模型;其中反向还可作为快速的逆问题求解器。Round-trip consistency turns reversibility into a practical trust signal for generative models, and Bidirectional training comes at negative cost, beating direction specialists in both directions, and the backward direction doubles as a fast inverse solver.

Diffusion-Convolutional Neural Networks
扩散卷积神经网络
arXiv:1511.02136 多模态 方法 OA · 绿色 被引 1390 · S2

通过引入扩散卷积运算,本文展示了如何从图结构数据中学习基于扩散的表示,并将其作为节点分类的有效基础。Through the introduction of a diffusion-convolution operation, it is shown how diffusion-based representations can be learned from graph-structured data and used as an effective basis for node classification.

Florence: A New Foundation Model for Computer Vision
Florence:面向计算机视觉的新基础模型
arXiv:2111.11432 多模态 方法 OA · 绿色 被引 1168 · S2

本文提出新的计算机视觉基础模型 Florence,通过融入来自 Web 规模图文数据的通用视觉-语言表示,将表征范围从粗粒度(场景)扩展到细粒度、从静态(图像)扩展到动态(视频)、从 RGB 扩展到多种模态(描述、深度等)。This work introduces a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine, from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth), by incorporating universal visual-language representations from Web-scale image-text data.

Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks
使用正则化深度 LSTM 网络进行基于骨骼动作识别的共现特征学习
arXiv:1603.07772 多模态 应用落地 OA · 绿色 被引 930 · S2

本文在每个时间步以骨骼作为输入,引入一种新的正则化方案来学习骨骼关节的共现特征,并提出一种同时作用于 LSTM 神经元门、单元和输出响应的新型 dropout 算法。This work takes the skeleton as the input at each time slot and introduces a novel regularization scheme to learn the co-occurrence features of skeleton joints, and proposes a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons.

Multi-Agent Forensic Reasoning for Generalizable Deepfake Video Detection
多智能体取证推理用于可泛化的深度伪造视频检测
arXiv:2608.06865 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

提出 FaceVid-Forensics-100K,一个大规模深伪视频数据集,包含 100,000 个视频,涵盖 33 种合成方法,覆盖换脸、表情重演与全脸合成;同时提出一个多智能体取证推理框架,由四个领域专家 Agent 分别从四个角度独立分析伪造线索。FaceVid-Forensics-100K is introduced, a large-scale deepfake video dataset comprising 100,000 videos and spanning 33 synthesis methods across face swapping, face reenactment, and entire-face synthesis, and a multi-agent forensic reasoning framework that employs four specialized domain-expert agents to independently analyze forgery cues from four perspectives.

Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Control
Enfold:将世界模型想象折叠进预测表征以实现超高效具身控制
arXiv:2607.26657 多模态 方法 OA · 绿色 被引 4 · S2

本文提出 Enfold,将构建未来的计算转移到由当前视觉上下文和语言指令预测出的表征中,并把世界生成器重塑为预测控制表征的来源,前提是其内部结构可被折叠(enfold)到当下。This work presents Enfold, which transfers this computation that constructs a future into a representation predicted from the current visual context and language instruction, and recast a world generator as a source of predictive control representations if its internal structure can be enfolded into the present.

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models
CLIP-CC-Bench:评估视频-语言模型中的段落级视频描述
arXiv:2608.04302 多模态 评测集 OA · 绿色 被引 0 · S2 + OpenAlex

CLIP-CC-Bench 为长视频描述提供了一个实用的评估框架,填补了现有短片段和仅 QA 基准的空白,并通过评分者间一致性(inter-judge agreement)与 bootstrap 排序稳定性量化该协议的内可靠性。CLIP-CC-Bench provides a practical evaluation framework for long-form video description, filling a gap left by existing short-clip and QA-only benchmarks and quantifying the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability.

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues
DuplexGen:自适应合成人-AI 轮替对话
arXiv:2607.26178 多模态 方法 OA · 绿色 被引 1 · S2

本文提出 DuplexGen,一个通过少量 slot 级人类偏好标注对 LLM 预测进行校准从而生成具有场景自适应轮换特性的对话框架;结果表明,使轮换合成具备场景特异性的关键是人类校准,而非单纯的语料规模或提示设计。DuplexGen is introduced, a framework for generating dialogues with scenario-adaptive turn-taking by calibrating LLM predictions against a small set of slot-level human preference annotations, and results show that human calibration, not corpus scale or prompt design alone, is what allows turn-taking synthesis to be scenario-specific.

RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance
RynnValue:通过时间距离扩展机器人价值基础模型
arXiv:2608.09853 多模态 方法 OA · 绿色 被引 7 · S2

RynnValue 是一个用于机器人操控的开源价值基础模型,用时间距离(即从观测到语言指定目标的有向 cost-to-go)替代任务内部锚点,将时间距离确立为通用机器人策略的可扩展监督目标和实用奖励接口。RynnValue, an open-source value foundation model for robotic manipulation that replaces task-internal anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal, establishes temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.

Scaling Inherently Interpretable Language Models
规模化天生可解释的语言模型
arXiv:2608.07594 多模态 方法 OA · 绿色 被引 3 · S2

Steerling-8B 在与训练计算量多 2–16 倍的开源同侪模型对比中仍保持竞争力,表明存在一种不同的可扩展范式:可解释性可以被设计进训练过程中,并随规模放大而提升。Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.

What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems
下一步编辑什么:对话系统中视觉对齐的图像编辑后续建议
arXiv:2608.07565 多模态 方法 OA · 绿色 被引 2 · S2

一个三阶段框架,用于减少建议编辑与当前图像之间的视觉不一致性,并将推荐 CTR、图像带走率以及用户平均对话轮次分别显著提升 39.90%(所有 p<0.05)。A three-stage framework to reduce visual inconsistencies between suggested edits and the current image, which significantly improves recommendation CTR, image take-away rate, and average conversation turns per user by 39.90% (all p<0.05).

Ego-OSCAR: Egocentric Open source Stereo CAptuRe System
Ego-OSCAR:自我中心开源立体捕获系统
arXiv:2608.08285 多模态 方法 OA · 绿色 被引 0 · S2 + OpenAlex

本文提出 Ego-OSCAR,一种用于野外自我中心数据采集的开源硬件、低成本、头戴式立体惯性采集设备,旨在成为众包自我中心采集中最廉价且可辩护的载体,降低任何团队大规模采集自我中心数据的启动门槛。Ego-OSCAR is presented, an open-hardware, low-cost, head-mounted stereo-inertial capture device for egocentric data collection in the wild that aims to be the cheapest defensible substrate for crowdsourced egocentric capture, and to lower the activation energy for any team that wants to collect egocentric data at scale.