论文

m-RNN 模型直接对给定先前词语和图像条件下生成下一个词的概率分布建模，相较于直接优化排序目标函数进行检索的 SOTA 方法，取得了显著的性能提升。The m-RNN model directly models the probability distribution of generating a word given previous words and an image, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.

提出极简的 VLP 模型 Vision-and-Language Transformer (ViLT)，其一体化设计将视觉输入处理大幅简化为与文本输入相同的无卷积方式；ViLT 比此前的 VLP 模型快达数十倍，同时下游任务性能具有竞争力甚至更优。A minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that the authors process textual inputs, showing that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance.

Contrastive Captioner (CoCa) 采用极简设计，对图文编码器-解码器基础模型联合使用对比损失与字幕损失进行预训练，从而兼具 CLIP 等对比方法与 SimVLM 等生成方法的能力。Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM.

本文提出 MiniGPT-4，通过一个投影层将冻结的视觉编码器与冻结的先进 LLM Vicuna 对齐，发现将视觉特征与先进大语言模型恰当对齐可获得类似 GPT-4 所展现的多种先进多模态能力。MiniGPT-4 is presented, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer to uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by G PT-4.

本文表明视频掩码自编码器（VideoMAE）是自监督视频预训练（SSVP）的数据高效学习器，并受近期 ImageMAE 启发，提出采用极高掩码比例的定制化视频管状掩码策略。This paper shows that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP), and proposes customized video tube masking with an extremely high ratio, inspired by the recent ImageMAE.

本文提出基于双向 Mamba 块（Vim）的通用视觉 backbone，通过位置嵌入标记图像序列，并利用双向 state space model 压缩视觉表征，具有成为下一代视觉基础模型 backbone 的巨大潜力。This paper proposes a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models and has great potential to be the next-generation backbone for vision foundation models.

本文提出 embodied language model，将真实世界连续传感器模态直接融入语言模型，从而建立词语与感知之间的联系，实现真实世界中的通用推理。This work proposes embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts to enable general inference in the real world.

本文提出极简的预训练框架 SimVLM，在广泛的判别式与生成式视觉-语言基准上显著超越既往预训练方法并取得新 SOTA，包括 VQA、NLVR2 以及图像描述任务。This work presents a minimalist pretraining framework, named SimVLM, which significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA, NLVR2, and image captioning tasks.

文章论证了 Transformers 可作为医学图像分割任务的强大编码器，并通过与 U-Net 结合，恢复了局部空间信息以增强更精细的细节。It is argued that Transformers can serve as strong encoders for medical image segmentation tasks, with the combination of U-Net to enhance finer details by recovering localized spatial information.

文章提出了 SegFormer，一个简单高效且强大的语义分割框架，将 Transformers 与轻量级 MLP 解码器统一，并在 Cityscapes-C 上展示了出色的零样本鲁棒性。SegFormer is presented, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders and shows excellent zero-shot robustness on Cityscapes-C.

分析表明，VisualBERT 能够在无任何显式监督的情况下将语言元素对应到图像区域，甚至对句法关系敏感，例如能够跟踪动词与其论元对应图像区域之间的关联。Analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

文章介绍了一种自监督视觉表征模型 BEiT（Bidirectional Encoder representation from Image Transformers），在图像分类和语义分割上的结果表明，该模型取得了与先前预训练方法相当的竞争性结果。A self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers, is introduced, and results on image classification and semantic segmentation show that the model achieves competitive results with previous pre-training methods.

在输入和输出直接进行 4 倍下采样与上采样的设定下，实验表明，基于纯 Transformer 的 U 形编码器-解码器网络优于完全卷积或 Transformer 与卷积相结合的方法。Under the direct down-sampling and up-sampled of the inputs and outputs by 4x, experiments demonstrate that the pure Transformer-based U-shaped Encoder-Decoder network outperforms those methods with full Convolution or the combination of transformer and convolution.

BLIP 通过引导式 caption 方式有效利用含噪网络数据，由 captioner 生成合成 caption，并由 filter 去除噪声样本；在以零样本方式直接迁移到视频-语言任务时，展现出强大的泛化能力。BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones, and demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner.

本文提出了首个用于睡眠阶段分类的深度学习方法，无需计算频谱图或提取手工特征即可端到端学习，利用了全部多变量多模态 PSG 信号（EEG、EMG、EOG），并能利用每个 30 秒窗口数据的时序上下文。This work introduces here the first deep learning approach for sleep stage classification that learns end-to-end without computing spectrograms or extracting handcrafted features, that exploits all multivariate and multimodal polysomnography (PSG) signals (EEG, EMG, and EOG), and that can exploit the temporal context of each 30-s window of data.

本文对度量学习文献进行了系统综述，阐述了每种方法的优缺点，并介绍了近期涌现的一系列强大替代方法，包括非线性度量学习、相似性学习与局部度量学习。A systematic review of the metric learning literature is proposed, highlighting the pros and cons of each approach and presenting a wide range of methods that have recently emerged as powerful alternatives, including nonlinear metric learning, similarity learning and local metric learning.

本文提出一种多任务深度学习的原则性方法，通过考虑各任务的同方差不确定性来加权多个损失函数，从而在分类与回归场景下同时学习具有不同单位或尺度的多种量。A principled approach to multi-task deep learning is proposed which weighs multiple loss functions by considering the homoscedastic uncertainty of each task, allowing us to simultaneously learn various quantities with different units or scales in both classification and regression settings.

本文通过深度多模态嵌入视觉与自然语言数据，提出了一种用于图文双向检索的模型，并引入结构化的最大间隔目标，使该模型能够显式地跨模态关联片段。This work introduces a model for bidirectional retrieval of images and sentences through a deep, multi-modal embedding of visual and natural language data and introduces a structured max-margin objective that allows this model to explicitly associate fragments across modalities.

从算法建模、应用和理论分析角度对 MTL 的综述，给出了 MTL 的定义，并将不同 MTL 算法分为五类：特征学习方法、低秩方法、任务聚类方法、任务关系学习方法和分解方法A survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses, which gives a definition of MTL and classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach,task relation learning approach and decomposition approach.

描述了一种无监督学习通用分布式句子编码器的方法，利用书籍文本的连续性，训练编码器-解码器模型以重建编码段落的周围句子。The approach for unsupervised learning of a generic, distributed sentence encoder is described, using the continuity of text from books to train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage.

论文表明，使用 Stanford Natural Language Inference 数据集有监督训练的通用句子表示，在广泛的迁移任务上能持续优于 SkipThought vectors 等无监督方法。It is shown how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks.