论文

一个能够将任意自然语言任务轻松映射为人类可读 prompt 形式的系统，并在覆盖多种任务的多任务混合数据上对预训练 encoder-decoder 模型进行微调。A system for easily mapping any natural language tasks into a human-readable prompted form and fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks.

文章论证了 Transformers 可作为医学图像分割任务的强大编码器，并通过与 U-Net 结合，恢复了局部空间信息以增强更精细的细节。It is argued that Transformers can serve as strong encoders for medical image segmentation tasks, with the combination of U-Net to enhance finer details by recovering localized spatial information.

研究表明，利用变分方法最新进展的深度生成模型与近似贝叶斯推断能够带来显著提升，使生成式方法在半监督学习上极具竞争力。It is shown that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.

本工作提出一种通过引入自注意力来提取可解释句子嵌入的新模型，使用一个二维矩阵表示嵌入，其中矩阵的每一行关注句子的不同部分。A new model for extracting an interpretable sentence embedding by introducing self-attention is proposed, which uses a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.

本工作指出，这些局部 patch 内部的注意力同样是构建高性能视觉 Transformer 的关键，并探索了一种新架构，即 Transformer iN Transformer (TNT)。It is pointed out that the attention inside these local patches are also essential for building visual transformers with high performance and a new architecture, namely, Transformer iN Transformer (TNT), is explored.

论文表明，使用 Stanford Natural Language Inference 数据集有监督训练的通用句子表示，在广泛的迁移任务上能持续优于 SkipThought vectors 等无监督方法。It is shown how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks.

本文提出一种权重裁剪的替代方案：对 critic 相对于其输入的梯度范数施加惩罚。其性能优于标准 WGAN，能以几乎无需调参的方式稳定训练多种 GAN 架构。This work proposes an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input, which performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning.

当将监督规模扩展到 680,000 小时的多语言、多任务数据时，所得到的模型在标准 benchmark 上泛化良好，在 zero-shot transfer 设置下常可与此前全监督方法的结果相当，且无需任何微调。When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning.