论文

一个能够将任意自然语言任务轻松映射为人类可读 prompt 形式的系统，并在覆盖多种任务的多任务混合数据上对预训练 encoder-decoder 模型进行微调。A system for easily mapping any natural language tasks into a human-readable prompted form and fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks.

研究表明，利用变分方法最新进展的深度生成模型与近似贝叶斯推断能够带来显著提升，使生成式方法在半监督学习上极具竞争力。It is shown that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.

本工作提出一种通过引入自注意力来提取可解释句子嵌入的新模型，使用一个二维矩阵表示嵌入，其中矩阵的每一行关注句子的不同部分。A new model for extracting an interpretable sentence embedding by introducing self-attention is proposed, which uses a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.

本工作指出，这些局部 patch 内部的注意力同样是构建高性能视觉 Transformer 的关键，并探索了一种新架构，即 Transformer iN Transformer (TNT)。It is pointed out that the attention inside these local patches are also essential for building visual transformers with high performance and a new architecture, namely, Transformer iN Transformer (TNT), is explored.

当将监督规模扩展到 680,000 小时的多语言、多任务数据时，所得到的模型在标准 benchmark 上泛化良好，在 zero-shot transfer 设置下常可与此前全监督方法的结果相当，且无需任何微调。When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning.