论文

Pointer sentinel-LSTM 模型在 Penn Treebank 上以远少于标准 softmax LSTM 的参数量达到 SOTA 语言建模性能，并开源了 WikiText 语料库The pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank while using far fewer parameters than a standard softmax LSTM and the freely available WikiText corpus is introduced.

这项系统性研究在数十项语言理解任务上比较了预训练目标、架构、无标注数据集、迁移方法及其他因素，并在涵盖摘要、问答、文本分类等的许多基准上取得了 SOTA 结果。This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.

名为 PaLM 的 540 亿参数、密集激活的 Transformer 语言模型取得了突破性性能，在一系列多步推理任务上超越了微调后的 SOTA，并在最近发布的 BIG-bench 基准上超越了人类平均水平。A 540-billion parameter, densely activated, Transformer language model, which is called PaLM achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark.

论文证明，使用标注数据进行微调，并允许模型查询外部知识源，能够在安全性和事实性这两个关键挑战上带来显著提升。It is demonstrated that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.

本文讨论了一种被称为大语言模型涌现能力的不可预测现象——若某项能力在小模型中不存在而在大模型中存在，则称为涌现。This paper discusses an unpredictable phenomenon that is referred to as emergent abilities of large language models, an ability to be emergent if it is not present in smaller models but is present in larger models.

一个仅依赖字符级输入的简单神经语言模型，仅从字符即可编码语义和正字法信息，表明在许多语言中，字符输入足以完成语言建模。A simple neural language model that relies only on character-level inputs that is able to encode, from characters only, both semantic and orthographic information and suggests that on many languages, character inputs are sufficient for language modeling.