论文

本文提出了一种通用的无监督学习方法——对比预测编码（Contrastive Predictive Coding），用于从高维数据中提取有用的表征，并在语音、图像、文本和 3D 环境中的强化学习四个不同领域取得了出色的性能。This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

本文提出了一种双时间尺度更新规则（TTUR），用于在任意 GAN 损失函数下使用 SGD 训练 GAN，并引入了 Frechet Inception Distance（FID），相比 Inception Score 能更好地捕捉生成图像与真实图像之间的相似性。This work proposes a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions and introduces the "Frechet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score.

名为 PaLM 的 540 亿参数、密集激活的 Transformer 语言模型取得了突破性性能，在一系列多步推理任务上超越了微调后的 SOTA，并在最近发布的 BIG-bench 基准上超越了人类平均水平。A 540-billion parameter, densely activated, Transformer language model, which is called PaLM achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark.

本文提出了一种通过创建在所有边缘设备之间全局共享的小型数据子集来改进非独立同分布数据训练的策略，并表明在 CIFAR-10 数据集上，仅共享 5% 的全局数据即可将准确率提升 30%。This work presents a strategy to improve training on non-IID data by creating a small subset of data which is globally shared between all the edge devices, and shows that accuracy can be increased by 30% for the CIFAR-10 dataset with only 5% globally shared data.

采用迭代的在线训练模式，按周节奏用新的人类反馈数据更新偏好模型与 RL 策略，并发现 RL 奖励与策略相对其初始化的 KL 散度平方根之间近似呈线性关系。An iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, and a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization is identified.

研究表明，APE 生成的提示词既可引导模型趋向真实性和/或信息量，也可通过将其前置拼接到标准上下文学习提示词之前来提升少样本学习性能。It is shown that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness, as well as to improve few-shot learning performance by simply prepending them to standard in-context learning prompts.

本文提出 Atlas，一个经过精心设计并预训练的检索增强大语言模型，能以极少训练样例学习知识密集型任务，并研究了文档索引内容的影响，表明该索引可便捷地更新。This work presents Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples, and studies the impact of the content of the document index, showing that it can easily be updated.

本文通过深度多模态嵌入视觉与自然语言数据，提出了一种用于图文双向检索的模型，并引入结构化的最大间隔目标，使该模型能够显式地跨模态关联片段。This work introduces a model for bidirectional retrieval of images and sentences through a deep, multi-modal embedding of visual and natural language data and introduces a structured max-margin objective that allows this model to explicitly associate fragments across modalities.

论文证明，使用标注数据进行微调，并允许模型查询外部知识源，能够在安全性和事实性这两个关键挑战上带来显著提升。It is demonstrated that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.

本文利用源自 CNN 的同一基础信息重新审视初始搜索与重排序两个检索阶段，显著改进了现有基于 CNN 的识别流水线。This work revisits both retrieval stages, namely initial search and re-ranking, by employing the same primitive information derived from the CNN, and significantly improves existing CNN-based recognition pipeline.

通过引入扩散卷积运算，本文展示了如何从图结构数据中学习基于扩散的表示，并将其作为节点分类的有效基础。Through the introduction of a diffusion-convolution operation, it is shown how diffusion-based representations can be learned from graph-structured data and used as an effective basis for node classification.

Manifold Mixup 在监督学习、对单步对抗攻击的鲁棒性、半监督学习以及留出样本的负对数似然（NLL）上，相较强基线均取得了大幅提升。Manifold Mixup achieves large improvements over strong baselines in supervised learning, robustness to single-step adversarial attacks, semi-supervised learning, and Negative Log-Likelihood on held out samples.

本文介绍了 CodeXGLUE，一个基准数据集，旨在推动面向程序理解与生成的机器学习研究，涵盖 14 个数据集上的 10 项任务，并提供模型评估与比较的平台。This paper introduces CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation that includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison.

本文提出新的计算机视觉基础模型 Florence，通过融入来自 Web 规模图文数据的通用视觉-语言表示，将表征范围从粗粒度（场景）扩展到细粒度、从静态（图像）扩展到动态（视频）、从 RGB 扩展到多种模态（描述、深度等）。This work introduces a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine, from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth), by incorporating universal visual-language representations from Web-scale image-text data.

于 2018 年 8 月 28 日中午 12:15 在 Pettit 微电子研究中心 102 A/B 室进行报告。Presented on August 28, 2018 at 12:15 p.m. in the Pettit Microelectronics Research Center, Room 102 A/B.

本文提出了不变风险最小化（IRM），一种用于在多个训练分布上估计不变相关性的学习范式，并展示了 IRM 学到的不变性如何与数据的因果结构相关，从而实现分布外泛化。This work introduces Invariant Risk Minimization, a learning paradigm to estimate invariant correlations across multiple training distributions and shows how the invariances learned by IRM relate to the causal structures governing the data and enable out-of-distribution generalization.

本文在每个时间步以骨骼作为输入，引入一种新的正则化方案来学习骨骼关节的共现特征，并提出一种同时作用于 LSTM 神经元门、单元和输出响应的新型 dropout 算法。This work takes the skeleton as the input at each time slot and introduces a novel regularization scheme to learn the co-occurrence features of skeleton joints, and proposes a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons.

研究表明 BigBird 是序列函数的通用逼近器，且具备图灵完备性，从而保留了二次全注意力模型的这些性质。It is shown that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model.

本文讨论了一种被称为大语言模型涌现能力的不可预测现象——若某项能力在小模型中不存在而在大模型中存在，则称为涌现。This paper discusses an unpredictable phenomenon that is referred to as emergent abilities of large language models, an ability to be emergent if it is not present in smaller models but is present in larger models.

本工作研究图结构输入的特征学习技术，并在程序验证任务上取得 SOTA 性能，该任务需将子图与抽象数据结构进行匹配。This work studies feature learning techniques for graph-structured inputs and achieves state-of-the-art performance on a problem from program verification, in which subgraphs need to be matched to abstract data structures.

本文针对 OOD 检测领域的近期技术发展空白，提出统一框架 generalized OOD detection（广义 OOD 检测），涵盖上述五类问题，即 AD、ND、OSR、OOD detection 与 OD。This paper addresses the gap in recent technical developments in recent technical developments in the field of OOD detection by presenting a unified framework called generalized OOD detection, which encompasses the five aforementioned problems, i.e.,AD, ND, OSR, OOD detection, and OD.

综述领域自适应与迁移学习，重点关注视觉应用及超越图像分类的方法，如目标检测、图像分割、视频分析或视觉属性学习。An overview of domain adaptation and transfer learning with a specific view on visual applications and the methods that go beyond image categorization, such as object detection or image segmentation, video analyses or learning visual attributes are overviewed.

提出 MultiMedQA 基准，整合六个现有医学问答数据集（涵盖专业医学、研究与消费者查询）及一个全新的在线医学问题搜索数据集，并提出针对模型答案的人工评估框架，揭示了 LLM 在医学领域的潜在应用价值。MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, is presented and a human evaluation framework for model answers is proposed, suggesting the potential utility of LLMs in medicine.

本文给出 ICL 的形式化定义，厘清其与相关研究的联系，并梳理讨论训练策略、提示设计策略及相关分析等高级技术。This paper presents a formal definition of ICL and clarify its correlation to related studies, and organizes and discusses advanced techniques, including training strategies, prompt designing strategies, and related analysis.

这篇立场论文定义了可解释性，阐述了何时需要（以及何时不需要）可解释性，并提出了一种用于严格评估的分类法，同时指出了迈向更严谨的可解释机器学习科学所面临的开放性问题This position paper defines interpretability and describes when interpretability is needed (and when it is not), and suggests a taxonomy for rigorous evaluation and exposes open questions towards a more rigorous science of interpretable machine learning.

本工作提出在海量文本语料上使用新的自监督目标 PEGASUS 对大型 Transformer 编码器-解码器模型进行预训练，并证明其在所有 12 个下游数据集上按 ROUGE 分数衡量均取得 SOTA 性能This work proposes pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective, PEGASUS, and demonstrates it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores.

从算法建模、应用和理论分析角度对 MTL 的综述，给出了 MTL 的定义，并将不同 MTL 算法分为五类：特征学习方法、低秩方法、任务聚类方法、任务关系学习方法和分解方法A survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses, which gives a definition of MTL and classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach,task relation learning approach and decomposition approach.

尽管方法简单，Decision Transformer 在 Atari、OpenAI Gym 和 Key-to-Door 任务上达到或超过 SOTA 无模型离线 RL 基线的性能Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

较为全面地列举了基于群体智能、生物启发、物理启发和化学启发（按灵感来源分类）的所有算法，这些算法已成为解决实际问题的流行工具A relatively comprehensive list of all the algorithms based on swarm intelligence, bio-inspired, physics-based and chemistry-based, depending on the sources of inspiration, that have become popular tools for solving real-world problems.

提出了能够平衡"延迟告警的上下文敏感成本"与"打断成本"的模型与推理流程，并通过对用户活动与通知内容的分析，描述了在不确定性下推理此类成本所面临的挑战Models and inference procedures that balance the context-sensitive costs of deferring alerts with the cost of interruption are presented and the challenge of reasoning about such costs under uncertainty via an analysis of user activity and the content of notifications is described.

引入了一个框架，通过提供简洁接口来跟踪实时能耗与碳排放、生成标准化的在线附录来简化核算，并为节能的强化学习算法建立排行榜以激励负责任的研究A framework is introduced that makes accounting easier by providing a simple interface for tracking realtime energy consumption and carbon emissions, as well as generating standardized online appendices, and creates a leaderboard for energy efficient reinforcement learning algorithms to incentivize responsible research.

一个仅依赖字符级输入的简单神经语言模型，仅从字符即可编码语义和正字法信息，表明在许多语言中，字符输入足以完成语言建模。A simple neural language model that relies only on character-level inputs that is able to encode, from characters only, both semantic and orthographic information and suggests that on many languages, character inputs are sufficient for language modeling.

描述了一种无监督学习通用分布式句子编码器的方法，利用书籍文本的连续性，训练编码器-解码器模型以重建编码段落的周围句子。The approach for unsupervised learning of a generic, distributed sentence encoder is described, using the continuity of text from books to train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage.

本工作训练了一个预测的计算最优模型 Chinchilla，使用与 Gopher 相同的计算预算，但参数量为 70B、数据量为 4 倍，达到 SOTA 平均准确率，比 Gopher 提升超过 7%。This work trains a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data, and reaches a state-of-the-art average accuracy, greater than a 7% improvement over Gopher.

本工作综述了可用于推断用户需求的贝叶斯用户模型研究，这些模型综合考虑用户的背景、操作和查询，并提出了一种智能用户界面的整体架构。This work reviews work on Bayesian user models that can be employed to infer a user's needs by considering a users' background, actions, and queries and proposes an overall architecture for an intelligent user interface.