TLDR
由于二次注意力的特性，大语言模型 (LLM) 消耗大量内存与算力。每新增一个 token 的成本都高于前一个，且其键值对必须无限期保存在内存中，难以持续。为此已提出多种替代方案以解决二次扩展问题，其中之一是将 LLM 改造为线性注意力。该思路因承诺以低成本取得 SOTA 性能而备受关注。然而，该研究方向尚未得到充分 propDue to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable. Several alternatives have been proposed to fix the quadratic scaling problem, one of which is retrofitting LLMs to use Linear Attention. This idea has attracted a lot of attention, given its promise to solve the quadratic scaling problem with state-of-the-art performance at low cost. However, this line of research has not been prop