Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Since the introduction of the transformer model by Vaswani et al. (2017), a\nfundamental question has yet to be answered: how does a model achieve\nextrapolation at inference time for sequences that are longer than it saw\nduring training? We first show that extrapolation can be enabled by simply\nchanging the position representation method, though we find that current\nmethods do not allow for efficient extrapolation. We therefore introduce a\nsimpler and more efficient position method, Attention with Linear Biases\n(ALiBi). ALiBi does not add positional embeddings to word embeddings; instead,\nit biases query-key attention scores with a penalty that is proportional to\ntheir distance. We show that this method trains a 1.3 billion parameter model\non input sequences of length 1024 that extrapolates to input sequences of\nlength 2048, achieving the same perplexity as a sinusoidal position embedding\nmodel trained on inputs of length 2048 but training 11% faster and using 11%\nless memory. ALiBi's inductive bias towards recency also leads it to outperform\nmultiple strong position methods on the WikiText-103 benchmark.\n