The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models

Mechanisms for encoding positional information are central for\ntransformer-based language models. In this paper, we analyze the position\nembeddings of existing language models, finding strong evidence of translation\ninvariance, both for the embeddings themselves and for their effect on\nself-attention. The degree of translation invariance increases during training\nand correlates positively with model performance. Our findings lead us to\npropose translation-invariant self-attention (TISA), which accounts for the\nrelative position between tokens in an interpretable fashion without needing\nconventional position embeddings. Our proposal has several theoretical\nadvantages over existing position-representation approaches. Experiments show\nthat it improves on regular ALBERT on GLUE tasks, while only adding orders of\nmagnitude less positional parameters.\n