Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

Transformer-based models have brought a radical change to neural machine\ntranslation. A key feature of the Transformer architecture is the so-called\nmulti-head attention mechanism, which allows the model to focus simultaneously\non different parts of the input. However, recent works have shown that most\nattention heads learn simple, and often redundant, positional patterns. In this\npaper, we propose to replace all but one attention head of each encoder layer\nwith simple fixed -- non-learnable -- attentive patterns that are solely based\non position and do not require any external knowledge. Our experiments with\ndifferent data sizes and multiple language pairs show that fixing the attention\nheads on the encoder side of the Transformer at training time does not impact\nthe translation quality and even increases BLEU scores by up to 3 points in\nlow-resource scenarios.\n