Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

We study the power of cross-attention in the Transformer architecture within\nthe context of transfer learning for machine translation, and extend the\nfindings of studies into cross-attention when training from scratch. We conduct\na series of experiments through fine-tuning a translation model on data where\neither the source or target language has changed. These experiments reveal that\nfine-tuning only the cross-attention parameters is nearly as effective as\nfine-tuning all parameters (i.e., the entire translation model). We provide\ninsights into why this is the case and observe that limiting fine-tuning in\nthis manner yields cross-lingually aligned embeddings. The implications of this\nfinding for researchers and practitioners include a mitigation of catastrophic\nforgetting, the potential for zero-shot translation, and the ability to extend\nmachine translation models to several new language pairs with reduced parameter\nstorage overhead.\n