Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters

Adapter layers are lightweight, learnable units inserted between transformer\nlayers. Recent work explores using such layers for neural machine translation\n(NMT), to adapt pre-trained models to new domains or language pairs, training\nonly a small set of parameters for each new setting (language pair or domain).\nIn this work we study the compositionality of language and domain adapters in\nthe context of Machine Translation. We aim to study, 1) parameter-efficient\nadaptation to multiple domains and languages simultaneously (full-resource\nscenario) and 2) cross-lingual transfer in domains where parallel data is\nunavailable for certain language pairs (partial-resource scenario). We find\nthat in the partial resource scenario a naive combination of domain-specific\nand language-specific adapters often results in `catastrophic forgetting' of\nthe missing languages. We study other ways to combine the adapters to alleviate\nthis issue and maximize cross-lingual transfer. With our best adapter\ncombinations, we obtain improvements of 3-4 BLEU on average for source\nlanguages that do not have in-domain data. For target languages without\nin-domain data, we achieve a similar improvement by combining adapters with\nback-translation. Supplementary material is available at\nhttps://tinyurl.com/r66stbxj\n