Detecting Word Sense Disambiguation Biases in Machine Translation for Model-Agnostic Adversarial Attacks

Word sense disambiguation is a well-known source of translation errors in\nNMT. We posit that some of the incorrect disambiguation choices are due to\nmodels' over-reliance on dataset artifacts found in training data, specifically\nsuperficial word co-occurrences, rather than a deeper understanding of the\nsource text. We introduce a method for the prediction of disambiguation errors\nbased on statistical data properties, demonstrating its effectiveness across\nseveral domains and model types. Moreover, we develop a simple adversarial\nattack strategy that minimally perturbs sentences in order to elicit\ndisambiguation errors to further probe the robustness of translation models.\nOur findings indicate that disambiguation robustness varies substantially\nbetween domains and that different models trained on the same data are\nvulnerable to different attacks.\n