Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translation

Is bias amplified when neural machine translation (NMT) models are optimized\nfor speed and evaluated on generic test sets using BLEU? We investigate\narchitectures and techniques commonly used to speed up decoding in\nTransformer-based models, such as greedy search, quantization, average\nattention networks (AANs) and shallow decoder models and show their effect on\ngendered noun translation. We construct a new gender bias test set, SimpleGEN,\nbased on gendered noun phrases in which there is a single, unambiguous, correct\nanswer. While we find minimal overall BLEU degradation as we apply speed\noptimizations, we observe that gendered noun translation performance degrades\nat a much faster rate.\n