Fine-grained linguistic evaluation for state-of-the-art Machine Translation

This paper describes a test suite submission providing detailed statistics of\nlinguistic performance for the state-of-the-art German-English systems of the\nFifth Conference of Machine Translation (WMT20). The analysis covers 107\nphenomena organized in 14 categories based on about 5,500 test items, including\na manual annotation effort of 45 person hours. Two systems (Tohoku and Huoshan)\nappear to have significantly better test suite accuracy than the others,\nalthough the best system of WMT20 is not significantly better than the one from\nWMT19 in a macro-average. Additionally, we identify some linguistic phenomena\nwhere all systems suffer (such as idioms, resultative predicates and\npluperfect), but we are also able to identify particular weaknesses for\nindividual systems (such as quotation marks, lexical ambiguity and sluicing).\nMost of the systems of WMT19 which submitted new versions this year show\nimprovements.\n