A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation

Recent advances in AI and ML applications have benefited from rapid progress\nin NLP research. Leaderboards have emerged as a popular mechanism to track and\naccelerate progress in NLP through competitive model development. While this\nhas increased interest and participation, the over-reliance on single, and\naccuracy-based metrics have shifted focus from other important metrics that\nmight be equally pertinent to consider in real-world contexts. In this paper,\nwe offer a preliminary discussion of the risks associated with focusing\nexclusively on accuracy metrics and draw on recent discussions to highlight\nprescriptive suggestions on how to develop more practical and effective\nleaderboards that can better reflect the real-world utility of models.\n