Evaluating NLG systems: A brief introduction

This year the International Conference on Natural Language Generation (INLG) will feature an award for the paper with the best evaluation. The purpose of this award is to provide an incentive for NLG researchers to pay more attention to the way they assess the output of their systems. This essay provides a short introduction to evaluation in NLG, explaining key terms and distinctions.

cs.CL Natural Language Processing Techniques Topic Modeling Semantic Web and Ontologies

View graph Source

Paper

Similar papers

Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent PapersMika Hämäläinen, Khalid Alnajjar2021 In Library

Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent PapersMika Hämäläinen, Khalid Alnajjar2021 · arXiv (Cornell University) In Library

Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their ImplicationsKaitlyn Zhou, Su Lin Blodgett, Adam Trischler et al.2022 · arXiv (Cornell University) In Library

NLG Evaluation: Past, Present, FutureEhud Reiter2026 · 0 citations In Library

Human evaluation of automatically generated text: Current trends and best practice guidelinesChris van der Lee, Albert Gatt, Emiel van Miltenburg et al.2020 · Computer Speech & Language In Library

LLM-based NLG Evaluation: Current Status and ChallengesMingqi Gao, Xinyu Hu, Jie Ruan et al.2024 · Computational Linguistics · 134 citations In Library

A Comparative Evaluation Methodology for NLG in Interactive SystemsH. Hastie, A. Belz2014 · International Conference on Language Resources and Evaluation · 16 citations In Library

Leveraging Large Language Models for NLG Evaluation: Advances and ChallengesZhen Li, Xiaohan Xu, Tao Shen et al.2024 · arXiv (Cornell University) · 57 citations In Library

A Tutorial on Evaluation Metrics used in Natural Language GenerationMitesh M. Khapra, Ananya B. Sai2021 · North American Chapter of the Association for Computational Linguistics · 7 citations In Library

A Survey of Evaluation Metrics Used for NLG SystemsAnanya B. Sai, Akash Kumar Mohankumar, Mitesh M. Khapra2022 · ACM Computing Surveys In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal