The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation

Recent text generation research has increasingly focused on open-ended\ndomains such as story and poetry generation. Because models built for such\ntasks are difficult to evaluate automatically, most researchers in the space\njustify their modeling choices by collecting crowdsourced human judgments of\ntext quality (e.g., Likert scores of coherence or grammaticality) from Amazon\nMechanical Turk (AMT). In this paper, we first conduct a survey of 45\nopen-ended text generation papers and find that the vast majority of them fail\nto report crucial details about their AMT tasks, hindering reproducibility. We\nthen run a series of story evaluation experiments with both AMT workers and\nEnglish teachers and discover that even with strict qualification filters, AMT\nworkers (unlike teachers) fail to distinguish between model-generated text and\nhuman-generated references. We show that AMT worker judgments improve when they\nare shown model-generated output alongside human-generated references, which\nenables the workers to better calibrate their ratings. Finally, interviews with\nthe English teachers provide deeper insights into the challenges of the\nevaluation process, particularly when rating model-generated text.\n