What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?

Crowdsourcing is widely used to create data for common natural language\nunderstanding tasks. Despite the importance of these datasets for measuring and\nrefining model understanding of language, there has been little focus on the\ncrowdsourcing methods used for collecting the datasets. In this paper, we\ncompare the efficacy of interventions that have been proposed in prior work as\nways of improving data quality. We use multiple-choice question answering as a\ntestbed and run a randomized trial by assigning crowdworkers to write questions\nunder one of four different data collection protocols. We find that asking\nworkers to write explanations for their examples is an ineffective stand-alone\nstrategy for boosting NLU example difficulty. However, we find that training\ncrowdworkers, and then using an iterative process of collecting data, sending\nfeedback, and qualifying workers based on expert judgments is an effective\nmeans of collecting challenging data. But using crowdsourced, instead of expert\njudgments, to qualify workers and send feedback does not prove to be effective.\nWe observe that the data from the iterative protocol with expert assessments is\nmore challenging by several measures. Notably, the human--model gap on the\nunanimous agreement portion of this data is, on average, twice as large as the\ngap for the baseline protocol data.\n