Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge

We present the ARC-DA dataset, a direct-answer ("open response", "freeform")\nversion of the ARC (AI2 Reasoning Challenge) multiple-choice dataset. While ARC\nhas been influential in the community, its multiple-choice format is\nunrepresentative of real-world questions, and multiple choice formats can be\nparticularly susceptible to artifacts. The ARC-DA dataset addresses these\nconcerns by converting questions to direct-answer format using a combination of\ncrowdsourcing and expert review. The resulting dataset contains 2985 questions\nwith a total of 8436 valid answers (questions typically have more than one\nvalid answer). ARC-DA is one of the first DA datasets of natural questions that\noften require reasoning, and where appropriate question decompositions are not\nevident from the questions themselves. We describe the conversion approach\ntaken, appropriate evaluation metrics, and several strong models. Although\nhigh, the best scores (81% GENIE, 61.4% F1, 63.2% ROUGE-L) still leave\nconsiderable room for improvement. In addition, the dataset provides a natural\nsetting for new research on explanation, as many questions require reasoning to\nconstruct answers. We hope the dataset spurs further advances in complex\nquestion-answering by the community. ARC-DA is available at\nhttps://allenai.org/data/arc-da\n