Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets

Ideally Open-Domain Question Answering models should exhibit a number of\ncompetencies, ranging from simply memorizing questions seen at training time,\nto answering novel question formulations with answers seen during training, to\ngeneralizing to completely novel questions with novel answers. However, single\naggregated test set scores do not show the full picture of what capabilities\nmodels truly have. In this work, we perform a detailed study of the test sets\nof three popular open-domain benchmark datasets with respect to these\ncompetencies. We find that 60-70% of test-time answers are also present\nsomewhere in the training sets. We also find that 30% of test-set questions\nhave a near-duplicate paraphrase in their corresponding training sets. Using\nthese findings, we evaluate a variety of popular open-domain models to obtain\ngreater insight into what extent they can actually generalize, and what drives\ntheir overall performance. We find that all models perform dramatically worse\non questions that cannot be memorized from training sets, with a mean absolute\nperformance difference of 63% between repeated and non-repeated data. Finally\nwe show that simple nearest-neighbor models out-perform a BART closed-book QA\nmodel, further highlighting the role that training set memorization plays in\nthese benchmarks\n