TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

TACRED (Zhang et al., 2017) is one of the largest, most widely used\ncrowdsourced datasets in Relation Extraction (RE). But, even with recent\nadvances in unsupervised pre-training and knowledge enhanced neural RE, models\nstill show a high error rate. In this paper, we investigate the questions: Have\nwe reached a performance ceiling or is there still room for improvement? And\nhow do crowd annotations, dataset, and models contribute to this error rate? To\nanswer these questions, we first validate the most challenging 5K examples in\nthe development and test sets using trained annotators. We find that label\nerrors account for 8% absolute F1 test error, and that more than 50% of the\nexamples need to be relabeled. On the relabeled test set the average F1 score\nof a large baseline model set improves from 62.1 to 70.1. After validation, we\nanalyze misclassifications on the challenging instances, categorize them into\nlinguistically motivated error groups, and verify the resulting error\nhypotheses on three state-of-the-art RE models. We show that two groups of\nambiguous relations are responsible for most of the remaining errors and that\nmodels may adopt shallow heuristics on the dataset when entities are not\nmasked.\n