Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness

We present an overview of the CLEF-2018 CheckThat! Lab on Automatic\nIdentification and Verification of Political Claims, with focus on Task 1:\nCheck-Worthiness. The task asks to predict which claims in a political debate\nshould be prioritized for fact-checking. In particular, given a debate or a\npolitical speech, the goal was to produce a ranked list of its sentences based\non their worthiness for fact checking. We offered the task in both English and\nArabic, based on debates from the 2016 US Presidential Campaign, as well as on\nsome speeches during and after the campaign. A total of 30 teams registered to\nparticipate in the Lab and seven teams actually submitted systems for Task~1.\nThe most successful approaches used by the participants relied on recurrent and\nmulti-layer neural networks, as well as on combinations of distributional\nrepresentations, on matchings claims' vocabulary against lexicons, and on\nmeasures of syntactic dependency. The best systems achieved mean average\nprecision of 0.18 and 0.15 on the English and on the Arabic test datasets,\nrespectively. This leaves large room for further improvement, and thus we\nrelease all datasets and the scoring scripts, which should enable further\nresearch in check-worthiness estimation.\n