EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering

We propose EXAMS -- a new benchmark dataset for cross-lingual and\nmultilingual question answering for high school examinations. We collected more\nthan 24,000 high-quality high school exam questions in 16 languages, covering 8\nlanguage families and 24 school subjects from Natural Sciences and Social\nSciences, among others.\n EXAMS offers a fine-grained evaluation framework across multiple languages\nand subjects, which allows precise analysis and comparison of various models.\nWe perform various experiments with existing top-performing multilingual\npre-trained models and we show that EXAMS offers multiple challenges that\nrequire multilingual knowledge and reasoning in multiple domains. We hope that\nEXAMS will enable researchers to explore challenging reasoning and knowledge\ntransfer methods and pre-trained models for school question answering in\nvarious languages which was not possible before. The data, code, pre-trained\nmodels, and evaluation are available at https://github.com/mhardalov/exams-qa.\n