F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering

Explainable question answering systems predict an answer together with an\nexplanation showing why the answer has been selected. The goal is to enable\nusers to assess the correctness of the system and understand its reasoning\nprocess. However, we show that current models and evaluation settings have\nshortcomings regarding the coupling of answer and explanation which might cause\nserious issues in user experience. As a remedy, we propose a hierarchical model\nand a new regularization term to strengthen the answer-explanation coupling as\nwell as two evaluation scores to quantify the coupling. We conduct experiments\non the HOTPOTQA benchmark data set and perform a user study. The user study\nshows that our models increase the ability of the users to judge the\ncorrectness of the system and that scores like F1 are not enough to estimate\nthe usefulness of a model in a practical setting with human users. Our scores\nare better aligned with user experience, making them promising candidates for\nmodel selection.\n