A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering

The dependency between an adequate question formulation and correct answer\nselection is a very intriguing but still underexplored area. In this paper, we\nshow that question rewriting (QR) of the conversational context allows to shed\nmore light on this phenomenon and also use it to evaluate robustness of\ndifferent answer selection approaches. We introduce a simple framework that\nenables an automated analysis of the conversational question answering (QA)\nperformance using question rewrites, and present the results of this analysis\non the TREC CAsT and QuAC (CANARD) datasets. Our experiments uncover\nsensitivity to question formulation of the popular state-of-the-art models for\nreading comprehension and passage ranking. Our results demonstrate that the\nreading comprehension model is insensitive to question formulation, while the\npassage ranking changes dramatically with a little variation in the input\nquestion. The benefit of QR is that it allows us to pinpoint and group such\ncases automatically. We show how to use this methodology to verify whether QA\nmodels are really learning the task or just finding shortcuts in the dataset,\nand better understand the frequent types of error they make.\n