If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques

In recent years, there has been an explosion of AI research on counterfactual\nexplanations as a solution to the problem of eXplainable AI (XAI). These\nexplanations seem to offer technical, psychological and legal benefits over\nother explanation techniques. We survey 100 distinct counterfactual explanation\nmethods reported in the literature. This survey addresses the extent to which\nthese methods have been adequately evaluated, both psychologically and\ncomputationally, and quantifies the shortfalls occurring. For instance, only\n21% of these methods have been user tested. Five key deficits in the evaluation\nof these methods are detailed and a roadmap, with standardised benchmark\nevaluations, is proposed to resolve the issues arising; issues, that currently\neffectively block scientific progress in this field.\n