The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations

Feature importance (FI) estimates are a popular form of explanation, and they\nare commonly created and evaluated by computing the change in model confidence\ncaused by removing certain input features at test time. For example, in the\nstandard Sufficiency metric, only the top-k most important tokens are kept. In\nthis paper, we study several under-explored dimensions of FI explanations,\nproviding conceptual and empirical improvements for this form of explanation.\nFirst, we advance a new argument for why it can be problematic to remove\nfeatures from an input when creating or evaluating explanations: the fact that\nthese counterfactual inputs are out-of-distribution (OOD) to models implies\nthat the resulting explanations are socially misaligned. The crux of the\nproblem is that the model prior and random weight initialization influence the\nexplanations (and explanation metrics) in unintended ways. To resolve this\nissue, we propose a simple alteration to the model training process, which\nresults in more socially aligned explanations and metrics. Second, we compare\namong five approaches for removing features from model inputs. We find that\nsome methods produce more OOD counterfactuals than others, and we make\nrecommendations for selecting a feature-replacement function. Finally, we\nintroduce four search-based methods for identifying FI explanations and compare\nthem to strong baselines, including LIME, Anchors, and Integrated Gradients.\nThrough experiments with six diverse text classification datasets, we find that\nthe only method that consistently outperforms random search is a Parallel Local\nSearch (PLS) that we introduce. Improvements over the second-best method are as\nlarge as 5.4 points for Sufficiency and 17 points for Comprehensiveness. All\nsupporting code for experiments in this paper is publicly available at\nhttps://github.com/peterbhase/ExplanationSearch.\n