A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution

With the increasing adoption of predictive models trained using machine\nlearning across a wide range of high-stakes applications, e.g., health care,\nsecurity, criminal justice, finance, and education, there is a growing need for\neffective techniques for explaining such models and their predictions. We aim\nto address this problem in settings where the predictive model is a black box;\nThat is, we can only observe the response of the model to various inputs, but\nhave no knowledge about the internal structure of the predictive model, its\nparameters, the objective function, and the algorithm used to optimize the\nmodel. We reduce the problem of interpreting a black box predictive model to\nthat of estimating the causal effects of each of the model inputs on the model\noutput, from observations of the model inputs and the corresponding outputs. We\nestimate the causal effects of model inputs on model output using variants of\nthe Rubin Neyman potential outcomes framework for estimating causal effects\nfrom observational data. We show how the resulting causal attribution of\nresponsibility for model output to the different model inputs can be used to\ninterpret the predictive model and to explain its predictions. We present\nresults of experiments that demonstrate the effectiveness of our approach to\nthe interpretation of black box predictive models via causal attribution in the\ncase of deep neural network models trained on one synthetic data set (where the\ninput variables that impact the output variable are known by design) and two\nreal-world data sets: Handwritten digit classification, and Parkinson's disease\nseverity prediction. Because our approach does not require knowledge about the\npredictive model algorithm and is free of assumptions regarding the black box\npredictive model except that its input-output responses be observable, it can\nbe applied, in principle, to any black box predictive model.\n