Researchers in explainable artificial intelligence have developed numerous methods for helping users understand the predictions of complex supervised learning models. By contrast, explaining the $\textit{uncertainty}$ of model outputs has received relatively little attention. We adapt the popular Shapley value framework to explain various types of predictive uncertainty, quantifying each feature's contribution to the conditional entropy of individual model outputs. We consider games with modified characteristic functions and find deep connections between the resulting Shapley values and fundamental quantities from information theory and conditional independence testing. We outline inference procedures for finite sample error rate control with provable guarantees, and implement efficient algorithms that perform well in a range of experiments on real and simulated data. Our method has applications to covariate shift detection, active learning, feature selection, and active feature-value acquisition.
Paper
Similar papers
Peer review
Summary
The paper aims to define Shapley values to decompose the conditional distribution of the outcome so to attribute the uncertainty of the outcome to individual variables. This aims to extend prior work, which has mainly focused on explaining the change in the conditional mean of the outcome.
Strengths
The manuscript highlight how methods for explaining the conditional variance can have highly different results from those for explaining the conditional mean. This is nicely shown in the simulation results in Section 6.3 and Figure 3. The authors also nicely highlight how methods for explaining the conditional variance can be useful in many contexts, such as for model calibration.
Weaknesses
1. There are a number of other papers that have studied how to explain the conditional variance of Y given X using Shapley values. For instance, Williamson and Feng 2020 use Shapley values to determine how well individual features predict Y using some user-defined scoring function V. If this scoring function is set to the deviance/KL divergence, then the resulting Shapley values are highly similar to those discussed in this paper. Moreover, this paper discusses how to efficiently perform statistical inference that sidestep computational issues that arise when considering every possible variable subset. The authors should put the current manuscript in context, given such existing works. 2. The link between conditional independencies and the information theoretic Shapley values defined in this paper follows very directly from how the Shapley values were defined. I don't see why the authors state that this connection is "deep" or "subtle". Perhaps the authors can clarify why this connection is far from obvious. 3. The proposed Shapley value is characterizing properties of the oracle distribution. Given the framing of the paper, the true target for statistical inference should be the Shapley value with respect to the oracle distribution. However, the result in Theorem 5.1 does not account for the uncertainty in estimating the oracle distribution and performs inference for a different target (seems to be the mean of the Shapley value of the estimated distribution). Moreover, it is not clear to me what the use case is for marginal coverage guarantees like that presented in Theorem 5.1. (Conditional coverage guarantees are ideal, but I agree that is unattainable.) I suggest better motivating (i) why are we doing statistical inference and (ii) what the target of statistical inference is. 4. This framework seems most relevant for explaining the uncertainty of either a continuous outcome or a multi-class outcome. For binary outcomes, the mean-variance relationship suggests to me that the results from using more typical Shapley values will be highly similar to those using this new definition. However, many of the experiments in Section 6 deal with binary outcomes. First, the authors should compare their results to those from using Shapley values for explaining the conditional mean. Second, the author should consider additional experiments dealing with other types of outcomes, to really highlight the utility of their proposed method.
Questions
N/A
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The authors discuss some limitations of their method.
Summary
I have previously reviewed this paper at another conference in which, after reviewers' discussion, it was a borderline reject, below my review is enriched with the previous conference discussions. This work proposes calculating a modified Shapley value where the coalitional game represents the entropy of the predictive distribution. A high-scoring feature will increase the entropy, and a low-score feature will decrease it. Sections 4 and 5 describe the properties of their value function, along with a couple of alternative value functions. They discuss how their value function connects to the notion of information gain, a decomposition of uncertainty into aleatoric and epistemic sources, and a PAC-like technique for testing if a feature's mean Shapley value is near zero. Various experiments are performed to illustrate use cases and validate correctness.
Strengths
- The idea of using KL divergence and Entropy in combination with Shapley Values to attempt to quantify the individual importance of features is novel and potentially very useful. Not many papers have delved into the topic of attributing uncertainty to input features. - Nice job deriving properties from the two main coalitional games they discussed - Extensive set of tests given.
Weaknesses
In the final discussion, the main reasons to reject were the following: - Developing metrics to verify the correctness of their results - Making their implementation consistent with the methods section. - Experimental Limitations: the experimental design is based on papers from Computer Vision, where pure covariate shift is identifiable. - Some missing discussions about related work
Questions
Since the paper was previously rejected and no major modifications have happened. Why would the authors expect the paper to be accepted this time?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
-
Summary
This paper introduces a method for explaining the uncertainty in predictions made by DNNs. The authors extend the Shapley values from explaining the value of DNN outputs to explaining the uncertainty of the DNN outputs. Then, the authors demonstrate the close relationship between these Shapley values and the conditional independencies between inputs and outputs. Additionally, the authors provide a theoretical bound to support the coverage of the explanation obtained by the proposed method.
Strengths
1. The paper is well-organized and easy to follow. The authors also provide a thorough literature review. 2. The authors prove some propositions and theorems to support the reliability of the proposed definition of Shapley values in DNNs. 3. It is good that the authors recognize the presence of both epistemic and aleatoric uncertainty in the prediction of DNNs.
Weaknesses
1. The motivation behind investigating the Shapley values of inputs towards the uncertainty of predictions is not convincing. While the proposed information-theoretic Shapley value reveals which variables contribute to prediction uncertainty, it fails to explain which variables are responsible for correct/incorrect predictions. The original Shapley values defined on the model output can provide this information. On the other hand, the results in Figure 1 do not demonstrate the superiority of the proposed information-theoretic Shapley value. This is because the features that increase predictive uncertainty can also be considered as pixels with negative Shapley values toward the correct prediction. Therefore, the information-theoretic Shapley value does not offer significant additional insights or information. 2. How should I understand the Shapley values defined on the value functions $v_{KL}$ and $v_{CE}$? When using these value functions, the output $v(N)$ is always 0. 3. Is the proposed information-theoretic Shapley value a local explanation for individual samples or a global explanation over all training samples? If it is a local explanation, what does the coverage of the explanation represent? If it is a global explanation, how is the value function defined over different samples? 4. In Figure 2, many perturbed inputs are assigned negative Shapley values for predictive uncertainty, which is perplexing. In my understanding, a negative contribution to the predictive uncertainty indicates that this feature is discriminative. However, the perturbed features are typically expected to be non-discriminative. 5. There is a lack of quantitative experiments to validate the accuracy of the proposed Shapley values in estimating the uncertainty/variance of the prediction. Figure 3(A) just provides qualitative results and Figure 3(B) focuses solely on the ranking of features. I suggest the authors conduct a quantitative experiment to show that the proposed method could precisely explain and estimate the variance of the prediction.
Questions
1. Further discussion is needed to clarify the motivation and advantages of explaining predictive uncertainty. 2. It is suggested that the authors perform a quantitative experiment to demonstrate the ability of the proposed method to accurately explain and estimate prediction variance.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors have discussed the limitation of the method.
Summary
Authors aim to explain uncertainty in model outputs by adapting Shapely value framework. In specific, authors try to explain predictive distributions through the lens of entropy, cross-entropy, KL and information gain. Authors offer some theoretic interpretation of the Shapley values adapted with these metrics and, through experiments, show the efficacy of the entropy based variant of Shapely values.
Strengths
1. I would like to thank the authors for presenting their work in a very thoughtful manner. I find most of the paper to be clear and intuition which follows propositions and theorems to be very helpful. I like the how the paper introduces theory to offer intuition and make some simplifications to make the method more practical. 2. With raising importance in uncertainty quantification in ML, the work presented in this paper is very relevant to the community. 3. The paper not only presents theoretical analysis of the modified variants of the Shapley values but also offers empirical evidence of how their proposed method works beyond the assumptions made in the theoretical analysis.
Weaknesses
I might be missing something, but I am unable to understand the specific details of the experiments by reading the main paper and the appendix. (even the supplementary code didn't have much documentation). For example, I couldn't find the details of ensembles method used. What is the size of ensemble (`B` in Sec 5)? Which type of ensemble [5] is being used? What was the batch size how many training epochs were run? I think there is a line of work on uncertainty estimation and it's interpretation which is missing and might be useful. (A subset of these papers might be useful: [1] , [2], [3], [4]) [1] From Predictions to Decisions: The Importance of Joint Predictive Distributions (https://arxiv.org/pdf/2107.09224.pdf) [2] Epistemic neural networks (https://arxiv.org/pdf/2107.08924.pdf) [3] Evaluating High-Order Predictive Distributions in Deep Learning (https://proceedings.mlr.press/v180/osband22a/osband22a.pdf) [4] Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Sampling (https://arxiv.org/pdf/2205.08766.pdf) [5] Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping (https://arxiv.org/pdf/2206.03633.pdf)
Questions
1. In Section 6, even through the problems were explained in some detail, I find Section 6 and Appendix C to be lacking in details of methods used and their hyperparameters. Can you please include these details? 2. Some parts of the paper refers to active-learning experiments in the experiments section, but it was not obvious to me how the experiments in the Section 6 are related to active learning. Can you clarify this? 3. In Section 5, authors present computation of entropy (h_t) with a single point estimate and then computation of aleatoric entropy (h_a) with an ensemble of particles. It is not clear how h_t is estimated when we use an ensemble of particles. Can you clarify this? 4. There is reference to script D in line 219. I don't think this was introduced before or referenced later. It would be useful to replace this with an appropriate term.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Please refer to the weakness section. I am willing to increase my score if authors provide more details about their experiments either in the main paper or the appendix.
I would like to thank the authors for addressing the points I raised. As mentioned in my review, I am increasing the score to accept.
I would like to thank the authors for their response, but some concerns still remain. Weakness 1. I'm clear about the difference between explaining uncertainty and explaining the error/model output. My concern is that explaining the model output usually provides more information than just explaining the uncertainty. Although there are some cases where the label is unknown, explaining the output value of each category may also help people understand the inference of the model. Weakness 2. When taking $v_{KL}$ and $v_{CE}$ as the value function, the output $v(N)$ given all input features is always zero, because it computes the distance between two same distributions ($p(y|x)$). Thus, my question is, what does the Shapley value in this scenario mean? Using the traditional reward function (maybe the model output or loss), the Shapley value refers to how much contribution each input feature has to the output score/loss. But with $v_{KL}$ and $v_{CE}$ as the value function, I don't know how to understand the Shapley value. Weakness 4. What does the miscalibration refer to? The authors did not answer why the randomly perturbed features were explained as discriminative features (leading to the over-confidence of the prediction).
Re: Official Comment by Reviewer DQBN
We thank R2 for taking the time to reply to our rebuttal. We respond to their new comments below. W1. Explaining individual predictions provides information at a different level of detail. Existing tools are already well-suited to these applications. However, as we note in our rebuttal, current methods may not be useful in other areas that rely on uncertainty evaluation such as active learning and OOD detection. Though one could in principle inspect SHAP explanations for each candidate label to probe for sources of uncertainty, this becomes infeasible with more than two or three classes and is unnecessarily indirect. Moreover, focusing on the first conditional moment ignores higher order information, as we explain in our motivating example at the top of Sect. 4 where $X, Z \sim \mathcal{U}(0, 1)^2$ and $Y \sim \mathcal{N}(X, Z^2)$. Standard SHAP values will assign zero importance to $Z$ here, although the variable does in fact provide information about the conditional distribution of $Y$. In summary, our proposed Shapley values *complement* rather than *supplant* existing alternatives. W2. Apologies for misunderstanding R2’s original comment – we see now that what R2 calls $N$ is what we call $[d]$. Often in Shapley style games, the value function has positive range with $v(\emptyset) = 0$. However, as R2 correctly observes, zero is the *ceiling* rather than the *floor* for $v_{KL}$ and $v_{CE}$. That is, these value functions have *negative* range, with $v_{KL}(\emptyset, \mathbf{x})$ representing the negative KL-divergence between prior and posterior distributions for $Y$. As we explain in Prop. 4.1 and the surrounding text, resulting marginal payouts $\Delta_{KL}(S, j, \mathbf{x})$ represent the information gained by $X_j = x_j$ when we already know feature values for coalition $S$, assuming the true target distribution is $Y \mid \mathbf{x}$. As we explain in Prop 4.2 and the surrounding text, resulting Shapley values represent the contribution in bits to the KL-divergence between prior and posterior distributions. W4. Miscalibration in this context means that the model is over-confident in its prediction. Perturbing a feature can lead to overconfidence if, for instance, the model infers a monotonic relationship between $X$ and $P(Y \mid X)$ during training and then sees an unusually high value for $X$ at test time. If random perturbations decrease predictive uncertainty, we infer that (a) the feature in question must be one the model relies on for making predictions, otherwise we would see no change in output; and (b) the model is extrapolating with too much confidence to anomalous regions of the feature space.
Re: Official Comment
Thanks again to R2 for taking the time to go through our rebuttal. Please let us know if there is anything more we can do to address reviewer comments or concerns. If we have resolved all major issues raised in the initial review, then we invite R2 to revise their score upward :)
Re: Rebuttal
Please let us know if there is anything more we can do to address reviewer comments or concerns. If we have resolved all major issues raised in the initial review, then we invite R3 to revise their score upward :)
Many thanks! I see how the authors addressed the questions and what has changed in the submissions. I find the paper in a good level of maturity. I will consider increasing my score after the discussion with other reviewers.
Re: Rebuttal
Please let us know if there is anything more we can do to address reviewer comments or concerns. If we have resolved all major issues raised in the initial review, then we invite R4 to revise their score upward :)
Thank you for the response
I thank the authors for their response. Couple of comments: 1. Re other methods that could be used for VI for conditional variance: * W&F is a general framework that can be instantiated with various definitions for predictiveness. One such value function could be the average squared error of the oracle model with respect to a variable subset, which corresponds to the average variance for the conditional distribution of Y given X_s. The VI measures in this paper can be defined for subsets of the population, leading to conditional variable importance measures. * Also, can one not apply the standard Shapley measure used in the literature, but to a ML algorithm that estimates the conditional variance as opposed to the conditional mean? Although the standard procedure doesn't have uncertainty quantification, it seems like it can be applied to estimate similar quantities. 2. I thank the authors for clarifying which parts of Theorem 4.5 are supposed to be original. Perhaps this can be clarified a bit more in the text. Yes, I agree part (c) is not that obvious, and the link to the faithfulness condition used in causal inference is interesting. 3. I thank the authors for clarifying the sources of uncertainty. As such, I would suggest revising the manuscript with these assumptions regarding consistency and coverage of the Shapley values is in fact asymptotic. I will increase my score by 1.
Re: Reviewer comment
Many thanks to R4 for taking the time to read through our rebuttal and revise their score upward. In reply to R4's comments: 1. We agree that W&F's framework (or, for that matter, the Slack et al. Bayesian method we originally cited) could be applied to the value functions we describe – indeed, they can be applied to *any* value function. Our novelty lies primarily in proposing and studying a number of information theoretic value functions that are not in widespread use, and illustrating their applicability for settings that require uncertainty quantification (e.g., active learning and out-of-distribution detection). In addition, we outline a conformal inference procedure for testing whether Shapley values for a given feature concentrate around zero, a task that is subtly different from both BayesSHAP (which is only for local inference, and would therefore require $n$ tests to draw a global inference) and SPVIM (which requires a global value function upfront, as opposed to the local measures we aggregate). As for R4's question about building a model to estimate conditional variance, the answer is yes – that is precisely what our $h$ function does, although details vary depending on whether we aim to explain the epistemic or aleatoric uncertainty (see Sect. 5). For examples of how this works in practice, see Sect. 6. 2. We thank R4 for reconsidering their position on Thm. 4.5, especially with regard to item (c). We also find the faithfulness connection to be quite interesting! We will be sure to clarify the relative significance of each item in the text, and will try to move some of the discussion from Appx. A to the main text if space allows. 3. R4's point on asymptotic consistency and coverage in Thm. 5.1 is well-taken. We have thoroughly revised the text to address R4's questions on this topic, including an expanded discussion on sources of uncertainty when estimating the oracle band. We thank R4 again for challenging us to work through this result more carefully. The manuscript is much improved following this revision. We reiterate our appreciation for R4's thoughtful review and reply. We would be happy to address any further questions or comments R4 may have.
We would like to once again thank all the reviewers for their contributions before the discussion period comes to a close. If there are any further comments or questions we can address, please let us know and we will do our best to reply promptly. We are very grateful for the helpful feedback and stimulating discussion!
Decision
Accept (poster)