To explain predictions made by complex machine learning models, many feature attribution methods have been developed that assign importance scores to input features. Some recent work challenges the robustness of these methods by showing that they are sensitive to input and model perturbations, while other work addresses this issue by proposing robust attribution methods. However, previous work on attribution robustness has focused primarily on gradient-based feature attributions, whereas the robustness of removal-based attribution methods is not currently well understood. To bridge this gap, we theoretically characterize the robustness properties of removal-based feature attributions. Specifically, we provide a unified analysis of such methods and derive upper bounds for the difference between intact and perturbed attributions, under settings of both input and model perturbations. Our empirical results on synthetic and real-world data validate our theoretical results and demonstrate their practical implications, including the ability to increase attribution robustness by improving the model's Lipschitz regularity.
Paper
Similar papers
Peer review
Summary
This paper derives robustness results regarding a general class of feature attribution methods referred to as “removal-based feature attributions”, which includes occlusion methods, but also Shapley values and LIME explanations. The authors study Lipschitz-continuity properties of these methods with respect to changes in the modeled function and the inputs. They find that, under Lipschitz assumptions on the classifiers output, the removal-based attributions are also Lipschitz-continuous w.r.t to deviation in the inputs and the model.
Strengths
**Generality.** The class of feature attributions studied in this work is kept quite general and it thus covers many relevant implementations such as Occlusion, LIME, and Shapley values. **Technical soundness.** I checked the proofs of some main results and was not able to identify any flaws This makes this work is a rigorous technical contribution in my view. **Good exposition and presentation.** The formalization and notation were introduced well, and the results were good to follow. The proofs that I checked were sufficiently detailed to convince me of the correctness of the theoretical claims. The tables help to get an overview over the various results. **Interesting results on robust summarization techniques.** I particularly like the results on the robustness of the summarization techniques and the statements of the most robust aggregation techniques under each set of axioms (Lemma 8 - Lemma 11). The do not rely on any assumptions and can be operationalized directly, for instance for robust data valuation as done in the recent work mentioned (Wang & Jia, 2023).
Weaknesses
**How robust should it be?** While the results are interesting, they are hard to interpret, because there is not optimal robustness. As the authors correctly state, too small robustness can be considered as invariance as in the sense of Adebayo et al. On the contrary, low robustness allows for adversarial attributions. Therefore, the computation of the robustness score allows for no clear interpretation. **Given the formalization done in prior work, the main results are not unexpected.** First, I would like to underline that central parts of the formalization in this work were transferred from Covert et al., 2021. For instance, the distinction between the removal strategy and the summarization technique and the representation of the methods in this scheme was introduced in this work. I think this should be pronounced more clearly. Further, we assume the model to be Lipschitz-continuous (Assumption 1), the probability divergence when changing the set of present features to be Lipschitz as well (Assumption 3), and we consider only attributions that can be represented as linear operator of the model outputs, it is not very surprising that we arrive at Lipschitz-continuity of the attributions w.r.t. to inputs and the model. While I see that intuitive results also need to be rigorously proven, I am uncertain about how many new insights are added. **Implications and Experiments.** The practical implications of the results still remain unclear to some extent. Taking a practitioner’s perspective, I wonder about what novel insights can be gained from this work. The finding that regularized networks have more robust (or less noisy) attributions, has already been established (e.g., by Shah et al, 2021, Dombrowski et al.). On a sidenote, the Lipschitz-continuity does not imply that the method needs to pass the parameter randomization test, as a constant function would still be Lipschitz-continuous. I would be more interested in some constructive way to operationalize the findings, for instance to prevent “adversarial” explanations as hinted in the text? Can they help us construct better techniques? I think these questions are interesting and to this end, to me, it is seems unfortunate that the experiments already conducted and some further experimentation towards this direction did not make it into the main paper. **Minor Points** A line of related work uses removal strategies to benchmark feature attributions (e.g., Tomsett et al., 2020; Rong et al., 2022), which could be discussed as potential a related work. I wonder if the results in this work bear some connection with attribution evaluation problem, which could be a possible application. Table 2 seems not to be referenced in the text and thus appears a bit context-less. -------------------------------------- **Summary.** A rigorously executed paper with general theoretic results on the robustness of attribution methods. However, the practical implications remain a bit unclear and there are no experiments in the main paper. Overall, the paper seems just above the bar of acceptance to me. I would be willing to further increase my score if the authors can convince me of the practical relevance and impact of their results. **References** Ian C Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. The Journal of Machine Learning Research, 22(1):9477–9566, 2021. Shah, Harshay, Prateek Jain, and Praneeth Netrapalli. "Do input gradients highlight discriminative features?" Advances in Neural Information Processing Systems 34 (2021): 2046-2059. Tomsett, Richard, et al. "Sanity checks for saliency metrics." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020. Rong, Yao, et al. A Consistent and Efficient Evaluation Strategy for Attribution Methods. In International Conference on Machine Learning (pp. 18770-18795). PMLR, 2022 Wang, Jiachen T., and Ruoxi Jia. "Data banzhaf: A robust data valuation framework for machine learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2023.
Questions
I would appreciate if the authors could elaborate on the following points: * How can the results be operationalized in practice? Can something constructive be derived from them? * How would authors quantify the degree of robustness that is desirable (e.g., not invariant, not noisy) * Do the results bear any connections to the benchmarking of feature attributions with is frequently done through removals (e.g., Tomsett et al., 2020; Rong et al, 2022)? A discussion of these points in the paper could help to strengthen this work and I would be willing to reconsider my overall assessment based on the authors' replies.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
2 fair
Limitations
I think the technical limitations are sufficiently clear, and mostly concern the Lipschitz-continuity assumptions and the bounded domains.
Summary
This paper theoretically analyzes the robustness of removal-based feature attributions against input perturbation, and model perturbation with different summary techniques. Empirical experiments on synthetic datasets support their theoretical analyses such as conditional sampling is more robust to model perturbations compared to baseline or marginal samplings. Experiments on real-world datasets (UCI wine quality and MNIST) give more insights regarding the robustness under different model training settings and comparison to gradient-based explanation methods.
Strengths
(1) The proposed theoretical analysis of removal-based feature attributions is technically sound. The input perturbation and model perturbation together with the summary technique cover the different aspects of removal-based attributions. (2) As there are more explanation techniques proposed, this work gives a good starting point to analyze the robustness limitations from a theoretical perspective, which can enable solid explanation evaluation on robustness among different explanation methods. (3) Besides providing theory proofs, empirical experiments also validate and support the findings from theoretical analyses. (4) Messages from the proposed theories are clear. The paper is well written and presented.
Weaknesses
(1) The analyses are only limited to several removal-based feature attribution methods. Extending the analyses to other explanation techniques such as gradient-based explanations would make the contribution stronger. In fact, gradient-based explanations are more popularly used. (2) The project is inspiring and gives insights into different removal-based attributions. However, the impact of the analyses can be broader if the authors can propose a technically sound robustness evaluation benchmark based on the theory. (3) To analyze the robustness is computationally costly if the input data is high-dimensional or the model is huge. The author should consider extending the proposed analyses on complex datasets and some experimental results would make the paper stronger.
Questions
The overall robustness proposed at the end of Section 4, which combines theorems 1 and 2, seems to be synthesized and is hard to adapt into a realistic scenario. For instance, the input perturbation robustness measures the consistency of explanations generated by one model, while model perturbation robustness analyzes the consistency given different (similar) target models. If use the overall robustness, it does not provide a clear message about different removal-based attributions.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Broader impact of the proposed framework should be discussed. More examples to use proposed robustness would be necessary to make the contribution stronger.
Summary
This paper studies the robustness properties of an explanation to small perturbations in the input space (i.e. like an adversarial example) and also to the model parameters. The authors use a number of Lipschitz-style bounds to then derive overall limits on how much explanations can change. Update: As the reviewers have addressed the main concern on the Lipschitz constant, I have no major reason to reject this paper.
Strengths
The bounds are principled and rigorous for explanations, as opposed to heuristic. The paper considers multiple removal techniques and summary statistics to show how the bounds change based on the methods. These bounds can sometimes lead to significant asymptotic differences.
Weaknesses
The elephant in the room is that the results all need some kind of Lipschitz or Lipschitz-like bound, and that the final bound depends on the constant. This constant could be significant and not necessarily ignored, but there does not seem to be any evidence that the actual Lipschitz constant is at all close to being small enough to be useful. There are barely any experimental results in the main paper. Almost the entirety is deferred to the appendix. The results seem to focus mainly on synthetic settings and linear settings where Lipschitz constants and other constants can be directly computed. They also focus on fairly simple datasets such as UCI wine and MNIST.
Questions
Can you measure or estimate the Lipschitz constants and show that these are meaningfully non-vacuous assumptions? It would be better to highlight some kind of experimental result in the main paper. Is there a reason or limiting factor preventing the work from applying to higher dimensional work such as text or image settings?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations discuss some conservativeness of the bounds, which could be addressed with work on certified robustness.
Summary
Previous research has primarily focused on the robustness of gradient-based feature attributions, but the robustness properties of removal-based attribution methods are not well understood. To fill this gap, the authors of the paragraph aim to theoretically analyze and characterize the robustness of removal-based feature attributions. They provide a unified analysis of such methods and prove upper bounds for the difference between intact and perturbed attributions under various settings of input and model perturbation. The authors validate their theoretical findings through empirical experiments on synthetic and real-world data, showcasing their practical implications.
Strengths
1. This paper addresses a crucial XAI problem: Interpretation robustness concerning both input and model perturbations. The study may benefit the derivation of the theoretical impact of some important tricks in XAI, such as baseline value selection and marginal distribution approximation. 2. The derivation of robustness considers input perturbation and model perturbation comprehensively. 3. Section 4 provides valuable insights, revealing that interpretation robustness relies on both the Lipschitz continuity constant and feature number for input perturbation, while it depends on the ∞-norm and feature number in the case of model perturbation. 4. The experiment observations in Section 5 are also enlightening, highlighting the advantages of removal-based attribution over gradient-based attribution.
Weaknesses
The paper should be re-arranged by including experiment results in the main content and moving some theoretical results to the appendix.
Questions
Please refer to the Weakness.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have adequately addressed the limitations of this work in the last section.
Thanks for your reply! 1. I think you may have missed the point about the Lipschitz constant---I understand that it plays a role in the theoretical results, and that it is hard to estimate. I am just looking for evidence that this assumption does not make the results vacuous, especially since networks can have notoriously large Lipschitz constants. It is not necessary to describe the current literature on estimating Lipschitz constants. For example, what you suggested is a great example of what would fit the bill here---if a Lipschitz-constrained network does in fact improve the robustness, then this implies that the constant can in fact be small enough to have non-vacuous implications in practice. 2. You also mention using weight decay to shrink an upper bound on the Lipschitz constant---can you expand more on this? I did not see much about this in the main paper. As weight decay has been used before to improve robustness of the model, it makes sense that this could also help robustness of the attribution, but is there a deeper theoretical link here?
Thank you for getting back to us! We answer your second question first, because it can help clarify the first question. - There is indeed a theoretical link between weight decay and a network’s Lipschitz constant. Weight decay shrinks an affine layer’s Frobenius norm, which upper bounds the layer’s spectral norm; and the product of all layers’ spectral norms upper bounds the network’s Lipschitz constant [R11]. We discussed this in the Experiments section in our Appendix, and in our revision we will be sure to highlight it in the main text. - Like you said, if our theoretical results were vacuous in practice, then reducing the Lipschitz constant of a network (or in practice some proxy to the Lipschitz constant) would not improve attribution robustness. However, we see that regularizing the Frobenius norm of a network, which upper bounds the network Lipschitz constant as discussed above, can indeed empirically improve attribution robustness (see Figure 4 in the Appendix and Figures R1-R2 in our rebuttal pdf). Hence, our theoretical results do bear practical implications. - On the other hand, we do recognize that some networks can potentially have very large Lipschitz constants (assuming we can somehow accurately approximate those Lipschitz constants). In such a situation, our theoretical results suggest that those networks should be avoided because the guarantees for their attribution robustness are not useful. In fact, those networks should also be avoided because the worst-case guarantees for their robustness against general adversarial attacks are not useful neither. This is why there are methods designed for improving the Lipschitz regularity of neural networks [R11-R13].
We would like to thank the reviewer again for the thoughtful review! We hope it is clear in our response that the Lipschitz bound in our analysis is useful. For example, reducing an upper bound of the Lipschitz constant empirically improves attribution robustness to input perturbations, which "fits the bill here" to show non-vacuous implications in practice.
Response to Rebuttal
Thank you for the clarifications. In my remark *"The Lipschitz continuity does not imply that the method needs to pass the parameter randomization test"*, I was referring to line 323-325 in the paper, which seems to suggest that the result in the previous Theorem 2 (robustness of removal-based explanations to model perturbations) already implies that we pass the parameter randomization test. However, as it is an upper bound, we cannot rule out constant attributions even in the case of model perturbation. While this is an interesting finding, I don't think it is a strict consequence of the theoretic analysis. I think if the write-up will be updated to incorporate the above major and minor points, in particular by * including the discussion on applications of the results mentioned here * including at least some experiments on the additional page * clarifying that the formulation of the problem was transferred from Covert et al. (2021) this work will be accessible to the broader audience and a nice contribution, which warrants acceptance at the conference in my opinion.
Thank you for getting back to us and for considering our work a nice contribution! Regarding the sanity check results, we recognize that the connection between our theory and experiments is not as clear as in the input perturbation case. The connection is that if you read the plots from right to left, we see that decreasing the model perturbation decreases the difference in attribution scores, which is implied by our theory via the bound in Theorem 2. However, your point is well taken, because our analysis does not imply that the difference must be large on the far right side; for this, we would require a lower bound rather than an upper bound. For example, for constant-valued attributions we would have $h(\text{summary}) = 0$, so the upper bound in Theorem 2 is always zero regardless of $||f - f’||$. We will substitute the word “naturally” with “empirically” in line 325 to avoid implying such a connection. In any case, given the significance of these sanity checks and the fact that they are the main benchmark for model perturbations, we thought that our findings were interesting and worth including. We will be sure to incorporate all the highlighted points in our revised Discussion and Experiments sections, as outlined in our responses.
Thank you for your response and clarification of my concerns. I think one future direction proposed by the authors, "It would be interesting to pursue a unified, theoretical analysis of the robustness of both classes of methods," would make a valuable contribution to the community. I raised my score to reflect my appreciation for the authors' efforts.
Decision
Accept (poster)