Sample based Explanations via Generalized Representers

Summary

The paper proposes a unifying framework for sample-based explanation methods via generalised representers. The framework proposes to approximate a general nonlinear predictive function using a surrogate from the Reproducing Kernel Hilbert Space (RKHS) (Surrogate function $f(x)=\sum_{i=1}\phi(x_i,x)=\sum_{i=1}\alpha k(x_i,x)$). The underlying explanation function $\phi$ is both necessary and sufficient to satisfy a given set of axioms (aka efficiency, continuity, self-explanation, symmetric zero, symmetric cycle, irreducibility axioms) and can be decomposed in two parts, namely a global component, viz. the alpha coefficients (constant over the whole input space), and a local one, viz. the kernel similarity between the evaluating point and each training instance (which clearly depends on the input). Existing sample-based explanation frameworks, including TracInCP, influence functions and representer points, are shown to be particular instantiations of the proposed framework depending on the choice of the alpha coefficients and the kernel function. Additionally, the framework suggests a new sample-based explanation strategy to reduce the computational storage burden of TracInCP. Experiments are conducted over a CNN model on CIFAR10 and MNIST, thus empirically highlighting the benefits of different choices of alpha coefficients and kernel functions.

Strengths

1. The idea of unifying existing sample-based explanation approaches is elegant, original and novel ( **Originality** ) 2. Overall, the paper is clear and self-contained ( **Clarity** ). Perhaps, the presentation can be improved by taking into account some of the suggestions highlighted in the Questions section.

Weaknesses

1. Code is not available. It would be good to release the code for replicating the analysis ( **Reproducibility** ). 2. While the theory unifying different sample-based explanation approaches is nice, the observations that can be drawn from it are rather limited to the analysis on the choice of different alpha/kernel functions ( **Significance** ). 3. The experimental analysis uses rather small and simple neural networks and represents a simple proof of concept (**Quality**). Since the authors compare against TracInCP, they should consider using the same experimental settings of the main paper (namely using ResNet architecture).

Questions

Overall, I liked the overall formalisation of sample-based explanation techniques. Can you please elaborate more on the following two aspects? 1. Additional considerations/insights that can be drawn from the analysis. 2. Scope of validity of experimental results. Are the observations valid for instance on other and larger network architectures than simple CNNs? Please, find below a list of possible points to improve the presentation: 1. Is Method 2 in Section 4.2 necessary for the presentation? 2. Line 206 should $\nabla_{\theta}$ be replaced with $\nabla_{f_\theta}$? 3. Line 222 replace the correct reference and Line 224 remove “show” 4. Would it better to summarise sample-based explanation strategies in the form of a table, thus highlight the different choices of alpha and kernel functions and the difference with the proposed strategy? 5. Can you please highlight more the difference between TracInCP and your strategy?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No additional suggestions to improve the paper

Summary

In this study, the authors conducted an axiomatic analysis of a measure that quantifies the influence of a given training data on predictions. Under several axioms, the authors demonstrated that an effective measure of influence is limited to the form of a suitable coefficient multiplied by a continuous and positive definite kernel function. Based on this finding, the authors showed that many existing influence metrics can actually be expressed in the form of a suitable coefficient multiplied by a kernel function. Furthermore, the authors proposed a new measure by combining Representer Point Selection and Neural Tangent Kernel.

Strengths

The strength of this study lies in the axiomatic analysis of the measure of data influence. Under Continuity Axiom, Self-Explanation Axiom, Symmetric Zero Axiom, Symmetric Cycle Axiom, and Irreducibility Axiom, the authors demonstrated that an effective measure of influence is limited to the form of a suitable coefficient multiplied by a continuous and positive definite kernel function. Furthermore, based on this finding, the authors showed that many existing metrics for measuring influence can indeed be expressed in the form of a suitable coefficient multiplied by a kernel function. The reorganization of these existing influence metrics from an axiomatic perspective represents a novel and significant contribution of this study.

Weaknesses

An essential weakness of this study is the insufficient discussion regarding the validity of various axioms. While Continuity Axiom appears to naturally require the continuity of the measure, the validity of the other axioms, Self-Explanation Axiom, Symmetric Zero Axiom, Symmetric Cycle Axiom, and Irreducibility Axiom, is not necessarily evident from the current discussions in the paper. In fact, Data Shapley [8] employs different axioms. Since the choice of axioms determines the appropriate measure, the discussion of the validity of these axioms becomes crucial in the axiomatic analysis. While the authors provide some intuitive explanations, they seem insufficient as a discussion on the validity of these axioms. For example, what are the similarities and differences between the axioms employed in Data Shapley [8] and the axioms considered in this study?

Questions

* Please discuss the validity of the axioms introduced in the paper. When they are appropriate and when they may be not? * What are the similarities and differences between the axioms employed in Data Shapley [8] and the axioms considered in this study? --- I have read the authors' rebuttal. The difference of the current study and Data Shapley [8] is partly solved. I strongly believe it should be discussed in detail in the paper.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors mentioned some possible future directions that are not addressed in the current study.

Summary

This paper studies a new framework for generating sample-based explanations for black-box machine learning algorithms. To explain a black-box model, the basic idea of sample-based explanations is to quantify how each training data is influencing the prediction of certain test data. The main contribution of this work is to give a natural set of axioms for defining explanation functional, and showed the equivalence of every explanation function that satisfies these axioms with Mercer kernels. Under this framework, the paper studies how to define the importance function based on given kernel functions and discusses some popular choices of kernel functions in the context of deep learning.

Strengths

This paper studies a very interesting and important question within the realm of interpretable (explainable) machine learning, making a very neat connection between a natural set of axioms for the explanation functional and kernel functions. The study is substantial and showcases promising results from numerical studies.

Weaknesses

The authors may benefit from expanding their discussion on the selection of kernels. For instance, the Inf-final kernel outperforms all other kernels in Table 1 but lags significantly in Table 2, according to the supplementary material. Given this inconsistency, it seems slightly misleading to simply promote the influential kernel in the main body of the paper without a more thorough discussion.

Questions

The paper would also be strengthened by additional discourse on the axioms, which are a key element of the study. How do existing sample-based representers fare against these axioms? Furthermore, is the axiom set minimal in its essence? For example, is it possible that the continuity axiom may not hold in some practical situation? The kernel function doesn't have to be continuous without the continuity axiom. Although the authors present the tracking representer as "a more scalable variant" due to its reduced computational burden, it consistently outperforms the other two approaches. Could this be partially attributed to early stopping? In addition, the target derivative is proposed as an approximation to avoid solving the Reproducing Kernel Hilbert Space (RKHS) regression, but it also consistently outperforms the original surrogate derivative. It would be beneficial to have further commentary on these points.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations.

Summary

This paper studies and proposes a set of desirable axioms for sample based explanations. This further demonstrates that the only solution satisfying the set of desirable axioms has the form of $\alpha_i K(x_i,x)$ (i.e., the product of two components: a global sample importance, and a local sample importance that is a kernel similarity between a training sample and the test point) where $K(.,.)$ is a psd kernel function. Moreover, Many existing sample-based explanation methods such as representer point selection [7], influence functions [5], and TracIn [6] can be viewed as specific instances of the broad class of generalized representers.

Strengths

The set of desirable axioms makes sense and the link between kernel theory and sample based explanations through set of desirable axioms is interesting. The viewpoints to connect the sample based explanations through the set of desirable axioms and the existing sample-based explanation methods is interesting.

Weaknesses

Training and learning a kernel model to approximate a given black-box function are challenging and computationally expensive. Method 2: approximation using the target function is not convincing to me because it has a loose connection to the kernel theory and the local sample importance is unclear in this method. Method 3 seems to be most practical, but it requires the feature map $\Phi$ is fixed. Regarding, Kernel 1: Penultimate-layer Embeddings, it is still unclear to me how to store $\alpha$ and compute the score $\alpha_i K(x_i,x)$ if the feature map $\Phi_{\theta_1}$ is updated all the times and we have multiple kernels along with the training progress. For Kernel 2 and 3, the proposed approach serves as a tool to explain [5] and [6] rather than introduce a new approach. The experiments conduct only for binary classification with limited data although the conclusion is interesting.

Questions

Please address my questions in the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not address the limitations and potential negative societal impact of their work

Thanks for Answers; Additional Clarifications

Thank you for the answers, which address most of my points/questions. I went also through the other reviews and the remaining concerns can be summarized as follows: 1. Scope of experiments and experimental methodology 2. Motivation about axioms and similarities/differences compared to Data Shapley (as suggested by reviewer SFDZ) 3. Valorizing novelty of proposed unified framework Regarding point 1, experiments on language data are a nice addition and therefore appreciated. However, the major concern about the experimental methodology and comparison with TracInCP is not well addressed yet. While I understand and emphatize the authors for the lack of computational resources required by running larger scale experiments, I think that the analysis I'm hoping to see is still feasible and would strengthen the validity and scope of the results. Indeed, the comparison with TracInCP (for instance using ResNet architecture) can be still conducted by choosing an off-the-shelf pre-trained model without incurring in large computation. Indeed, in the experiments, you almost always leverage the last checkpoint and the analysis is mostly focussed on the computation of the coefficients with fixed feature map. Can you please elaborate more on this aspect? Regarding point 2, I agree and therefore support author's answers. Indeed, different axioms lead to different forms of explanation functions. The axioms provided in this paper aim at unifying previous sample-based explanations and establish a connection with kernel functions. However, I'm curious to see what is the opinion of reviewer SFDZ. Regarding point 3, it is still unclear to me what are the novel insights that can be drawn from the unifying framework and what are possible directions for future research. In the hope of valorizing your nice analysis, could you please comment on this aspect? Perhaps the following table might seed some thought on addressing this question... | Choice of kernel / Choice of coefficients | Method 1 (solving regression) | Method 2 (no regression, replacement of surrogate) | Method 3 (solving regression and tracking gradients) | |---|---|---|---| | Kernel 1 (neural embeddings Eq. 9) | Representer Point Selection | ? | ? | | Kernel 2 (neural tangent kernel Eq. 11) | ? | ? | TracInCP | | Kernel 3 (influence functions Eq. 13) | ? | Influence functions | ? |

Reply

I would like to thank the authros for adding detailed discussions. The relationship between Data Shapley and Generalized Representer will be an interesting and important topic. In the response, "additional focus in generalized representers of explaining a model prediction on a particular test sample" is the key point making these two studies different. However, if I understand correctly, we can use Data Shapley for the same purpose by choosing the performance score function $V(\theta)$ as the predictted output on a particular test sample. If this is the case, how the axioms of Data Shapley and the axioms of Generalized Representers differ in this particular setting?