Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning.
Paper
Similar papers
Peer review
Summary
This paper proposes measuring sensitivity to data in some problems in machine learning through what they refer to as the “memory perturbation equation” (MPE), derived using the Bayesian learning rule. They argue that sensitivity properties of machine learning models are not as well understood as they could be in the literature, and therefore propose a remedy to this issue that generalizes some prior approaches. Much of the paper is about justifying the equation and showing that some prior works are special cases. There is also an experimental section that provides some empirical validation of some equations.
Strengths
Although I am not familiar with this general area of research, which seems to be related to influence functions in machine learning, I found the paper interesting and novel. I also found the topic to be highly relevant for NeurIPS and an important one for machine learning in general. The fact that the approach generalizes some prior approaches in a non-trivial way is powerful.
Weaknesses
While I’m generally supportive of this paper, I had a hard time following many of the details in this paper and cannot verify aspects of the approach. This is partly because of my lack of familiarity with the subject, but also because I think the authors have not provided enough background and explanation for a reader with little know-how around related topics. Also, I think the presentation can be improved significantly – please see my detailed questions/comments for some suggestions. I will rely heavily on more knowledgeable reviewers to provide a more thorough assessment.
Questions
Some questions/comments follow: - For the title, I don’t think “model’s” is the most suitable choice of term. I suggest using either “model” or “models’”. - “MPE” also stands for most probable explanation in Bayesian modeling, so this could be a point of confusion for some, without context. - I’m not sure how the word “memory” is suitable for what the authors intend in this work. This is what is written in Section 1: “These highly sensitive examples can be seen as those characterizing the model’s memory; the model is highly sensitive to them and perturbing them can make the model forget its essential knowledge”. Perhaps the authors could explain and justify their choice of term? It may help to connect any other work that they deem to be relevant or inspirational. - For Fig 1, what is the dataset on which a) and b) panels are based on? Are these all for FMNIST? - What is the relation between output function f_i(.) and the terms in equation 1? I don’t think this is ever specified. Is f_i the same as l_i? - Several of the equations (such as those in Section 3) are stated with pointers to prior work but without clear derivation. I understand that there are space limitations, but I had trouble following the correctness of the equations. - There is a claim in line 207 about how the proposed approach is more principled than some prior work – as someone not familiar with the field, I did not observe much justification for this claim, or perhaps just did not understand it clearly. - “Sharpness” is mentioned in line 227 but never defined. There are other such terms that don't get enough explanation. - There are typos, grammatical errors or other such issues in at least the following lines: 39, 41, 59, 61, 63, 68, 110, 113, 124, 161, 221, 236, 244, 249, 268, etc. Note that many of the early typos are due to the incorrect use of verbs for single vs. plural nouns. I suggest the authors carefully review the paper and fix these errors.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Limitations are not mentioned in enough detail.
Summary
This paper studies the problem of the model’s sensitivity to its training data (e.g., the counterfactual of how the model's performance will change if trained without certain data), commonly referred to as the "data influence" problem. This paper presents the "memory perturbation equation (MPE)" to study the model’s sensitivity to such perturbations and uses Bayesian learning rule (BLR) approximated using an exponential family distribution. Then, the BLR can be used to estimate the deviation caused by the removal of a group of data. The paper shows that MPE with Gaussian posterior recovers the celebrated Influence Functions (INF) in both linear and nonlinear cases, and INF can be considered as a special case of the MPE. The work also provides a variety of empirical results on MNIST to CIFAR-10 that shows the estimated sensitivity highly correlates with the actual deviation that would be caused by removing the data and thus can be used as its predictions.
Strengths
This paper is an interesting addition to the research line on "data influence" problems. The paper is well-written. The conceptual development of this paper is clear and smooth. The methodology development and its derivations are solid and the paper also provides insightful remarks supporting the narrative. The method presented in the paper is general and can be potentially applied to a variety of work. Experiment results are nicely presented, clear and interesting.
Weaknesses
I am listing some minor issues that may be improved. In this work, despite strong correlations to the Influence Function have been shown, the comparison to data influence methods in general isn't comprehensive. There are often works talking about recovering linear influence function or using Taylor approximation to extend to nonlinear cases. It would be nice to see how it relates to or compares to broader works, such as [1] or TracIn [2], just to name a few. [1] Repairing Neural Networks by Leaving the Right Past Behind. NeurIPS 2022. [2] Estimating training data influence by tracing gradient descent. NeurIPS 2020. The empirical results of the proposed approach seem isn't especially strong. As we can see, the correlation between prediction and the actual counterfactual deteriorates quickly as the model becomes bigger. It seems quite well for MNIST/MLP, but less satisfactory for LeNet/FMNIST and CNN/CIFAR-10. I appreciate the honesty of authors that show these results in a straightforward manner such that the reader can directly see the real capability of these methods. Because of that, I would think the relatively weak empirical results do not pose a major compromise to the contribution of this work.
Questions
I was wondering about the approximation error for using exponential family distributions or Gaussian priors. It would be nice if the paper can provide some discussions on this to help better understanding. What is the computational overhead of the proposed method, in terms of time complexity and memory demand?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The empirical results are relatively weak. The computational demand and its scalability are unclear.
Summary
In this paper, the authors introduce the Memory Perturbation Equation (MPE), a generalization of sensitivity measures of models that uses the recently proposed Bayesian Learning Rule (BLR) as a foundation. Thanks to its Bayesian foundation, the MPE can be used for used for non-converged models and for non-differentiable loss functions.
Strengths
The biggest strength of the paper is the writing. This is an *extremely* well written paper that conveys the idea beautifully. It flows extremely well and is very easy to read and follow along. The generality of the MPE is also mind boggling. This is an extremely general method that I think will be pivotal for deep learning. Lastly, the experiments section did a great job verifying the theory of the method and giving a taste of all of the potential uses for it.
Weaknesses
While I think the writing is fantastic, I think a small exposition on the BLR would due wonders, especially because the paper cites specific equations form the BLR paper. To make the paper more readable, a small section in the appendix with details relevant to the paper and its experiments would do wonders. While I understand the theory of the method, I have a slight practical disconnect between the experiments from this paper and the BLR. Since Adam naturally arises from the BLR, the posterior variances can be constructed easily and, crucially, the posterior covariance would be diagonal, sidestepping messy matrix inversions. But, from reading the appendix section, it seems like the authors are doing the opposite and are instead getting the Hessian of the loss function (in the appendix the authors write: "Variance computation requires inversion of matrices. For large problems, we use a Kronecker factored Laplace variance approximation as implemented in the laplace package[5]"). If this is indeed the case, then I do find this slightly concerning and is a major limitation of the method as computing Hessians are expensive and practically infeasible for modern architectures. Moreover, it doesn't fall in line of the mantra of the paper which is to leverage the BLR to get these sensitivity measure for practically free. Please let me know I have misunderstood what is going on.
Questions
N/A
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
4 excellent
Limitations
A limitation section is missing from the paper, which I think is important to have.
Summary
The paper studies the sensitivity of machine learning models to training data. Previously, sensitivity was often studied through empirical investigations which was costly and do not always generalize across models. In this paper, the authors present the memory perturbation equation (MPE), based on the Bayesian learning rule, as a unifying equation to understand sensitivity of generic ML algorithms to training data. The MPE has two features: Sensitivities to examples is estimated by using natural gradients of those examples alone, and examples with larger natural gradients contributes more to the sensitivity. The authors show that the MPE when specialized to Gaussian posteriors, recovers influence functions in linear models and deep learning, and examples with high prediction error and predictions variances are the most influential and the sensitivity is obtained by multiplying the two; The sensitivity can be estimated cheaply whenever natural gradients are cheap to compute; Finally, the authors show empirically that the MPE can be used to accurately estimate generalization performance on image classification tasks.
Strengths
- The paper is well written and easy to follow for non-experts. - The proposed MPE formulation is well supported by theoretical foundations based on bayesian learning rule. And according to the authors, the proposed MPE has several nice and intuitive theoretical properties for Gaussian posteriors. - The experiments are well designed to show various aspects of the the properties of the proposed sensitivity measure, which is interesting and clear.
Weaknesses
- I don’t see much discussions on related work or empirical comparisons to state-of-the-art methods - The example datasets used in the empirical evaluations seem to be mostly smaller datasets like MNIST and CIFAR.
Questions
Out of curiosity, can the proposed MPE or part of the proposed perturbation-based formulation be used to perform machine learning data valuation, i.e., to evaluate the importance of groups of data samples towards a machine learning task (e.g., https://bair.berkeley.edu/blog/2019/12/16/data-worth/)
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have adequately addressed the limitations in their discussions section. I don’t see any potential negative impact.
Summary
The paper presents a generic method, relying on some Bayesian analogy, to perform instance-wise sensitivity analysis of learned models. A generic updating rule and derivative is presented, as well as its different variations on popular models (Neural networks and Gaussian models). One notable use of the proposed sensitivity indices is to estimate the genralisation capabilities of a given model. Experiments are performed on classical MNIST/FMNIST data sets to confirm that the behaviour of the method is the expected one.
Strengths
+: a versatile method to perform sensitivity analysis +: a strong technical proposal, at least from what I can assess
Weaknesses
See questions for more detailed comment -: link with Bayesian approach remains to some extent unclear in the general case. -: the approach mainly consider instance-wise sensitivity indices, and it is unclear whether this is sufficient in general? (minor) -: the paper contains a number of typos, and a final read could be useful to correct them (e.g., L41 "can estimated", L63 "the expressions shows", L59, "data as large", L73, "such topics as rare", ...)
Questions
* Bayesian approximation validity: In L92, I found it very strange that one can assume that there is always a proportional link between a Bayesian posterior and the exponential of a Loss function. Sure, the exponential allow to turn the additivity of the loss function into a product form mimicking Bayesian updating, but is it guaranteed that any loss function can be related to a legitimate (proper) prior/likelihood pair in a standard Bayesian framework? Similarly, except for the Gaussian case, what is the quality one can expect from an appriximation using the exponential family? This is a bit hard to figure out, as all examples given in the rest of the paper rest on the (specific) Gaussian case? A good summary of this whole question would be "How Bayesian (in the sense of satiisfying the axioms and coherence of Bayesian approaches, as expressed by, e.g., De Finetti) actually is the BLR rule?" * In echo to my previous question, do we have an idea of how often the premisses of Theorem 2 ($q_\lambda=p(\theta|\mathcal{D})$) are true? Verifying it seems highly non-trivial to me. * Thanks to the underlying assumptions made, it makes perfect sense to consider that perturbation effects are additive and that samples have independent effects on the model. However, it would seem more reasonable to consider that, in practice, samples do have interactions between them and that the impact on the learned model are not merely additive, hence the need to include some possible dependencies/interactions between the samples. Maybe the authors could clarify a bit the situations where the additivity assumption makes sense? * While the removal of an observation is perfectly interpretable from the data perspective, I would appreciate some clarification about the $\epsilon_i$ perturbation. From what I get, this would correspond to weighting the data in the learning scheme, and see what happens in this case. Is this correct, or would it be more accurate to consider that the data can move within a given neighboorhood within the input space (a perturbation that would probably make more sense from a physical viewpoint, even if I am fine with the weighting interpretation). * Suggestion: I do not really see the advantage of repeating figures that can be found in some other places in Figure 1, especially as a reader will not have the necessay elements to properly interpret them at this stage. Maybe remove them to better discuss some other points?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
See questions about some possible limitations, not especially discussed/mentioned in the paper.
Thanks for the response
I thank the authors for their response to my comments. Again, I suggest they edit the paper slightly to make it more readable for a generic reader. I also think more details about terms such as "memory" and "sharpness" are important; hopefully they will make such edits in a revision.
Thanks for the clarification + additional elements
Dear authors, Thank you for your various clarifications and pointers linking Bayesian approaches and loss functions, after which I would gladly raise my score. Regarding the additivity, I gavce a look at Equation (19), however I have the feeling that my point still remain after this. To clarify a bit waht I wanted to say: I did not mean additivity and interactivity in the predictive model, but in the way sensitivities with respect to data removal are computed: from what I get, the MPE obtained by removing two pieces of data amounts to the sum of removing each data point individually. This is the additivity assumption I was mentioning. It may be the case that the covariance matrix mentioned do take care of that, but even in this case it is then approximated by a summative form over each removed data point. Best regards
We are happy to see an increase in the score. Thank you! Regarding the “additivity assumption”, this does not cause a problem in the first approximation in Eq. 19 but (as you point out) only in the second approximation where the sum over examples is used. This is why we did the experiments in Fig. 3a and 3b to understand the effect on a real problem. We will improve our writing to avoid the confusion. To understand why the assumption does not cause a problem, we give the equation below for linear regression where the “exact” deviations are indeed additive. That is, the following holds exactly for the new model $\theta_*^{\backslash \mathcal{M}}$ obtained by removing a group of examples $\mathcal{M}$ from $\theta_*^{}$, $S_{*}^{\backslash \mathcal{M}}\theta_*^{\backslash \mathcal{M}} - S_*^{} \theta_*^{} = - \sum_{i \in \mathcal{M}} x_i y_i, \qquad -\frac{1}{2} S_*^{\backslash \mathcal{M}} + \frac{1}{2} S_*^{} = \frac{1}{2} \sum_{i \in \mathcal{M}} x_i x_i^T$ where $S_*^{\backslash \mathcal{M}}$ denotes the precision for the new model. The expression, when written in terms of the parameters or the function values, has a covariance matrix that correlates examples.
Thanks for the rebuttal
I have read through the reviews as well as the author's response. I appreciate the authors for their dedicated work and thanks for the response to my comments as well as for providing additional results. As has been pointed out by multiple reviewers, the presentation of this paper could be improved for better accessibility to a broader audience. The questions on approximation errors and computational overhead are not directly addressed. The runtime for the experiments conducted in this work or its order of magnitude is unknown. Its practicality cannot be readily assessed. I recommend the authors to better clarify these parts shall the paper be published. I would keep my score as is. I hope the authors compile the new results and additional discussions into the paper or its Appendix. Regards, Reviewer YYDA
We thank the reviewer for the feedback and we appreciate their comments! We will improve the presentation and add new results and additional discussions to the paper. >The questions on approximation errors and computational overhead are not directly addressed. In the rebuttal, we show that a better estimate of variance (better approximation quality) gives better results; see Fig. I in the 1-page PDF. >The runtime for the experiments conducted in this work or its order of magnitude is unknown. Its practicality cannot be readily assessed. The method is practical and scales well to large problems. For example, in Fig. I, we used Adam, which requires less computation than iBLR but gives slightly worse variance estimates. Both require the computation of Jacobian-vector products but can be scaled to large problems as they both use diagonal weight-space covariances. For Fig. 2, 3, and 4a, we analyzed trained models (e.g., to compare to ground truth deviations), so we used KFAC-Laplace which is a bit more computationally expensive but can still scale to large problems.
Thanks to the authors for your rebuttal! I don’t have further questions.
Decision
Accept (poster)