A Bayesian Approach To Analysing Training Data Attribution In Deep Learning

Training data attribution (TDA) techniques find influential training data for the model's prediction on the test data of interest. They approximate the impact of down- or up-weighting a particular training sample. While conceptually useful, they are hardly applicable to deep models in practice, particularly because of their sensitivity to different model initialisation. In this paper, we introduce a Bayesian perspective on the TDA task, where the learned model is treated as a Bayesian posterior and the TDA estimates as random variables. From this novel viewpoint, we observe that the influence of an individual training sample is often overshadowed by the noise stemming from model initialisation and SGD batch composition. Based on this observation, we argue that TDA can only be reliably used for explaining deep model predictions that are consistently influenced by certain training data, independent of other noise factors. Our experiments demonstrate the rarity of such noise-independent training-test data pairs but confirm their existence. We recommend that future researchers and practitioners trust TDA estimates only in such cases. Further, we find a disagreement between ground truth and estimated TDA distributions and encourage future work to study this gap. Code is provided at https://github.com/ElisaNguyen/bayesian-tda.

Paper

References (34)

Scroll for more · 22 remaining

Similar papers

Peer review

Reviewer wqES5/10 · confidence 3/52023-06-28

Summary

The paper studies the challenges in measuring the performance of training data attribution (TDA) methods arising from stochasticity in training large deep neural networks. Specifically, the authors use various existing approaches to obtain many samples from the posteriors of the model weights instead of a point estimate with and without a training instance whose influence needs to be computed. The authors then compute the mean and variance of ground truth attribution/influence scores; and measure the performance of various attribution methods by looking at the correlation of these mean and variance with the ground truth scores.

Strengths

- Thorough experiments comparing various Leave-One-Out (LOO) or attribution approximation methods including influence scores and variants (GD, GS), additional training step (ATS). - The paper provides further empirical evidence to previous observations [1] that common approximations used for LOO actually estimate slightly different objects and are susceptible to randomness in training arising from initialization of model weights, batch order, etc.

Weaknesses

- It is not clear if the phrase Bayesian perspective on TDA is useful, authors could say more on the connection between using Bayesian DL methods, Student t-test for measuring the noise in TDA estimates and Bayesian perspective on TDA. - In eq 8, how are the T samples from the two posteriors (with and without a training instance) paired. - It is also not clear if the observations made here only apply to measuring the attribution performance wrt to the ground truth attributions. That is, the main recommendation to focus on low signal to noise pairs seemly to apply only to evaluation. Did the authors perform such an evaluation? How does this recommendation affect the use of influence scores downstream applications such as correcting label mistakes, removing biased instances, etc. [1] Bae, Juhan, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B. Grosse. "If Influence Functions are the Answer, Then What is the Question?." Advances in Neural Information Processing Systems 35 (2022): 17953-17967.

Questions

See Weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes, limitations are discussed.

Reviewer fqXw4/10 · confidence 4/52023-07-04

Summary

This paper aims to examine training data attribution (TDA) methods from a Bayesian perspective, assuming the learned model parameters are samples from a posterior distribution. The paper illustrates how this perspective might affect TDA methods. It conducts experiments comparing and contrasting different TDA methods when run several times on model parameter samples from an approximate posterior. Experiments consider subsampled versions of MNIST and CIFAR10, and examine two model classes CNN and ViT+LoRA.

Strengths

The topic is very interesting. As a researcher and practitioner that has worked extensively with TDA methods, I have found myself thinking about (and navigating) the relationship between randomness in model parameters and TDA methods. I think the community stands to benefit from this line of investigation. The paper communicates many of its ideas quite clearly. Figures 1 and 2 are especially helpful. At a high level, the experimental setup is a sensible approach. Anonymized source code is included.

Weaknesses

The paper is lacking in its theoretical treatment of the subject it aims to explore (effectively limited to Figure 1). This reduces its value to the community. Conclusions are drawn based on experiments conducted via (relatively small) samples from a few simple approximations of the posterior. The paper would benefit considerably from some mathematical analysis and discussion of how a Bayesian model or perspective might affect the validity of TDA methods, such as influence functions. For example, how does treating $\theta$ as a random variable affect the application of the implicit function theorem used to derive influence? The Hessian of the loss is a function of $\theta$, for some set of values in the support of $\theta$ this Hessian is not invertible. Does this set have zero measure? Can it be safely ignored? It seems that there are numerous details that go unexamined. Overall the writing is quite clear. But crucially, I found the presentation of the test statistic (Equation 11) to be confusing. As such it was difficult to interpret (and thus to review) the subsequent experimental results. (See questions below. I continue the review assuming that the p-values quantify the probability that the samples of $\tau$ could be generated by a random variable having mean equal to 0.) At a high level, the experimental setup makes sense to me. However, some of the analytical steps in processing the collected data seem weak. For instance, on line 160 it is stated that “[this] treatment [...] poses a novel challenge for evaluation”. Yet if I understood correctly, the experiments yield samples from random variables that are compared in pairs, $\tau$ and $\tau’$. There are numerous established ways to compare two sample-sets, and to quantify the probability that they come from the same underlying distribution. A two sample t-test may be an option. Without much theoretical analysis to rest on, the experiments on two small datasets and two models struggle to convince me of the general claims. For example, the experiments on training set size (Section involve only three size settings, and don’t show clear trends. While the discussion on model complexity only considers two models which differ in both size and architecture. Influence functions have been successfully used to reshape model behavior through the removal of identified sets of training instances. Several of the works cited in this paper achieve this. Therefore, the IF approximation must correlate with LOO at least on average for some types of training instances. How can this be reconciled with the results (e.g. in Figure 5)? This is not really discussed. Can they be explained via low noise pairs? Minor: Line 81: While Koh & Liang showed that IFs could be applied to neural networks, they did not derive the method. I suggest citing the original robust statistics papers on IF and/or the infinitesimal jackknife. Figures 3 and 4 would benefit from larger labels.

Questions

Regarding the presentation of the statistical test: In Equation 7 $\tau$ is redefined as a random variable. The randomness stems from $\theta$ conditioned on the observed dataset and the removal of j. (It might be helpful to indicate that with the notation.) In Equation 10, it states that the hypotheses being tested are whether \tau does or does not equal 0. This is unclear. I assume these hypotheses refer to $\tau$’s mean, and that the t-test is intended to determine the probability that the samples of $\tau$ could be generated by a random variable having mean equal to 0. Is this correct? In Equation 11, what version of the T-test are you using? It would be helpful to cite your statistical method. Is the test designed to consider the pairing of the posterior (and perturbed posterior) samples? In Equation 11, I assume that t should be indexed by j (and is also a function of the test sample z), but it’s unclear whether $t$ is computed for every pair of posterior (and perturbed posterior) samples. The left-most term of the numerator suggests this. If so, how are you aggregating these sets of values to get a single value per train-test pair? Other: Is it common to combine these approximate posterior sampling techniques? What are the implications of doing so?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The experiments in this paper consider two models. The paper notes limitations in the size of the datasets considered, explaining that it opted to spend computation time/budget on an exhaustive analysis of TDA values instead. However, if the objective was to comment on issues with TDA in practice, this choice (and limitation) is considerable. The results would be more convincing if the experiments had focused on only a subset of the test-train pairs, but considered larger (more realistic) dataset sizes, and a broader set of model architectures and/or sizes.

Reviewer fmRD7/10 · confidence 3/52023-07-06

Summary

This paper presents a Bayesian perspective on Training Data Attribution (TDA), a technique that identifies influential training data for model predictions. The authors propose treating the learned model as a Bayesian posterior and TDA estimates as random variables. This approach reveals that the influence of individual training data often gets overshadowed by noise from model initialization and batch composition. Consequently, they suggest using TDA only when certain training data consistently influences model predictions, despite noise factors.

Strengths

- Well written, easy to follow. Thanks! - Studies an important practical problem and provides a practically relevant recommendations. - Strong analysis of experiments

Weaknesses

- The paper identifies difficulties with current practice and provides some recommendations, it does not discuss a solution or paths towards a solution of the TDA problem.

Questions

l 65: what do you mean by "more global"?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

n.a.

Reviewer CbTs6/10 · confidence 4/52023-07-06

Summary

In this paper the authors investigate training data attribution (TDA) through a bayesian lens by explicitly considering the randomness in estimating the model parameters with and without a given training example. To generate approximate bayesian posteriors on model parameters the authors use deep ensembles and SWA, then consider multiple different methods of TDA. They find that, using a t-test, TDA estimates often exhibit high variance, indicating high signal to noise ratio. This variability is often dominated by noise from model initialization and seems to increase with training set size and model complexity. They also show strong consistency between different groups of TDA methods.

Strengths

The manuscript is well written and clear, and the problem setting important. Measuring the variability in downstream TDA estimates caused by variability in model parameters is an interesting perspective that adds to the growing body of literature aiming to understand various methods of TDA. The conclusions drawn by the authors regarding sources of randomness (initialization vs batching) seem to align with current intuitions around TDA estimates and present a new way to measure the quality of any proposed TDA method by considering those pairs for which the LOO estimates are statistically significant.

Weaknesses

One of the weaknesses is the small dataset sizes considered. The authors restrict datasets to 150 training examples and 900 test examples for MNIST and 500 train/test examples for CIFAR10. At these small dataset sizes, we might expect higher variability and correlation between training and test examples. For example, if the model only sees a single 3 written a particular way during training, then we might expect high variability in differently trained model’s predictions at a similar 3 in the test set. However, seeing many such 3s in a larger dataset should decrease the variability of posterior predictions. Although the conclusions drawn and trends observed at these small scales are interesting, it remains to be seen whether they hold for larger datasets. As a starting point, the authors should consider running the same experiments on a larger training set but consider only a small random subsample in their analysis. This should give a stronger signal for the behavior of the random variable $\tau$ for larger datasets. The other main set of experiments that would greatly strengthen the paper is the consideration of the downstream tasks for which TDA is useful, for example in mislabel identification. Since these tasks may not depend on exact LOO estimates [1], the inherent variability in TDA estimates may or may not be a relevant consideration. It would be interesting to see the extent to which the variance of $\tau$ estimates makes these tasks difficult or even potentially ill-posed. [1] Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question?

Questions

- Is there a hypothesis for why so many more MNIST3 examples seem to have low p-values compared to CIFAR10 examples? It would be interesting to see whether these low p-value examples have some special properties that could be exploited to identify them efficiently. - It seems that the model complexity analysis should be considered relative to the dataset we are attempting to perform TDA on. Is there any way to quantify this value? Although a ViT model may exhibit high $\tau$ variability on a simple dataset like CIFAR10 or MNIST3, it may behave differently on a much more complex datasets like ImageNet.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Reviewer fmRD2023-08-14

Thank you

I appreciate the explanation.

Reviewer fqXw2023-08-15

I appreciate the authors taking the time to write a detailed rebuttal and to provide additional experiments. However, many of my concerns remain unaddressed, e.g., "it is unclear which additional theoretical treatment would add value": I struggle with this response. As a researcher and practitioner who regularly uses these methods, I am saying that the paper hasn’t provided me with much insight beyond what I have already observed empirically or read in related works. I therefore question the extent of its value to the community. I agree that in practice TDA methods show a great deal of stochasticity. At least when applied to non-convex models optimized via SGD. Perhaps, what I’m struggling most with here is the framing. If the paper was simply positioned as saying: “User be warned, most TDA values in neural networks are dominated by retraining noise”, then I would be less critical. (Albeit variations of this point have been made before). But the paper is positioned as a “Bayesian perspective” on TDA, and as such I was expecting the subject to be investigated more theoretically, and I was hoping it might present some insights into the observed stochastic behaviors of these methods, e.g., provide a theoretical explanation for the training data points which are consistently high influence for a test point (what you call low-noise pairs). Moreover, what if the reader were interested in a simpler model class, like linear regression or logistic regression? These are important use cases for TDA methods. But influence functions are not “inherently stochastic” in linear regression (nor in logistic regression–assuming sufficient optimization). The practitioner would need to consider the application of the TDA methods to the Bayesian version of these methods, which aim to explicitly model uncertainty in the parameters. This is left unaddressed, e.g., how would one apply IF to simple Bayesian models (ones where parameter uncertainty is explicitly represented)? “Even in the deterministic setup, the IFT and inverse Hessians are generally inapplicable due to the non-convexity and high dimensionality of the deep learning optimization problem. They are not novel challenges introduced by our probabilistic treatment.” I agree that some TDAs already suffer theoretical issues when applied to neural networks. However, this is not a reason to avoid explicitly naming them and discussing their implications. In practice, these shortcoming are often circumnavigated by either continuing to train from an optimized checkpoint, or by keeping all but the last layer parameters frozen (e.g., Koh and Liang), effectively leading to a logistic regression model over a fixed feature extractor. Neither of these approaches are examined or discussed. “We add a damping term λ to ensure the positive definiteness (PD) of Hessian”: This damping term is an example of something that may be interesting to examine theoretically, could it be connected to the prior in a Bayesian learning framework? “we ran an experiment with a 3-layer CNN that matches ViT+LoRA in #trainable parameters. The results confirm that increasing model complexity decreases the statistical significance of TDA estimates ” : There are still only 3 observations here. When the dataset size experiments were increased from 3 observations to 6 observations the trend changed. What makes the authors believe that this claim about model complexity is more robust? I still struggle with the setup of the main statistical test. As far as I can tell, Equation 11 does not correspond to an unpaired, 1 sample student’s t-test. Assuming the p-values were nonetheless correctly calculated, it’s still unclear whether this test is appropriate given the mixed nature of the posterior samples. The samples are not IID. The “seed” samples from DE are arguably IID, but the subsequent samples from SWA are not. This should at least be discussed. As such I stand by my original score.

Authorsrebuttal2023-08-18

Response to reviewer (1/2)

We thank the reviewer for the follow-up comments and suggestions that will improve the quality of our paper. > I am saying that the paper hasn’t provided me with much insight beyond what I have already observed empirically or read in related works [...] Perhaps, what I’m struggling most with here is the framing. [T]he paper is positioned as a “Bayesian perspective” on TDA, and as such I was expecting the subject to be investigated more theoretically. Thank you for the clarification and we are glad to confirm that the reviewer shares a similar experience regarding the stochasticity of TDA. We fully agree that our framing is broader than our main focus and this is potentially a critical issue. We propose to address this issue by changing the title to delineate the scope more precisely and convey the message more concretely: “A Bayesian Approach to Analysing Training Data Attribution in Deep Learning”. We will also update the abstract and introduction to make clear that our scope is the application of TDA on deep learning models. We would like to clarify further that we do not aim for a theoretical contribution. We study how the stochasticity in the training process of deep models fluctuates the TDA estimates. We apply prominent Bayesian deep learning approaches like Deep Ensemble and Stochastic Weight Averaging to model the randomness in the deep learning training procedure. We find that training process stochasticity has a considerable effect on TDA, as we observe significant variances in the scores. We further find that the tested approximation methods fail to fully capture the ground-truth variance. Based on this observation, we recommend evaluating the approximation methods for not only replicating the mean leave-one-out (LOO) measure but also the variance therein. We do such an evaluation for not only influence function (IF) but also other approximate TDA techniques like additional training step (ATS), grad-dot (GD) or TracIn, and grad-cos (GC). > I was hoping it might present some insights into [...] e.g., provide a theoretical explanation for the training data points which are consistently high influence for a test point [and] how would one apply IF to simple Bayesian models [...]. Moreover, what if the reader were interested in a simpler model class, like linear regression or logistic regression? We agree that probing the link between sample traits and TDA noise is important. However, even for simple model classes, characterizing TDA value distribution is inherently complex. For example, even if we assume a simple Gaussian posterior for the learned parameter $p(\theta| X)=\mathcal{N}(\mu,\Sigma)$, the resulting IF value for logistic regression corresponds to $$ IF(z_\text{test},z) -y_\text{test} y \cdot \sigma(-y_\text{test}\theta^\top x_\text{test})\cdot\sigma(-y\theta^\top x)\cdot x_\text{test}^\top H_\sigma^{-1} x$$ (Koh & Liang 2017). where $\sigma(\cdot)$ is the sigmoid function. Note that the product of the sigmoids of Gaussian random variables does not yield a tractable distribution that can be represented with a closed-form mean and variance formulae. Likewise, linear regression with an $\ell^2$-loss results in the following formula for IF: $$ IF(z_\text{test},z) = - (x_\text{test}^T H^{-1} x) (x_\text{test} \cdot \theta - y) (x\cdot \theta - y) $$ Here, again, the product of two Gaussian random variables does not yield a distribution with tractable mean and variance, making it rather non-trivial to analyse the theoretical behaviour. > influence functions are not “inherently stochastic” in linear regression (nor in logistic regression–assuming sufficient optimization) We agree that we need to be careful with this statement and that influence functions are not “inherently stochastic”. We apologise. In the paper, we will explicitly restrict our scope to deep learning models. > However, this is not a reason to avoid explicitly naming them and discussing their implications. Yes, agreed. We will discuss the mentioned issues with influence functions in the manuscript in §3, under paragraph *TDA methods likewise estimate random quantities*.

Authorsrebuttal2023-08-18

Response to reviewer (2/2)

> In practice, these shortcoming are often circumnavigated by either continuing to train from an optimized checkpoint, or by keeping all but the last layer parameters frozen (e.g., Koh and Liang), effectively leading to a logistic regression model over a fixed feature extractor. Neither of these approaches are examined or discussed. The mentioned approaches (i.e. continuing to train from optimized parameter, freezing the model up to the last layer) reduce the stochasticity in the parameter. However, as the reviewer would be aware from _Bae et al._’s work (_If Influence Functions are the Answer, Then What is the Question?_), these approaches lead to a gap between influence scores and what they set out to measure, which is the leave-one-out retraining (LOO). In our study, we aim to explore the variance in TDA scores based on the original LOO definition, and observe how various TDA approximation methods capture this variability. > “We add a damping term λ to ensure the positive definiteness (PD) of Hessian”: This damping term is an example of something that may be interesting to examine theoretically, could it be connected to the prior in a Bayesian learning framework?: The damping term could indeed be seen as an isotropic Gaussian prior centred at the origin. We thank the reviewer for pointing this out and will add this comment to the paper. > When the dataset size experiments were increased from 3 observations to 6 observations the trend changed. The relation between training set size and the variability of TDA scores is indeed not linear, and we will update §4.3 *Training set size* for the final version of the paper. We would like to correct this observation by stating that an increased training set size leads to an increased variability in TDA scores *up until a certain point*. We observe that TDA scores tend to be smaller with larger training sets: | Training set size | Mean TDA score (LOO) | Mean variance of LOO | | --- | -------------------- | -------------------- | | 30 | 0.242 | 0.098 | | 60 | 0.042 | 0.041 | | 90 | 0.048 | 0.093 | | 120 | 0.073 | 0.091 | | 150 | 0.030 | 0.019 | | 180 | 0.017 | 0.019 | We believe this makes sense because a single training sample tends to attribute less to test samples overall, as the training set size increases. If all train-test pairs consistently exhibit low TDA scores, the variance in scores decreases, resulting in smaller p-values, which we observe as the drop in p-values for larger training sets, such as $|D|\in{150,180}$ in additional experiment 3 of the rebuttal pdf. > What makes the authors believe that this claim about model complexity is more robust? We believe that our claim about model complexity (more complex models tend to exhibit larger variability in their TDA scores) is robust to changes in the observed trend because we consistently observe that model complexity mainly affects the variance in TDA scores more strongly than the mean score itself: | Model | Mean TDA score (LOO) | Mean LOO variance | | ----------- | -------------------- | ----------------- | | 2-Layer CNN | 0.030 | 0.019 | | 3-Layer CNN | 0.047 | 0.045 | | ViT+LoRA | 0.058 | 0.224 | This observation aligns with the intuition of using a mostly frozen model to compute IFs (Koh & Liang, 2017): With more parameters, we increase the stochasticity of the training procedure, in turn diminishing the reliability of TDA scores. > As far as I can tell, Equation 11 does not correspond to an unpaired, 1 sample student’s t-test. We use a 1-sample Student’s t-test for a single random variable, TDA score $\tau$. There is no pairing of samples. Note that $z_j$ and $z$ are fixed variables in Equation 11. > The samples are not IID. The “seed” samples from DE are arguably IID, but the subsequent samples from SWA are not. This should at least be discussed. Sure, for the mixture-of-Gaussian posterior, we do not sample IID. Instead, we perform a version of stratified sampling, where we fix the number of samples from each centroid. Within each centroid, the sampling is IID, as the reviewer has pointed out. We believe that the stratified sampling approach used in this study is unlikely to introduce significant bias in our statistical analysis. This is because we have ensured an equal number of samples from each stratum, and the strata (centroids) themselves are exchangeable, meaning that their order does not affect the overall outcome of the analysis. We thank the reviewer for this critical observation. We will include this discussion.

Reviewer wqES2023-08-17

Thanks for the rebuttal

I would like to thank the authors for additional experiments and responding to my questions.

Reviewer CbTs2023-08-18

Thanks for the detailed response. After reading the other reviewer responses and thinking more about this work myself I have a few more reservations. The additional experiments address some of my concerns but it is still difficult to support the claims made since the noise could be stemming from many different sources that are difficult to disentangle in these settings. Although, I recognize the difficulty in attaining gold-standard LOO retraining TDA estimates, we know from [2] that LOO does not necessarily correlate with IF (and the other methods as well given the analysis in Figure 5), so comparing the two is a bit misleading, especially as neural network sizes grow larger and this discrepancy increases. One option is to use PBRF training, as in [1, 2], which we know corresponds well with IF. Applying the same analysis to this training regime would give us a more clear picture of the stochasticity of IF. As it stands, the observed variability is not so surprising when we consider what the existing TDA methods measure as compared to true LOO retraining. Another consideration is that TDA methods are often used to analyze the predictions of a **specific** model, rather than a general class of models. In these cases we care only about the specific sample from the unknown model posterior and not the posterior itself. The suggested analysis above of the PBRF training regime would more closely align with this problem setting. In light of these concerns I will be adjusting my score down one point. [1] Studying Large Language Model Generalization with Influence Functions. Roger Grosse ̊:, Juhan Bae ̊:, Cem Anil ̊ et al. 2023. [2] Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question?

Authorsrebuttal2023-08-20

We thank the reviewer for the great additional points to the original review. > [...] it is still difficult to support the claims made since the noise could be stemming from many different sources that are difficult to disentangle in these settings. We agree that in practice, the source of noise will be diverse and potentially difficult to disentangle. But in our experimental setting, we used random initialisation and batch composition as sources of noise, which could effectively be controlled: “In a variant of DE with the initialisation as the source of randomness (\textbf{DE-Init}), we train each of $T_\text{DE}$ randomly initialised parameters $\theta^{(t)}\_0$ on either $\mathcal{D}$ or $\mathcal{D}\_{\setminus j}$. (...) We also consider the batch composition in stochastic gradient descent (SGD) as the source of randomness (\textbf{DE-Batch}). (...)” [§4.1, lines 181-186] Controlling only these two factors are sufficient to support the claim that training process stochasticity (via random initialisation and batch composition) leads to variance in TDA scores [§4.5, lines 287-288]. > [...] we know from [2] that LOO does not necessarily correlate with IF [...], so comparing the two is a bit misleading, especially as neural network sizes grow larger and this discrepancy increases. We thank the reviewer for initiating an interesting discussion around the seminal work of Bae et al. (2022). We are aware that IF (one of the tested TDA methods in this work) does not exactly correspond to LOO from Bae et al. (2022) and the discrepancy increases with larger networks, as the reviewer has described. Nevertheless, we strongly believe that the ultimate goal of TDA is to predict the counterfactual outcome of removing a training sample, rather than the modified PBRF objective in Bae et al. (2022). We thus argue that it is necessary to make an empirical comparison between the ground-truth LOO and approximate TDA methods like IF, precisely to quantify the aforementioned discrepancy. [§4.5, lines 291-293]. Please let us know if there is any misunderstanding on the reviewer’s comments from our side. > [...] the observed variability is not so surprising [considering] what the existing TDA methods measure as compared to true LOO retraining. We are not completely sure if we understand the reviewer’s comment correctly. Our understanding of the reviewer’s comment is: because existing TDA methods are very crude approximations of the true LOO retraining, it is indeed unsurprising that TDA methods show great variability. We believe there could be a bit of confusion. Our main point is that even the true LOO retraining, which is considered the ground-truth target for approximate TDA methods, exhibits stochasticity: “Generally, we observe many TDA measurements, ground-truth and estimations likewise, are unstable with non-significant p-values (> 0.05).” [§4.2, lines 204-205]. Because the ground-truth target is stochastic, our point is that we need to adjust our evaluation protocol to embrace the inherent stochasticity of the task and treat TDA values as random variables [§4.5, lines 287-293]. We measure the Pearson and Spearman correlations of TDA estimate’s mean and variance to study how well approximate TDA methods capture the ground-truth LOO scores in both mean and variance. We’d be happy to discuss further if we have not understood the reviewer’s comment correctly. > Another consideration is that TDA methods are often used to analyze the predictions of a specific model, rather than a general class of models. In these cases we care only about the specific sample from the unknown model posterior and not the posterior itself. This is a great point. Indeed, in practice, the starting point would be a fixed trained model. It would also make sense to use a fixed model trained on the original dataset $\theta_\mathcal{D}$, rather than the posterior $\theta\sim p(\theta|\mathcal{D})$. However, to answer the question “how does the model output change when a sample $z_j$ is removed from the training set?”, we should inevitably introduce multiple possibilities for the counterfactual model $\theta_{\mathcal{D}\setminus j}$. There is no well-defined notion of a *unique* trained model obtained from re-training a model without a certain sample because the exclusion distorts the batch composition, for example. Such an ambiguity in $\theta_{\mathcal{D}\setminus j}$ is well-captured via Bayesian model posterior $\theta\sim p(\theta|\mathcal{D}_{\setminus j})$. We extend this posterior viewpoint to the model built from the original dataset $\theta\sim p(\theta|\mathcal{D})$, but one could still consider treating this as a fixed variable. We will discuss this option in the final version. We are glad to be able to continue the constructive discussion. We will incorporate the discussion here to the final manuscript. In case there is any misunderstanding from our side or the reviewer has any follow-up questions, we will remain available.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC