Response to reviewer (2/2)
> In practice, these shortcoming are often circumnavigated by either continuing to train from an optimized checkpoint, or by keeping all but the last layer parameters frozen (e.g., Koh and Liang), effectively leading to a logistic regression model over a fixed feature extractor. Neither of these approaches are examined or discussed.
The mentioned approaches (i.e. continuing to train from optimized parameter, freezing the model up to the last layer) reduce the stochasticity in the parameter. However, as the reviewer would be aware from _Bae et al._’s work (_If Influence Functions are the Answer, Then What is the Question?_), these approaches lead to a gap between influence scores and what they set out to measure, which is the leave-one-out retraining (LOO). In our study, we aim to explore the variance in TDA scores based on the original LOO definition, and observe how various TDA approximation methods capture this variability.
> “We add a damping term λ to ensure the positive definiteness (PD) of Hessian”: This damping term is an example of something that may be interesting to examine theoretically, could it be connected to the prior in a Bayesian learning framework?:
The damping term could indeed be seen as an isotropic Gaussian prior centred at the origin. We thank the reviewer for pointing this out and will add this comment to the paper.
> When the dataset size experiments were increased from 3 observations to 6 observations the trend changed.
The relation between training set size and the variability of TDA scores is indeed not linear, and we will update §4.3 *Training set size* for the final version of the paper. We would like to correct this observation by stating that an increased training set size leads to an increased variability in TDA scores *up until a certain point*. We observe that TDA scores tend to be smaller with larger training sets:
| Training set size | Mean TDA score (LOO) | Mean variance of LOO |
| --- | -------------------- | -------------------- |
| 30 | 0.242 | 0.098 |
| 60 | 0.042 | 0.041 |
| 90 | 0.048 | 0.093 |
| 120 | 0.073 | 0.091 |
| 150 | 0.030 | 0.019 |
| 180 | 0.017 | 0.019 |
We believe this makes sense because a single training sample tends to attribute less to test samples overall, as the training set size increases. If all train-test pairs consistently exhibit low TDA scores, the variance in scores decreases, resulting in smaller p-values, which we observe as the drop in p-values for larger training sets, such as $|D|\in{150,180}$ in additional experiment 3 of the rebuttal pdf.
> What makes the authors believe that this claim about model complexity is more robust?
We believe that our claim about model complexity (more complex models tend to exhibit larger variability in their TDA scores) is robust to changes in the observed trend because we consistently observe that model complexity mainly affects the variance in TDA scores more strongly than the mean score itself:
| Model | Mean TDA score (LOO) | Mean LOO variance |
| ----------- | -------------------- | ----------------- |
| 2-Layer CNN | 0.030 | 0.019 |
| 3-Layer CNN | 0.047 | 0.045 |
| ViT+LoRA | 0.058 | 0.224 |
This observation aligns with the intuition of using a mostly frozen model to compute IFs (Koh & Liang, 2017): With more parameters, we increase the stochasticity of the training procedure, in turn diminishing the reliability of TDA scores.
> As far as I can tell, Equation 11 does not correspond to an unpaired, 1 sample student’s t-test.
We use a 1-sample Student’s t-test for a single random variable, TDA score $\tau$. There is no pairing of samples. Note that $z_j$ and $z$ are fixed variables in Equation 11.
> The samples are not IID. The “seed” samples from DE are arguably IID, but the subsequent samples from SWA are not. This should at least be discussed.
Sure, for the mixture-of-Gaussian posterior, we do not sample IID. Instead, we perform a version of stratified sampling, where we fix the number of samples from each centroid. Within each centroid, the sampling is IID, as the reviewer has pointed out. We believe that the stratified sampling approach used in this study is unlikely to introduce significant bias in our statistical analysis. This is because we have ensured an equal number of samples from each stratum, and the strata (centroids) themselves are exchangeable, meaning that their order does not affect the overall outcome of the analysis. We thank the reviewer for this critical observation. We will include this discussion.