Scalable Influence and Fact Tracing for Large Language Model Pretraining

Thank you for the great suggestions! > "For each fact, there are an average of 2.5 'ground-truth' sentences out of 19.6M sentences total". Does this mean any training or testing examples contain 2.5 relations? The statement needs further clarity. We have updated the wording to: "For each fact, there are an average of 2.5 'ground-truth' *entailing* sentences out of 19.6M sentences total." Each fact is a single relation, but on average 2.5 of the 19.6M T-REx sentences entail a given fact. For example, the fact ("The Old Man and the Sea", "author", "Ernest Hemingway") has relation type "author"; an entailing T-REx sentence is "The Old Man and the Sea is a short novel written by the American author Ernest Hemingway in 1951 in Bimini, Bahamas, and published in 1952". > In the experiments, although the pass rate of records extracted by StarTrack is lower than that of BM25, training a model on StarTrack's examples leads to greater improvement. I wonder if limiting the selection to only the top 3 or top 5 examples—or even just the top example—would reverse this outcome. Great question! Tail-patch results for k=1, 3, and 5 are now reported in Appendix A.4.2. In all cases, TrackStar has a tail-patch score over 2x higher than BM25, in line with the original result. > For C4 evaluation, an NLI model is used. However, the accuracy cannot be fully guaranteed since C4 consists of longer documents. Human annotation of a subset is needed. The entailment model we use is trained for factual consistency evaluation, including training on summarization data with longer premises (https://aclanthology.org/2023.emnlp-main.127/), and 49.0% of C4 passages have length <256 tokens. We have also updated our results to split input C4 passages by sentences (using regex matching, taking the maximum entailment score for a sliding window of three sentences) when evaluating whether a C4 passage entails a fact (Section 4.3). We find that this alleviates the effect where longer C4 passages had high false-positive entailment rates. Based on blinded manual annotation of 100 C4 passages marked as entailing and non-entailing respectively using this method, we find a false positive rate of 13% and a false negative rate of 11%. Of the incorrect annotations, most (54%) are semantically related passages that give a high likelihood of a fact but may not strictly entail it. For example, a passage describing a famous Austrian person does not imply that they were from Vienna, but it is statistically likely.

Thank you for the great questions! > The tail-patch metric used to evaluate influence will probably depend on the templates used. It is unclear how reliable such metric is… I wonder how much will the results rely on these templates? Do you have experiments with different template variations? This is a good point! We have added results with two additional template variations to Appendix A.4.1. For example, the "place of birth" factual relation now has three templates: "[entity0] was born in the city of: [entity1]", "[entity0] was born in [entity1]", and "The birthplace of [entity0] is [entity1]". For all templates, our main trends hold: classical retrieval methods (BM25 and Gecko) perform substantially better for attribution (MRR and recall@10), but TrackStar performs over 2x better for influence (tail-patch score). > If I understood correctly, the hessian approximation proposed requires access to an evaluation dataset. The authors should explain how this would work in the case of no have access to the full evaluation dataset. Yes, the task-specific Hessian approximation requires access to an evaluation dataset. This task-specific approach defines semantic "dimensions" that should be downweighted (in this case, downweighting examples that support the overall task structure rather than an individual fact). When there is no a priori known task, the non-task-specific Hessian can be used, at a slight cost to performance (Experiment 5 in Table 1). More generally, the determination of semantic dimensions that should be downweighted is somewhat subjective – it's not so much that we need the eval data for approximation as that we use a set of examples to define what semantics are most relevant. As a topic for future work, it might be possible to use synthetic data to induce this as well. > It seems to me that lots of "hacks" are needed to get the approach to work (e.g., gradient correction, normalization, gradient compression via projection etc.) But I suppose this is a limitation of influence functions rather than of the proposed technique. Each of the proposed methods is grounded in previous work, practical motivations, and theoretical justification: * Gradient second moment and Hessian correction are theoretically motivated to align gradient dot products with change in loss (https://arxiv.org/abs/2303.14186). This also approximates the Fisher information metric between model gradients. * Gradient compression via projection is necessary to reduce dimensionality enough to store an index of gradients for efficient retrieval. Gaussian random projection has theoretical guarantees from the Johnson-Lindenstrauss lemma, and it has been applied in previous work (https://arxiv.org/abs/2002.08484, https://arxiv.org/abs/2303.14186, https://arxiv.org/abs/2402.04333, see also dimensionality ablations in Section 5.2). * Unit normalization is motivated as $\ell$-relative influence that constrains overall loss change (https://arxiv.org/pdf/2003.11630), reducing the contribution of "loud" examples with high gradient norm. > It is unclear to me that scaling influence functions to LLM is practical/reliable at all, as it seems that spurious correlations between the input fact and the pretraining examples to play a big role. For instance, looking at the errors provided in the Appendix, many of the retrieved proponents seem to completely unrelated to the fact-- some don't even have any word overlap such as example ID 862. The examples in Table A.1 were selected to demonstrate types of proponents we observed – in particular, to show headroom/error patterns – and they are not necessarily representative of the distribution of proponents. We have added Table A.2, where we report the top proponent for a random sample of facts. Across all 5K facts, the top proponent has word overlap with the input or target entity for 84.4% of facts, and there is at least one such proponent in the top 10 for 94.9% of facts. > Could you provide a discussion on real-world scenarios where scaling influence functions to LLM would be helpful/practical? Scaling TDA / influence functions to LLMs can be useful for data curation (selecting pretraining data to target particular evaluation tasks, e.g. https://arxiv.org/pdf/2401.12926, https://arxiv.org/abs/2402.04333), interpreting underlying causes of model predictions (e.g. https://arxiv.org/pdf/2405.13954, https://arxiv.org/abs/2308.03296), and diagnosing misbehaviors (https://arxiv.org/abs/2409.16978) by identifying mislabeled or harmful training examples. As a concrete example, when debugging incorrect factual predictions in our work, we uncovered several ambiguous facts in T-REx. For example, the prediction that Victoria Park is in the United Kingdom (Table A.1) is labeled as incorrect by T-REx, but the top proponents from TrackStar show that there actually is a Victoria Park in the United Kingdom described in several pretraining examples.

Thank you for the helpful comments! > The different design components of the proposed method are mostly adaptations of previously proposed ones, whereas the proposed method is a combination of them. Hence, the technical novelty of this work is a bit limited. Our work combines previous approaches under one framework, and our evaluations demonstrate which components are necessary to scale gradient-based influence methods to realistic settings for LLM pretraining. As far as we are aware, our work is the first to run quantitative evaluations of influence methods at LLM pretraining scale (>1B-parameter models, unfiltered pretraining corpora). > Overall, the proposed designs are quite heuristical and not very solidly justified. For example, why can we enforce a block-diagonal structure for the auto-correlation matrix or perform unit normalization on the input vectors? What are the assumptions for doing these and are there some theoretical basis for them? The block-diagonal structure in the autocorrelation matrix is enforced for computational tractability, in line with previous works (e.g. https://arxiv.org/abs/2308.03296), although in future work it may be possible to relax this. The unit normalization is based both on empirical results (https://arxiv.org/abs/2205.12600, https://arxiv.org/abs/2402.04333, https://arxiv.org/abs/2205.11482, https://arxiv.org/abs/2405.13954) and theoretical justification as $\ell$-relative influence (details in https://arxiv.org/pdf/2003.11630). Theoretically, $\ell$-relative influence identifies training examples that maximize the query example loss change while constraining the overall loss change.

Thank you for the insightful comments! > Tail-patch probability increase works worse with model scale (Fig 2). Do you believe this to be caused by a single instance mattering less for a larger model, or due to the inherent better performance of the underlying model? It would be interesting to distribute these scores for cases where the underlying model was right/wrong in its prediction. We include results split by whether the model was right or wrong in Appendix A.4.3. Indeed, we find that tail-patch scores are over 2x higher for incorrectly-predicted facts vs. correctly-predicted facts. For incorrectly-predicted facts, it appears easier to push the model probability towards the correct prediction because the model does not already have a high correct probability. Thus, the effect where larger models have lower tail-patch scores is likely due at least partially to the higher performance of the larger models (higher "correct" probabilities before tail-patching). We also note that the tail-patch scores for the 8b model do not appear to have plateaued relative to projection dimensionality (Figure 2), which also likely contributes to its lower tail-patch scores. To control for these potential confounds, we only compare tail-patch scores within a given model size.