Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs

Summary

This paper proposes a `goldfish loss', a loss that excludes some tokens in each training data sequence from the loss computation, with the aim of decreasing verbatim memorization of sequences. Which token is excluded is decided with a function G(x_i). The authors try two different drop functions, one of which drops 1/k tokens randomly, and other drops every kth tokens. To ensure similar tokens get dropped for duplicate documents, in the second method, a hashing approach is used to make sure that if a token is dropped, preceeding occurrences of that token will be dropped as well if they share h=13 previous tokens. To test this loss, the authors first train a 7B Llama2 model over 100 wikipedia documents for 100 epochs that in total have around 200K tokens. Without goldfish scenario, the verbatim memorization rate is 84/100 articles, while with a goldfish loss with k=4 no articles get memorized. Next, the authors train a TinyLlama-1.1B model on a subset of RedPajama v2 (single epoch), with around a 2-4M wikipedia tokens mied in, repeated 50 times in random locations to simulate data duplication. The total amount of tokens trained on is 20B. They show that the goldfish loss substantially decreases the memorization rates in this training setup. They show that it has little to no effect on a range of benchmark scores (though those scores are often at chance). The also show that model strained with goldfish loss have Mauve scores similar to the control models.

Strengths

- This paper proposes a simple loss that can be used to reduce memorization. It's simplicity will make it more likely that the loss could actually be adapted - The loss seems to be very effective in reducing memorization rates, while not impacting validation loss - The paper is well written and easy to follow

Weaknesses

- It is a bit unclear to me why it is necessary to mix in the wikipedia sequences at such a high rate. RedPajamas is intended to provide a reasonable version of a training corpus, why not just train on that alone? Mixing in data with a repetition rate of 50 times is quite unusual, I wouldn't call that 'normal training conditions'. It makes me wonder if the effect is less strong if the method is applied to red pajamas alone. - Several of the benchmark scores are barely above chance level (e.g. Winogrande has a chance performance of 50%, performance of the trained models is hardly above that), while this strictly speaking supports the statement that there is no performance drop, a consistent random performance is no evidence that the method does notnegatively impact performance. While the scores are above chance for several other benchmarks (e.g. Arc-C performance is 40%, while chance is 25%, PiQA performance is around 62%, chance is 50%), the scores are low, making it difficult to judge the impact of the method on performances.

Questions

- Did I understand correctly that you mix in 2-4M tokens 50 times, so that 100-200M = 0.5% of the total corpus is heavily repeated corpora (that may also be included in the RedPajama corpus)? Why do you call that 'normal training conditions'? Wouldn't training on RedPajama itself be normal training conditions? - I do not really understand why the gold fish loss (per supervised token count) would be lower than the regul ar loss. Do you have an explanation for that?

Rating

5

Confidence

4

Soundness

2

Presentation

4

Contribution

2

Limitations

- One limitation that I would like the authors to address is my point about the chance level performance on benchmarks (though it may be better to just remove the benchmarks). Perhaps it would be worth including a few benchmarks that are better at small scale as well? - I don't think there are any negative societal impacts of the work.

Summary

This paper tackles the issue of memorization in large language models (LLMs), where models reproduce verbatim training data, posing copyright, privacy, and legal risks. The authors propose a novel technique called "goldfish loss" to mitigate this problem during training. Instead of calculating the next-token prediction loss on all input tokens, goldfish loss computes it on a pseudo-random subset, preventing the model from learning entire training sequences verbatim. The paper demonstrates the effectiveness of goldfish loss in reducing extractable sequences with minimal impact on downstream benchmark performance and language modeling ability.

Strengths

- The goldfish loss is a simple yet effective technique for mitigating memorization during training, distinct from existing post-training methods. - Introducing the robust handling of duplicate passages with hashing is a very interesting technique that makes the goldfish loss possible in practice. - The paper provides strong empirical evidence supporting the effectiveness of goldfish loss in reducing memorization, particularly in extreme scenarios designed to induce memorization with minimal impact on downstream tasks. - The simplicity and ease of integration of the goldfish loss into existing training pipelines make it a viable solution for real-world LLM development.

Weaknesses

- While empirically effective, the goldfish loss lacks theoretical guarantees regarding the complete prevention of memorization. The paper acknowledges this limitation and highlights the possibility of adversarial extraction techniques circumventing the goldfish loss. - The paper would benefit from a more in-depth discussion on the computational complexity introduced by the hashing mechanism used for robust duplicate handling. Analyzing the trade-off between hash context size (h) and memorization prevention, as well as the computational overhead of hashing, would strengthen the paper.

Questions

- he paper explores different values of k (drop frequency) and h (hash context size). A more detailed analysis of the sensitivity of goldfish loss to these hyperparameters would be beneficial. How do different values affect memorization prevention and downstream performance?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

- The paper does not include an analysis of the computational overhead introduced by the hashing mechanism.

Summary

This work presents a new goldfish loss to mitigate memorization during pretraining. Specifically, for every k tokens during training, the loss for one token is skipped to prevent the exact memorization of the entire string. Experiments on 7B llama-2 and 1.1B tinyllama demonstrate that the goldfish loss can effectively reduce memorization while maintaining performance on downstream tasks.

Strengths

1. The idea is conceptually simple and very easy to implement. 2. The goldfish loss can effectively reduce the risk of exact memorization. This is demonstrated in multiple experiments in this paper and through multiple different metrics. Even for more advanced attacked (e.g., membership inference attack, beam search, etc.), the proposed method also show significant improvement.

Weaknesses

1. The main empirical results to support the claim that goldfish loss will not hurt downstream performance in Figure 4. While I do see in the figure that the goldfish loss can achieve similar performance as the standard loss, the stand loss itself also only achieves very little gain over the control model (with BoolQ being the only exception). Therefore, the only real empirical result is that goldfish loss will not hurt gain on BoolQ, which is still good evidence but insufficient to support the main claim of the paper. 2. While I'm confident that the goldfish loss can prevent exact memorization, I'm not sure if it actually prevents the model from learning the content that the model is not supposed to learn. It is very likely that the model learns the content, but can only generate it in a slightly paraphrased way. If this is really the case, the impact of this paper seems limited. There is no related investigation or discussion in this paper. 3. There are some inclarities about the experiment settings (see questions below).

Questions

1. Why are some of the experiments conducted on llama-2 and others on tinyllama? 2. How do you implement the hash function mentioned at line 128? 3. Do you also apply the goldfish loss on RedPajama? 4. Can you explain more about why the gold loss work even when sampling with zero temperature (i.e., greedy decoding)? In such a case, the toy example from line 104-113 will not work, as the token with the maximum probability remains the same. Is it the case that the generalization probability on those tokens in fact quite small?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors provide a good discussion of limitations in Sec. 7

Summary

This work focuses on the issue of memorization, where a language model can be prompted to exactly repeat a document or sequence from its training data. The authors introduce a loss which masks random tokens in a sequence during training. To ensure the same mask is applied to duplications of a sequence, or a same sequence in a different context, they always use the same mask for the same words using a hash table. They then show through different training paradigms that their loss reduces memorization while having minimal impact on performance and training time.

Strengths

Strengths A very simple but efficient approach, adapted to the specificity of the exact memorization framework. Tests are quite relevant and show very strong performance, even in adverserial settings (and aknowledge weaknesses to those). Main usage drawbacks are mentioned and convinvingly argued to be a small cost for high gain : absence of guarantees, and a need for a small amount of additional training. Math is clear, work is reproducible, and experiments address main concerns and baselines relevant to the field.

Weaknesses

(1) Results on benchmarks seem to show that while verbatim repetition is avoided, knowledge is still retained (Fig. 4). Do you have any qualitative pointers on output variation between goldfish and non goldfish loss in those settings? Is the output a paraphrase of learnt text? My worry is that the strict "exact copy" metrics used to evaluate the loss might be too artificial, and might not give a complete idea of data leakage. While this deviates from the definition of memorization in 2.1, motivation in introduction would push for analysis in this direction. (the privacy/copyright/PII motivation is harmed where sensitive data might still filter through, using different words).

Questions

Mostly a curiosity question: Do you have any idea of the impact on the model’s generalisation capabilities? Similarly to the reasoning behind dropout, the proposed loss seems to avoid a form of overfitting, and might refocus learning on higher level representations.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

While I am quite impressed by the actual results and work, my main concern (and reason for ethical concerns) is the way motivation is explained. Trials have shown in Europe, (and to my knowledge in the US) usage of copyrighted data in a for-profit setting without authorisation is forbidden (ex: RGPD compliance, copyrighted books.). This includes training the model with this data, even if it does not output exactly the same data word for word. The proposed loss, especially with the high performance reported in the paper could therefore have the negative outcome of hiding this unlawful usage. In the introduction (and to a lesser extent the conclusion), it sometimes appears that this is argued to be a feature. → In the “copyright risk for the consumer” example, it seems to be argued that if the model does not reproduce the unauthorised copyrighted data, they will be protected from copyright infringement. It is my understanding that using a tool made with that code also falls under copyright. → The “copyright risk for the provider” example follows a similar idea. Making profit as creator/provider/user of a tool made with copyrighted code falls under copyright. (On the other hand, the privacy risk example seems a very interesting and promising use case). I strongly recommend clarifying both in the text, and in either limitation or some ethics paragraph that the usage of non-authorised or stolen private data is forbidden, and a potential misuse of this work.