Emergent and Predictable Memorization in Large Language Models

Summary

This work presents an empirical study on predicting the memorization of examples in large language models. Specifically, using the Pythia model, they test two ideas for memorization prediction: (1) prediction via smaller models, and (2) prediction via partially trained models. Through extensive experiments on models with different scales, this work shows that while both method achieves non-trivial results, neither of them can reliably predict memorization with high recall. Therefore, this paper also includes experiments on selecting the best method w.r.t recall and discovers interesting trends.

Strengths

1. This work is an extensive empirical study on an important and timely topic. 2. The analysis conducted in this paper are highly connected to real use cases and the findings also lead to practical suggestions. 3. This is a very well-written paper with clear motivation, clean organization, and an honest, insightful limitations/future work section. 4. I find the additional analysis in the appendix to be very interesting and can add significant value to this work. I would encourage the authors to move them to the main paper especially considering the current main paper is not very long.

Weaknesses

The limitations section in the main paper is insightful and honest. I agree with everything written there and do not think any of the points is a severe flaw. 1. As the authors acknowledged in the limitations, the use of k-extractible and the memorization score simplifies the problem. In practice, the attacker may not need to extract exactly the same information as the training data. 2. Also mentioned in the limitations, the analysis in this work does not touch data properties or data contents, which are important future works. 3. Also mentioned in the limitation, despite the large amount of experiments done in this work, it is still unsure how the findings will generalize to other models and data. 4. Minor presentation issue: I find the takeaway in LINE 308-310 hard to understand.

Questions

1. Why is there a 6 in Equation 2? 2. For Figure 1 and Figure 3, are the correlations computed using memorization scores or are they computed using 0/1 values? I assume it’s the former, but I don’t think it has been clarified in the paper. 3. Are the precisions and recalls computed at the token level or at the example level? 4. I’m wondering what is the upper bound of the prediction is. In Figure 1, the values at the diagonal are all 1, which makes sense as they are the same model. However, what will that value be if the initialization seed or the data order are different?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have included a great limitations section in Sec. 6 in the main paper.

Summary

This paper discusses the issue of memorization in large language models, which can lead to the output of sensitive data. The authors propose to predict which sequences will be memorized by partially-trained checkpoints. They use the Pythia model in experiments and plot scaling laws for forecasting memorization, allowing them to provide equi-compute recommendations to maximize the reliability of such predictions.

Strengths

1.The paper is well structured and addresses an important problem 2.The memorization score and extractibility are easy and effective

Weaknesses

1.The authors set k=32 and evaluate the first 64 tokens from each sequence because of the computational costs, which deviates greatly from the actual situation 2.They only use the Pythia models for evaluation

Questions

In practical scenarios, how can we know in advance which data should be memorized well (such as factual knowledge) and which data should not be remembered (such as private personal data)? If we know in advance, the proposed method will no longer be needed; if we do not know, how can we improve the reliability of the model specifically after forecasting the distribution of the memorized data points?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately discussed the limitations in the last section.

Summary

This paper introduces the difficulties of both forecasting a future model’s memorization of specific training data and forecasting the memorization behavior of fully-trained LLMs from partially-trained checkpoints. Scaling laws proposed for forecasting memorization, and recommendations for maximizing forecast reliability given a set compute budget to make this prediction.

Strengths

1、The core idea of of scaling laws for forecasting memorization makes sense to me, especially for larger models. 2、Proposed method is quite simple and works very efficient. 3、The paper is clearly motivated and written in a smooth and easy-to-read manner. For example, describes in detail the connections and differences with related work.

Weaknesses

1、The essence of scaling laws is just the method that the author borrowed and tried instead of his own innovation. 2、The model used in the experiment in this paper is relatively uncommon and not convincing.

Questions

Why not utilize LLaMA (Alpaca/Vicuna) or Falcon or other popular LLMs for experiments?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper focuses on predicting memorization of specific data point prior to training the model. The main approach of the paper relies on scaling law, where the memorization property of a large model is predicted using a smaller model, or a partially trained model. The findings are (1) small models are not good at predicting the memorization patterns of large LMs, causing very low recall score. (2) using intermediate checkpoint to predict also attains low recall. (3) the prediction of memorization recall and precision does not follow the scaling law.

Strengths

1. The setting of predicting whether a specific data is memorized is novel and interesting. The problem is well-motivated. 2. The approached tried in the paper are reasonable: predicting the memorization behavior of a large model using a smaller LM, using a intermediate checkpoint, and fit a scaling law curve.

Weaknesses

1. none of the proposed approaches produce the desirable recall that's needed for this approach to be viable. Even though there is a scaling law pattern, the best recall still seem to be not good enough. 2. If the developer know which data is sensitive, can they just remove the data from training distribution, and how much would that hurt performance? If they can just remove these data without harming performance, the basic motivation of this problem might be questioned.

Questions

see above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

n/a

Summary

This work empirically predict the memorization behaviors as measured by k-extractible and memorization scores. They tested on two settings 1) use small model to predict the memorization behavior or larger model, 2) use partially trained model to predict the fully trained model. The main finding is that 2) is has much higher correlation than 1). Also included scaling laws and discussions on emergent memorization.

Strengths

* Clearly motivated and clean experiments with clear conclusions * Well written and flows well * interesting scaling law for emergent memorization A clean paper with a clear message, addressing an important enough problem should be accepted.

Weaknesses

* Not too obvious if the methods are good enough for practitioners to predict undesirable memorization due to lack of absolute metrics, in particular with respect to PII or something else people care about. Seems like random sequences were used in the experiments, which may have different behavior than PII * Only tested on Pythia model suite and limited range of data (partly addressed in limitations).

Questions

The heatmap is figure 3 is a wash, some variant of the Hinton diagram might show such correlation more clearly than color scales.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes