Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. Understanding this memorization is important in real world applications and also from a learning-theoretical perspective. An open question in previous studies of language model memorization is how to filter out "common" memorization. In fact, most memorization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training. We identify and study counterfactually-memorized training examples in standard text datasets. We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.
Paper
References (61)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
This paper defines counterfactual memorization as the difference between a model's expected performance on a sample when it is included in the training data and that when it is not. The performance is measured by per-token prediction accuracy. The paper studies common patterns across counterfactually memorized samples. It further defines counterfactual influence similarly to counterfactual memorization, except that the performance is measured on any sample, and studies the correlation between counterfactual memorization of a sample and its counterfactual influence maximized over some set of data.
Strengths
- It is interesting to study whether the existence of one training document will affect model predictions. - The mathematical definition in Eq. (1) - (4) are reasonable. - Ablation studies including 3.3 and 3.4 are well-explained.
Weaknesses
Overall: - The research problem can be more crisply defined. It is unclear what type of information the paper aims to study. Overall per-token top-1 accuracy is not sophisticated enough to be reflective of memorization of "sensitive information" (L2) or "details of specific events" (L111). - There are technically mistaken statements. E.g. L32 says counterfactual memorization is a measure of an expected change. But it is indeed the difference between two expected values where the expectations are over different distributions. - The quantitative definition about which information is considered "common" or "rare" is unclear (L48). While the paper uses 2^21 documents (L95), only 38000 are identified as being a near-duplicate with at least one other example (L199). Is two near-duplicated examples considered common, and one considered rare? The 38k seems to be a small portion and it's unclear if the notion of "common" vs. "rare" is significant enough to study. - It is unclear what the common patterns found across counterfactually memorized text are (L50). - Please use "counterfactual memorization" instead of simply "memorization", whenever proper, to avoid confusion. Analyzing Counterfactual Memorization: - There are contradicting claims. E.g. L107 says high (counterfactual) memorization happens for unconventional texts and multilingual texts, while L126 says ill-formatted text or foreign languages are hard and L125 says hard examples have low counterfactual memorization scores. - There are non-refutable claims. E.g. L123 says easy examples tend to have low scores. "Simple samples" are defined as those with high accuracy, and two high accuracy scores should apparent have a relatively small difference. From Memorization to Influence: - Authors claim that small memorization scores have small max-influence scores and larger influence scores requires larger memorization scores (L253-254), but there doesn't seem to be a concentration of densities around the x=y line in Figure 4(b). Some typos: - L21: revious $\rightarrow$ Previous - L38: considerded $\rightarrow$ considered - L109: most often $\rightarrow$ are most often
Questions
- L22-25: Are sensitive data, such as, phone numbers and usernames, "common" and frequently occurring? - L30: How do you quantitatively define "common" and "rare"? - L38-39: Why is the memorization of $x$ $\textit{counterfactual}$? - L38-39: Could you clarify what "predicts $x$ accurately" means? It is implausible that given an initial token, an autoregressive language model will be able to generate an entire document $x$ perfectly via greedy decoding. - L75: Does |D| refer to the number of documents? - L93: But T5-base has ~220M parameters, instead of ~112M. - There seems to be no control over the number of subsets S and the number of subsets S' being similar and both big. It is possible that a training sample appears in all 400 samples. Then the expected measure M is a bad approximation. What's a theoretical error bound? Please also see Weaknesses. If the review misses important aspects, please feel free to address them.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The authors discussed limitations but not ethical concerns. Both are reasonable.
Summary
This paper introduces a concept "counterfactual memorization," defined as the anticipated change in a model's output when a specific training example is omitted. Experimental analysis was conducted on three widely employed text corpora in the domain of language modeling, and the phenomenon of memorization was identified in all instances. Furthermore, the authors assess the impact of each memorized training example on the validation set and generated texts. This novel approach offers direct evidence pinpointing the origin of memorization during testing. Despite these intriguing insights, the paper has notable limitations. The experiments performed lack robustness and some of the assertions made are lacking in clarity.
Strengths
1.The paper well-written and easy to follow. 2.It is of great significance to explore memorization in neural language models. 3.The proposed new definitions of can serve as references for future works, and the experimental analysis is detailed and interesting.
Weaknesses
1.The paper lacks a section on related work, which should summarize and analyze the current state and challenges of relevant research. 2.This paper only uses the Transformer-based language backbone equivalent to T5-base (encoder-decoder framework). Can the experimental results be applied to other architectures and models of different scales? 3.Some typos need to be fixed. For example, “revious” (line 21) should be “Revious”. The word at the beginning of a paragraph is capitalized.
Questions
1.Does the result of the model depend on the size of the subset ? Is the result sensitive to the value of r? How did the author determine that r=0.25? 2.Some domains are represented much more frequently than others in the datasets we studied (Line 136). What are these domains, and what are their characteristics?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have adequately addressed the limitations.
Summary
This paper proposes a metric for whether a sequence is memorized based on the degree to which exposure during training increases the probability of producing the sample.
Strengths
The proposal here is strong, and remedies a real problem in the memorization literature: a failure to consider the *inherent* predictability of any given sequence (a property which this paper calls "simplicity"). This is a significant gap that I'm happy to see work on. The results cover basic questions such as the impact of the number of epochs. They also explain the reasoning behind choosing the number of model runs they apply their method on, and this analysis is sound. I also like that they measured not only the impact of a given sample on the model's tendency to emit itself but also its influence on the model's tendency to emit a different sample.
Weaknesses
As far as I can tell, it is hard to actually get a counterfactual metric for memorization of highly duplicated sequences because they are more likely to appear across different training subsamples. This metric inherently favors rare samples, because prior exposure to the duplicated samples will decrease the impact of each additional exposure. This paper doesn't distinguish between "simplicity" due to duplication and simplicity due to inherently predictable sequences, like repetitions of a single character. "the choice of data has a relatively minor effect on memorization" but the corpora used are all ones that avoid code and multilinguality, favoring natural English language text. It is not clear that this property would hold for significantly different corpora. The authors acknowledge this issue to some degree, pointing out that one of the largest differences between the datasets is likely due to the higher-quality texts in Wiki40B. The authors only qualitatively check their conjecture that "if a validation set example x0 contains similar information, infl(x ) x0) could be large", but it should be fairly easy to quantify whether the similarity between any pair of samples increases the influence. Minor: - fair number of typos, please proofread before publication
Questions
I'm curious about counterfactual memorization with respect to the training timeline. This paper uses training subsampling as a way of controlling whether a particular sequence has occurred, but I think it is worth asking whether that sequence was counterfactually memorized when the model actually encountered it for the first time, which may be at different points of training. Given a model trained on a known ordering of its corpus, we could consider the impact of a sample in the context of training by comparing checkpoints before and after exposure. Have you considered this question, explored to some degree by Biderman et al. (https://arxiv.org/pdf/2304.11158.pdf)? I would love to see results if so.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
4 excellent
Limitations
The corpora studied are all fairly similar (focusing on natural language English text), so the results are limited to those settings. The authors don't acknowledge this as completely as I'd like, although they do mention that higher quality text has different behavior than a corpus with malformatted text or multilingual contamination. The authors also acknowledge that counterfactual memorization is inherently going to assign less memorization value to sequences with near-duplicates throughout the corpus, and describe this as a decision about what type of memorization they consider. The authors also acknowledge that the models they are using are much smaller than the large language models of interest today.
Summary
This work studies memorization in neural language models. The major scientific question in this work is how to filter out common memorization in language models. To this end, this work first formulates a notion of counterfactual memorization which characterizes how a model’s predictions change if a particular document is omitted during training. Then, this work estimates counterfactual memorization on three standard text datasets, and confirms that rare memorized examples exist in all of them. In addition, this work also identifies an inverse correlation between number of duplicates and counterfactual memorization as compared with previous definitions of memorization. Finally, this work extends the definition of counterfactual memorization to counterfactual influence, and studies the impact of memorized examples on the test-time prediction of the validation set examples and generated examples. Overall, this paper is well-written and flows smoothly. This work begins with a detailed definition of counterfactual memorization and then estimates and analyzes counterfactual memorization of training examples in standard datasets. The experiments identify the examples sampled at high, intermediate and low memorization. Analysis covers the impact of number of models, number of training epochs. This work also shows the potential for inference attacks in large language models in a learning theoretical perspective.
Strengths
1. This is an interesting paper that analyzes the counterfactual memorization in neural language models. The paper is written in a clear and concise manner. The problem is well defined, and the analysis is technically sound with insightful findings. 2. This work extends the definition of counterfactual memorization to counterfactual influence, and studies the impact of memorized examples on the test-time prediction of the validation set examples and generated examples --- thus has the potential to provide new perspectives of studying the inference attack problem in large language models.
Weaknesses
1. The efficiency of the proposed approach can be improved for a practical use. Currently, identifying the memorization levels of the training examples, as well as the counterfactual influence, requires extensive computation overheads by training multiple models --- which may become a major concern for its application in large language models and with larger scale training sets. 2. The experiments can be also broaden to leveraging different backbone language models to provide more useful insights towards how existing language models vary in the effects of counterfactual memorization. This may lead to an intuition on how those language models would be sensitive to inference attacks and stimulates the development of defense approaches.
Questions
Line 21: "revious" -> "previous"
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have provided detailed and useful descriptions about the limitation.
Summary
The study proposes two novel metrics: “counterfactual memorization” and “counterfactual influence”, the first one can measure how a model’s predictions change if a sample is omitted in training, and the later one measure how a sample in the training set influences the prediction of a validation sample. The authors also analyze the two metrics on 3 standard datasets and discover some interesting results, like “different data sources display different memorization profiles”, ”number of models needs”, “memorization does quantitatively decrease when data is repeated more often”, the strong influencer ”also only receive tiny influence from all the rest of the training examples”.
Strengths
1. Great motivation and important research question, how to measure a sample’s influence on the trained model’s prediction and the sample’s influence on a validation sample. 2. The counterfactual viewpoint is interesting and easy to understand. “counterfactual memorization” and “counterfactual influence” are useful metrics. 3. The analysis and some findings are insightful for inspiring future study. 4. The presentation is clear and easy to follow.
Weaknesses
1. I am surprise that the authors do not refer the causal inferences studies, since “counterfactual” is an important concept in statistical causality. Judea Pearl, Donald Rubin and many scholars did many great work about counterfactual and causal effect, such as structural causal model, potential outcome model [2]. From the perspective of causality, the equation 1 and 3 could be reformulated by causal effect, counterfactual inference, which are important research topic and have been studies well in statistical causality communities. So I suggest the authors can study some classical causality studies [3][4], which can help you to make the research better. 2. I am curious the proposed two metrics on simple models, such logistic regression, SVM. Large language model (LLM) is popular and important, while the simple models are also useful and efficient for many applications. In the meantime, I think the simple models, like logistic regression, are much easier to measuring the two metrics, since the LLM is complex and have more unobserved confounders. So my second suggestion is that the authors could try some simple models. 3. It is much clearer to summarize your analysis and findings in one table. 4. Typos: e.g. revious in line 21. [1] Causal inference in statistics: An overview, J Pearl [2] Causal Inference Using Potential Outcomes, DB Rubin,2005 [3] Causality: Models, Reasoning and Inference, J Pearl 2009 [4] Causal Inference for Statistics, Social, and Biomedical Sciences, G.W. Imbens, 2015
Questions
1. The authors could study some classical causality studies [3][4], which can help you to make the research better; 2. The authors could try some simple models and do the similar analysis.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Besides the limitations the authors list in section 7, the authors also mentions that it is hard to evaluate the impact of a single training example for the costly computation, I hope the authors could compare their methods with influence function in [5], which can identify training sample most responsible for a given prediction, and discuss their advantages and disadvantages.
Summary
This paper formulates a notion of counterfactual memorization to measure the "one-point" generalization performance of the model (they do not convey this definition as this). Equipped with this definition, the authors conduct plenty of experiments to explore the components that are related to such counterfactual memorization of large language models.
Strengths
This paper is well-written, and the analysis of what kind of samples tend to be memorized is interesting. The experiments in this paper are systematic as many of the related components are considered.
Weaknesses
Despite the strengths mentioned above. I think this paper has the following weakness. 1) The definition of counterfactual memorization is nothing but the generalization performance of the model on one single point. In fact, taking expectation to $x$, the counterfactual memorization becomes expected generalization. Thus, in my opinion, studying it from the perspective of distribution is exactly studying the generalization of LLM, which obviously is not a new topic. 2) The definition of counterfactual memorization is quite similar to some classical notations in generalization theory, e.g., algorithmic stability and $\epsilon$-differential privacy. 3) Here is my major concern. The performance of the model is evaluated by the 0-1 prediction loss on the task of next-word prediction (NWP). However, I think this is not a reasonable metric to measure the performance of NWP. The metric is reasonable for the classification task. However, for the NWP task, I do not think there exists a "ground-truth" label as in classification. For example, the sequence "I like eating _" followed by the words "apple" or "banana" are all valid predictions.
Questions
As the authors clarified, "Both the neural language models and training sets used in this work are orders of magnitude smaller than modern standards such as GPT-3". Though conducting experiments on these LLM is impossible. My question is, does the samples that are proven to be easily memorized can not have diverse patterns during inference, even for larger models?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
As the authors clarified in their paper, they have no experiments on larger-scale pre-trained model.
Summary
The paper studies an important problem of the "memorizing" effect in neural language models, particularly focusing on rare or isolated pieces of information in the training data that get memorized, as opposed to widely duplicated or common information. It formulates a notion of "counterfactual memorization" that measures how a model's predictions change when a particular training example is excluded during training. It analyzes counterfactual memorization on several standard text datasets and studies patterns in what gets memorized. Examples with unconventional text formats tend to have high memorization. It extends the definition to "counterfactual influence" which quantifies the impact of a memorized training example on the inference phase, which helps trace the source of predictions. Overall, the paper provides a novel perspective and tools to systematically study the memorization of information in neural language models.
Strengths
This paper presents a novel and significant perspective on studying information flow in neural language models. The notion of counterfactual memorization is an original contribution to quantifying the memorization of isolated pieces of information. This moves beyond prior work that focused more on duplicated content. Extending this to counterfactual influence to trace predictions back to training examples provides a new capability. The mathematical formulation of counterfactual memorization is clear and well-motivated. The empirical methodology is sound, with adequate models trained to converge and sensitivity analysis on the number of models needed. The results support the claims and provide new insights into memorization. Overall, I find this a very solid, insightful paper with multiple novel contributions to an important problem.
Weaknesses
One potential weakness of this paper is the application of such analysis remains elusive. I can imagine many interesting questions could be answered with the provided concept, e.g., what kind of stereotypes/social biases in the model are influenced by what data in the training samples?
Questions
The core idea of counterfactual memorization is intuitive, but can you provide some theoretical justification for why the formulation in Equation 1 captures the memorization of information? Is there any formal argument that you can make?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
Yes.
Response
Thanks for the authors' responses. The response addresses some of my concerns. But my major concern "The performance of the model is evaluated by the 0-1 prediction loss on the task of next-word prediction (NWP)" does not been addressed. The authors admit that there exist many possible words in the NWP task, and they said ambiguity exists in the image classification task as well. However, the main difference between the image classification task and the NWP task is that during the training and inference stage, "underlying ground truth" labels do not exist in the training stage. For example, suppose we have an image classification model which is trained to recognize the species of animal, we will not expect it to recognize the background of images, as the background labels do not involved in the training stage. However, for the NWP task, all possible words exist in the training stage (with the format of a vocabulary list), thus it is improper to compare image classification and the NWP task. Back to the possible words situation, I insist on my opinion that for the NWP task, we should recognize it as a generative task instead of a classification task. For example, if the sentences "I like eating apples" and "I like eating bananas" are existed in the training set. Obviously, the optimal solution of ERM objective predicts "apples" and "bananas" respectively with a probability of $0.5$, under the condition of "I like eating _" (this makes training loss become zero). Since this paper explores memorization in a theoretical manner, I highly suggest the authors consider this situation (I think commonly existed). By the way, I note a highly related paper https://arxiv.org/pdf/2308.03296.pdf recently published, which studies the memorization paper as in this paper. This paper studies memorization at the distribution level. The opinion conveyed in the reference seems to be opposite of this paper. I directly copy it on page 15 of the reference as follows. "Second, while the true training tokens are used as the inputs to the network, the “labels” for the pseudo-gradient calculation are sampled from $P(yˆ|x)$. While it may appear odd for the labels not to match the inputs in an autoregressive setting, "
0-1 accuracy, another reviewer's perspective
While I agree with you that perplexity presents a more complete portrait of learning, I think that it's worth noting that the influence functions paper does not claim to focus on memorization, but on generalization. In fact, I would argue that relying on perplexity risks conflating the two concepts, as it discards the question of whether the model has memorized, rather than simply learning, a datapoint. This points to a problem in the memorization literature broadly, which repeatedly fails to define memorization as a phenomenon distinct from learning and even from generalization. This paper is at least an attempt to add nuance to the current notion of memorization, which is often based on accuracy as here, on k-elicitation, or on some partial metric based on k elicitation at best. As such, I believe that it has significant value for the *memorization* literature specifically, beyond the general learning framed by the influence functions paper.
Further clarification on 0-1 prediction loss
Thanks very much for the prompt response! **Image classification**: when we say the same ambiguity exists in image classification, we meant that there could be multiple **foreground objects** in the same image, and the same image can actually be predicted as multiple labels "correctly". There are many examples of such images exist in standard image classification training sets. While we cannot post link directly, one example we could point the reviewer to is Fig. 5 of [1]. For example, under the "Mountain Bike" category, there is an image of a cat behind the bike wheel, and another image with a truck loading lots of bikes. Under such inputs / contexts, both labels are equally likely or correct. In the terminology of NLP, all possible classes (bike, cat, truck) exist in the same vocabulary and are involved in the training stage. **Common practice in the field**: We acknowledge such ambiguity exist (in both image classification and language modeling), and we do not claim comparing with the groundtruth label is the best way to handle it. However, this is a simple and effective metric that is widely adopted in both image classification (top-1 accuracy) and language modeling (validation perplexity computed by the likelihood on the groundtruth next token). We follow this convention in our study here. Moreover, as reviewer 9fjM (Thank you!) mentioned, we focus on study of memorization in this paper, previous studies in language model memorization actually mostly rely on text matching, and our study is an important step towards distinguishing memorization and generalization. **Opposite opinions**?: Thank the reviewer for a pointer to the interesting paper studying LLM generalization with influence function. Regarding the quote provided by the reviewer (emphasis added by us): > Second, while the true training tokens are used as the inputs to the network, the “labels” for the pseudo-gradient calculation are sampled from $P(\hat{y}|x)$. While it may appear odd for the labels not to match the inputs in an autoregressive setting, this is indeed the correct **sampling procedure when the goal is to approximate** $\mathbf{G}$. We note that this paper suggested such operation specifically for *the purpose of approximating* $\mathbf{G}$, which is defined as an expected value and approximated via such a *sampling procedure*. This is not directly arguing against using the groundtruth next-token as a *performance evaluation criterion*. Therefore, we do see contradiction in the messages conveyed by the two papers. **Study on generated texts**: Finally, while not a main focus of this paper, we did study the counterfactual influence on model generated text in the last paragraph of section 5 (L279). We hope this study and the clarification above address the reviewer's concern. [1] Siddiqui, S. A., Rajkumar, N., Maharaj, T., Krueger, D., & Hooker, S. (2022). Metadata archaeology: Unearthing data subsets by leveraging training dynamics. arXiv preprint arXiv:2209.10015.
Thanks for your response
Thanks for the author's response. About the image classification ambiguity, if there exist images with both cats and bikes in it. Suppose half of these images are labeled with cat and the others are labeled with bikes. Then I think the classification model will output logits with 50% weight on the cat and 50% weight on the bikes when the input is one of these images. Clearly, this example is similar to the one in the language model. For the definition of memorization, I agree with the proposed memorization in this paper explains some of it. But my point is that such criteria built upon 0-1 loss may not cover all possible situations in the language model. A simple example is when we say memorization, we intuitively think the language model memorize all possible combination in the training set. For example, if there are only "I like eating apples" and "I like eating bananas" in the training set. If the language model totally memorizes all training sets. Then it will only predict "apples" or "bananas" within the condition "I like eating". Clearly, this is intuitively a memorization. However, when measured by the 0-1 loss, none of the training data "I like eating apples" nor "I like eating bananas" are memorized. Because the prediction probability on "apples" will vary from 0 to 0.5 (the probability can be arbitrarily small by increasing the categories of fruits) when "I like eating apples" appears in the training set. By the way, I think memorization should be measured under the sufficiently trained model. For the above example, if the model predicts "apples" with probability 1, under the condition "I like eating". Then, I think this model is not sufficiently trained as it does not approximate the ERM.
Clarification on types of Memorization being studied
Thanks for the reviewer's clarification. We now get a better understanding of where the disagreement lies --- it seems the reviewer has a different interpretation of *memorization* from ours in the paper. The term "memorization" is very intuitive yet result in many *different* formal definitions when put into the contexts. It seems the reviewer is focusing on the situation where a LM **fully memorized** the training set, or, when the LM fully capture the probability distribution of the underlying training data. We agree this is an interesting and important topic of study. However, as far as we know, there is no clear evidence even the largest LMs nowadays have achieved this goal of *fully* memorizing the training set (i.e. achieved the Bayes optimal model). Even if we want to study this, it would be quite difficult to approach. On the other hand, most study of LM memorization (including ours) focus on the case where a LMs have not *fully* memorized the training set. In this case, one of the main question is which examples are memorized and which are not. In other words, the main subject is not how *the model* behaves when it memorized the training set, but which *training examples* are memorized by a model when trained on them. Our paper goes one step further with a goal of trying to distinguish among the memorized examples, which one are counterfactually memorized, which one are memorized via "generalization". The reviewer's example of "I like eating _" falls into the category of memorization via "generalization", because it is a generalizable, distributional wise property of the (English) language itself. Our counterfactual memorization focuses on a different kind of memorized examples that could potentially be important in the contexts of privacy and copyright. For example, if the prefix is "My social security number is _", then it is not very concerning if the LM "fully memorized" all the training examples and learns the "general rule" that the completion should be XXX-XX-XXXX where "X" could be any digits. However, it could be quite concerning if my particular SSN combination is memorized by the model and generated by the model. Our paper tries to capture this type of memorization. **Summary**: the reviewer's suggestion seems very valuable in the study of model behavior under **full memorization**. However, our study is in a **different regime** (following previous studies of LM memorization) and focus on identifying which examples are (counterfactually) memorized and which are not. In this regime, the measurement by 0-1 loss is reasonable. Please let us know if this clarifies the disagreement. Thanks again for the suggestions and engaging discussions!
Reponse
Yes, I think this clarifies the disagreement. For LLM, I agree with you that the 0-1 metric is suitable to measure memorization when considering copyright or privacy. I suggest the author clarify such differences at the beginning of this paper. I think my concern is addressed, and raise my score to 6.
Thank you very much
We really appreciate the time you put into the review, suggestions and discussions! Thank you for raising the score! We will make sure to make the introduction clear on the specific types of memorization we study in the paper.
Thanks for the clarification!
Thank you for your clarifications, especially for the analysis and discussion about influence function.
Thanks for the response!
Thank you for the reply, which clarified many of my doubts, and look forward to the progress during the rebuttal being added to the final paper.
Official Comment by Reviewer PCjJ
Thank you for the clarification, and it has addressed many of my concerns. I would like to raise my score to 7. I eagerly anticipate seeing the updates from the rebuttal incorporated into the final paper.
Thanks for the response
I’ve engaged with another reviewer elsewhere here, but I think you’ve addressed my own comments.
Decision
Accept (spotlight)