Rethinking LLM Memorization through the Lens of Adversarial Compression

Summary

This paper proposes a new definition of memorization, the Adversarial Compression Ratio (ACR), based on a compression argument. ACR provides an adversarial perspective on measuring memorization, offering the flexibility to assess memorization for arbitrary strings with relatively low computational requirements.

Strengths

1. The authors proposed a new metric to address the challenge of defining memorization for LLMs. This metric provides a simple and practical perspective on what memorization can mean, making it useful for functional and legal analysis of LLMs. It contributes to both the research area and real applications. 2. The authors have examined several unlearning methods using ACR and raised several problems for existing works, prompting further exploration into model memorization. 3. The paper is clearly written and well-organized. It is easy to follow the authors' ideas and understand their approaches. The authors use clear figures, i.e., Figure 1, to show their approach. The notations and experimental results are clear and easy to read. 4. The authors have provided a comprehensive literature review and show the importance of proposing this new definition and metric for LLM memorization.

Weaknesses

1. Some concepts need further clarification or justification. For instance, why do we need this adversarial view, not the natural text, for the compression argument? This assumption "if a certain phrase exists within the LLM training data (e.g., is not itself generated text) and it can be reproduced with fewer input tokens than output tokens, then the phrase must be stored somehow within the weights of the LLM" needs further justification. The authors should have provided more analysis on the threshold used in "The threshold is a configurable parameter of this definition $\tau(y)$" 2. The authors should have done some efficiency analysis on Algorithm 1.

Questions

1. Besides Algorithm 1 (using GCG), are there any other options, as discussed in Line 190? 2. As you mentioned here "This case suggests that we cannot safely rely on completion as a metric for memorization because it is too conservative." in Line 243, why and how to solve this?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Current conclusions are limited since they are just made from two specific LLMs.

Summary

The paper introduces a novel metric Adversarial Compression Ratio (ACR) to assess memorization in LLMs. The authors first contend that the conventional understanding of memorization may not be fully adequate for evaluating the memorization ability. Hence, ACR provides a quantitative measure by proposing that a model memorizes a piece of training data if it can reproduce it from a significantly shorter prompt, and presents a practical algorithm called MINIPROMPT to approximate the ACR for any given string. The authors validate the ACR through a series of experiments and case studies, demonstrating its effectiveness in various scenarios, including the detection of memorized content following attempted unlearning methods.

Strengths

1. The paper presents a new and innovative approach to defining and measuring memorization in LLMs, which is a significant contribution to the field. 2. The validation on existing unlearning methods is interesting.

Weaknesses

1. The ACR metric may inherently favor the detection of memorization in longer sequences due to the potential for a more substantial compression ratio when a shorter prompt reproduces a longer string. This bias could significantly impact the evaluation of an LLM's memorization of some concise knowledge. 2. The introduction on miniprompt is a bit brief, and since the optimizer algorithm in the experiment relies on the Greedy Coordinate Gradient Descent, the author might also consider providing a brief introduction to GCG in the main text to be more reader-friendly.

Questions

1. How does the ACR metric account for variations in the length of expressions conveying the same knowledge, such as "bird can fly" versus "it is a well-known fact that birds possess the ability to fly"? 2. How would you interpret it if two model, A has a larger portion memorized than B with a smaller average compression ratio? Which of the two metrics is more representative in measuring the memorization ability?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

Yes.

Summary

The authors propose adversarial compression ratio (ACR) as a novel metric for assessing memorization in LLMs. ACR compares the lengths of the smallest prefix string evoking a target completion from the model with the length of the completion. The shorter the prefix, the higher the compression ratio, and subsequently, the stronger the memorization. The authors further leverage an optimization-based approach using GCG, searching over the entire sequence length to find the shortest possible prefix, and demonstrate the effectiveness of such an approach in finding prompts which bypass in-context unlearning on LLaMA2-7-Chat & famous quotes.The authors perform further experiments on unlearning benchmarks such as TOFU and trying to forget harry potter, showing that ACR discovers that models are still able to reproduce significant portions of these datasets even after unlearning was applied. The authors also perform a scale-based analsyis as well as a comparison to data unlikely to be memorized to confirm that memorization increases with scale and the robustness of their proposed method, respectively.

Strengths

- The paper is well written and positioned. It provides an excellent overview of the field of unlearning and motivates the contribution well. - The experimental setup is exhaustive and well motivated. - The proposed metric is demonstrably able to detect content which is still memorized within the model despite unlearning

Weaknesses

- The optimization process, as mentioned by the authors, does not necessarily have to be completely accurate, sometimes underestimating the error ratio. It would be interesting to see how large this error can be by pehaps comparing the budget given to GCG. While it is not necessary for a solution to be optimal, an idea of how close the algorithm is, on average across a small sample size, would be relevant. - The method identifies exact memorization, but does not account for paraphrases. While it is unlikely that a paraphrase would be memorized & easily reproduced, while the exact string forgotten, this might be a consequence of unlearning. What do you think about this issue?

Questions

See above

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Summary

### Summary - This paper proposes a new metric for measuring memorization in LLMs. - Their proposed metric called Adversarial Compression Ratio (ACR) is capable of measuring memorization for arbitrary strings at a reasonably low compute. - The definition of memorization, the authors propose in the paper is based on a compression arguments which goes something like this - "a phrase present in the training data is memorized if we can make the model reproduce the phrase using a prompt (much) shorter than the phrase itself." - To operationalize this definition, the techinque requires finding the shortest adversarial input prompt that is optimized to produce the sentence under consideration as its output. - The ratio of input to output tokens is defined as ACR. - There are various ways to measure memorization - Discoverable Memorization - This was proposed by Carlini and essentially measures if a prefix elicits the suffix as the response. - This definition is very permissive since definition only tests for exact completions. - It's easy to evade by just tweaking a few tokens to avoid exact match - It also requires validation dataset to tweak generation hyperaparameters such as top-p, top-k etc. - Extractable Memorization - This definition defines extractable memorization as a string which elicits the string in the response. - Since this allows any arbitrary string as prompt, this definition is very restrictive as it allows the prompt to have the entire string in question. For e.g., "repeat the following string: X" -> X - Counterfactual Memorization - Measures the difference in model performance with a model trained with the example versus one trained without it - Since LLMs are expensive to train, this definition is quite impractical. - Membership Inference Tests which are commonly used to test if a model was trained on a particular example datapoint has the following problems - It's very restrictive. Akin to plagiarism, it is okay to read copyrighted books but copying is problematic - Have brittle evaluation - Based on the ACR ratio, ACR(M, y) = |y|/|x| where |x| = argmin |x| s.t. M(x) = y the authors propose a notion of $\tau$-compressible memorization i.e. if ACR(M,y) > $\tau(y)$ - This metric can be aggregated over a dataset to report an average compression ratio or another metric called portion memorized which measures the proportion of data with MCR > $1$ - GCG (Greedy Coordinate Gradient) is used to find the the adversarial prompt from earlier work which is a common algorithm. - The authors also show that in context unlearning can fool completion but not this adversarial notion of compression

Strengths

### Strengths - This definition is consistent with common notions of memorization - Bigger models memorize more - Random sequences are not compressible (zero compressibility) - sentences from a data source which are not part of training data have zero compressibility - Famous quotes have the highest value for ACR - The paper is well written and easy to follow. The background is thoroughly covered.

Weaknesses

## Weakness - The paper is well motivated and tackles an important problem. However, independent of the problem of false negatives, I wonder if false positives might be a bigger problem here. What if as a result of GCG algorithm, you are able to elicit a generation from a model which was never seen by the model during the training. I would suggest adding an experiment and some discussion around this.

Questions

## Questions - Line 125: "Hard to arbitrate: training data is often not released." - Even if the training data is not released, can a publisher not run MIA against the LLM to determine effectively if their training data was used during training. I don't quite agree with this claim - Line 313: Can you clarify what you mean "quells any fears that GCG is merely relaying that the gradients are more informative on some examples than others"

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This paper proposes Adversarial Compression Ratio (ACR), a metric for assessing memorisation in LLMs; ACR is defined as the ratio between the length of the generation we need to test memorisation for and the length of the shortest prompt that can elicit such a generation. A question I have is whether "length" is always suitable as a complexity measure; for example, y can be very long but also very "simple" (such as the "mmmmmmmm...mmm" messages from /r/microwavegang, which occur in some pre-training corpora). To solve such a combinatorial minimisation problem (finding the shorter prompt that elicits a given generation), the authors propose using Greefy Coordinate Gradient [GCG, Zou et al., 2023]. Overall, It is a very interesting approach and paper; however, it would be useful to compare ACR with other methods for testing for memorisation.

Strengths

1) Interesting approach for memorisation testing

Weaknesses

1) Lack of comparisons with other methods for solving the same task 2) Not sure why "number of tokens" may be a suitable measure of complexity

Questions

Is ACR more effective at detecting memorisation compared to the method proposed in e.g., https://openreview.net/forum?id=TatRHT_1cK ?

Rating

4

Confidence

3

Soundness

2

Presentation

4

Contribution

2

Limitations

Lack of comparisons with related methods for solving the same task

Thank you for your responses.

I have read the authors' responses. Most of my concerns have been addressed. I will keep my score as 7 Accept.

Author Response

Thank you for bringing this up. The omission of these licenses is an oversight on our part. We'll promptly fix this in our updated draft. For reference, the Wikipedia Data is from the Pile which is released under the MIT License and the code is Copyrighted (c) 2020 EleutherAI, and the specific license for the Wikipedia subset is CC BY-SA 4. The TOFU data is released on HuggingFace with MIT License.

Author Response

Thank for you highlighting the potential impact of our work. We agree that we were missing a discussion of the ethical implications of using ACR metrics in evidence in a legal preceding. Our work touches broadly on the ethics of copyright, but accepting the US standard (and not reevaluating the motivations for copyright law altogether) we feel that our work can help lawyers and regulators reach a precedent that upholds an artist's right to their own work. We have added a longer discussion to the draft.