Summary
The paper addresses the problem of membership inference in LLMs, where the goal is to determine whether a particular document or set of documents was used for training. The authors motivate this with concerns about the use of copyrighted material. They propose the insertion of _ghost sentences_ — sentences made up of randomly sampled words — into online documents. To test whether a model has been trained on documents containing a specific ghost sentence, they test two methods: 1. Computing the perplexity of the ghost sentence under the model probability distribution compared with the perplexity of other, randomly sampled sentences. 2. Prompting the model to guess words in the ghost sentence. A significantly lower perplexity (in case 1) or a successful guess (in case 2) indicates the presence of the ghost sentence in the training data. The authors perform various experiments covering both pretraining and fine-tuning settings, different model size, ghost sentence lengths and other parameters. They show that, given enough ghost sentences, their method can successfully detect the presence of ghost sentences in the training data of LLMs.
Strengths
- The authors motivate the last-k word test by the need for easy-to-use membership inference methods that can be deployed by average users. I agree that this is an important and often overlooked aspect that can stand in the way of a broad adoption of such methods.
- Besides the ease of use, the method also comes with the advantage of not relying on output probabilities that are often not available for commercial models.
- The authors do a good job at covering various settings in their experiments. I particularly appreciate that they perform an experiment to determine whether ghost sentences inserted during pretraining are still detectable after alignment training.
- The introduction serves as a very good overview and summary of the paper.
Weaknesses
- My main criticism of the paper is the lack of novelty. The authors cite Wei et al. [1], who propose a very similar approach of introducing random sequences of characters (instead of random sequences of words) into documents. They mention three disadvantages of [1] when compared to their own method: 1. The model trainer could filter out the random character sequences. 2. There is a risk of false positives due to the prevalence of such random sequences in training data. 3. Random character sequences do not allow for a user-friendly detection method. I think that criticism only holds to some degree: 1. The model trainer could filter out random sequences of words using a method similar to the proposed perplexity test applied to a previously trained, small language model. This would arguably be harder than simply filtering out very uncommon sequences of characters, but still seems feasible. 2. If the sequence of random characters is sufficiently long, this is not an issue, because a collision becomes very unlikely. 3. It seems possible to me that the last-k prompt could be used with random sequences of characters, where instead of asking the model to complete a sentence, once asks it to complete a sequence of characters.
- I would have liked a comparison with [1] in terms of detection success, and an experiment confirming or refuting point 3. above.
- Taken independently, neither the perplexity-based hypothesis test is new [2] nor the idea of checking whether the model can determine a correct word in a sentence better than random guessing [3,4].
- The instruction tuning task “Continue writing the given content” is very similar to the pretraining task and not a task one would expect during instruction tuning.
- It seems like when applying the method to pretraining data, an unrealistically large percentage of the data needs to be ghost sentences (cf. Table 3b), given that modern LLMs are trained on trillions of tokens. However, that is the training phase where most data is used and thus the most relevant one for detecting the use of copyrighted material. It could in principle be that the required percentage decreases to manageable levels for large models that have been shown to memorize more data than small models. To avoid the cost of (continuous) pretraining of such models, one way to test that hypothesis might be to follow an approach similar to that of [1] and try to find instances of random word sequences in the training data of large models with publicly available training sets such as BLOOM.
- Minor: The sentence starting in 185 is not grammatical. And the first half of Sec. 4 has worse writing than the rest of the paper. Besides several typos and grammar mistakes, the beginning of the paragraph in ll. 324-329 ("To figure out the average repetition $\mu$ [...]") does not seem to seem to make sense in the context of the rest of the paragraph.
- Minor: Throughout the paper, the authors use the term 'word length' several times, which I would interpret as the number of characters in a word. It seems like what is rather meant is the number of words in a ghost sentence. If that is the case, I suggest to change the term to 'sentence length'.
[1] Wei, Johnny Tian-Zheng et al. Proving membership in LLM pretraining data via data watermarks. arXiv preprint arXiv:2402.10892, 2024.
[2] Carlini, Nicholas et al. Membership inference attacks from first principles. 2022 IEEE Symposium on Security and Privacy.
[3] Lukas, Nils et al. Analyzing leakage of personally identifiable information in language models. 2023 IEEE Symposium on Security and Privacy.
[4] Chang, Kent K. et al. Speak, memory: An archaeology of books known to ChatGPT/GPT-4. arXiv preprint arXiv:2305.00118, 2023.
Questions
- 195-196: As far as I can see, we can always upper bound $1/V^*$ by $1/V_g$. Seeing ghost sentences other than the one we are currently interested in gives the model information about the set of words from which the ghost sentences are created, but no information beyond that. Thus, can't we simply plug $V_g$ into Eq. 3? Since we can choose $V_g$ arbitrarily large, we do not need to hope that we get $n_g=2$. Could you clarify?
- Why do you use beam search for decoding instead of sampling-based decoding methods that are more commonly used for general-purpose LLMs? How would using such a decoding method change your results?