Summary
This paper questions the effectiveness of MIA approaches which are currently used to evaluate the leakage of large langage model (LLM) pre-training data. The authors evaluate several MIA methods from previous works. They find that the performance of MIAs is quite poor on several LLMs trained on the Pile when evaluated on subsets of the Pile from different domains. This finding seems to contradict previous works which reported much better MIA performance. The authors explain this difference by arguing that the previous works used member and non-member data from different time ranges, resulting in distribution shift of non-members with respect to members. Therefore, previous MIAs likely performed distribution inference rather than membership inference, with distribution inference being an easier task. The authors also question the relevance of the task’s definition for natural lanauage where it is possible to construct different but similar samples (which are technically non-members), but should reasonably be viewed as members. MIAs are shown not to be robust to such samples, which are wrongly classified as non-members in the high-confidence regime.
I think that this work is timely and tackles an important question, which is relevant for privacy auditing and for copyright infringment. The paper does not propose a new method, so the technical novelty is very limited. However, it performs a large-scale evaluation of several state-of-the-art MIA approaches, whose results challenge the strong performances of MIAs reported in some of the previous works (Shi et al, 2023 and Meeus et al, 2023). This is an intriguing finding and the authors propose a plausible explanation for these results. The authors further challenge the definition of MIAs in the context of LLMs, showing that MIAs are not robust to small changes in the sample.
However, the paper in its current form does not seem ready for publication, as it is imprecise on several important points neccessary to substantiate its claims and the benchmarking of MIAs seems incomplete and to suffer from a methodological flaw. Most of these points seem fixable and I hope that the authors will address them in their rebuttal.
Reasons to accept
- The authors perform an extensive evaluation of several MIA approaches applied to LLMs, on several datasets and models. I think that this benchmarking of methods is valuable for the community. The authors also make their code available.
- The evaluation reveals the intringuing finding that current MIA perform quite poorly. As this finding directly challenges results from some of the previous works, and a plausible explanation is offerred, I believe that publishing the paper can be beneficial to the community.
- An important finding is that the MIA decision boundary is not robust to small changes in the text. The authors show that a slightly modified version of a member text that would be considered as a member by a human (but is technically a non-member as per the strict MIA definition) would be classified as a non-member. This challenges the current definition of MIAs and could motivate interesting developments in the field.
Reasons to reject
- Potential methodological flaw: the Reference-based and min-k prob attacks seem to be tuned on the test data. I re-read Appendices A.3.1. and A.4 several times looking for a mention of validation data and I found only one, for the DATABLATIONS model, while the Reference-based tuning is done on PYTHIA-DEDUP models (for which there is no mention of validation data). Tuning the methods on the same data on which final performances are reported is a methodological flaw because it biases the results. It also gives an unfair advantage to the methods being tuned compared the ones which are not tuned. This is especially problematic here since the extensively tuned Reference-based attack is found to perform « best » in the end. This can be fixed by using a validation dataset for tuning the methods which is independent from the test set.
- Unclear exactly how the author’s analysis of temporal shift relates to the previous works of Shi et al, 2023 and Meeus et al, 2023. The paper should clarify whether the same experiment settings (LLMs, data domains, temporal ranges) as the two papers are reproduced, or if the author only speculate that these two papers suffer from temporal shift. Making this very precise is needed to support the abstract claim that « this apparent success can be attributed to a distribution shift », else the claim should be more nuanced. For instance, going through these two papers, I understand that in the arXiv domain, Shi et al uses members and non-members from before 2016 and after 2023, respectively, Meeus et al uses members before February 2023 and non-members after February 2023, and the current paper uses members from July 2020 and non-members from later months. Since the temporal shift is slow (Figure 6), it likely affects the former paper more than the latter. Furthermore, Meeus et al. also uses a second dataset where there is no temporal shift and uses longer documents for which MIA risk is likely to be higher. These differences are not discussed in the paper.
- Limited evaluation: the paper focuses on 200-word texts. When it comes to copyright infringment, longer documents such as news articles and books pose a bigger concern. As mentioned by the authors and shown by Shi et al, 2023, MIA risk increases with text length. Therefore, it is likely that MIAs might still work for, e.g., books. If this is true, the paper’s core claim would not hold. There is no discussion of this in the paper.
- A point is made that the MIA definition is not aligned with how we reason about privacy, but no better definition is proposed. The link between this part and the rest of the paper is not very clear. The usefulness of experimenting with large edit distances of 10 and 25 is unclear to me: if a large fraction of the text is replaced with random tokens, would the text still be considered to be a member by a human? Even for an edit distance of 1, replacing a token with a random token does not guarantee that the text still makes sense (which would be necessary in any notion of neighborhood). The empirical part of this section seems insufficiently developed.
Questions to authors
- How do the authors ensure that their MIA evaluation setting (non-members from the Pile test data) does not suffer from distribution shift ? For instance, is there any reference stating that the Pile data was randomly shuffled before being split into train and test ?
- Other works have already challenged the reliability of MIA predictions, see [A] and [B]. For instance, [A] shows that neighboring non-member samples would be classified as members which is related to the point made in this paper. Therefore the insight that MIAs are brittle is not new (although demonstrating it empirically in the LLM setting is). The paper does not position itself with respect to these works. Can the authors elaborate on their differences with [A] and [B] and add a discussion of previous works challenging MIA reliability ?
[A] Rezaei, S., & Liu, X. (2022). On the Discredibility of Membership Inference Attacks. arXiv preprint arXiv:2212.02701.
[B] Hintersdorf, D., Struppek, L., & Kersting, K. (2021). To trust or not to trust prediction scores for membership inference attacks. arXiv preprint arXiv:2111.09076.
- How does the state-of-the-art MIA method of Meeus et al, 2023 perform compared to the other approaches?
- Figure 1: How are the confidence intervals computed, e.g.., over how many samples?
- Sec 3.2.2 – Resampling of non-members to have low overlap. This step seems over-engineered and, as the authors notice, results in distribution shift. A simpler approach would be to rank the non-members by the MIA scores and check if MIA confidence is negatively correlated with n-gram overlap. Did the authors consider this approach ?
- Benchmarking of MIA approaches : Are the results statistically significant ?
- ArXiv temporal shift experiment: there is a difference in the pipeline used to construct members and non-members (S3 vs arXiv API), which could introduce a distribution shift beyond the temporal shift. To substantiate the claim that « both arXiv bulk access and API should yield the preprints in the same manner regardless », can the author apply the API to retrieve some of the data before July 2020 and check that after pre-processing the documents match ?
Other comments:
- Please consider adding the definition of a sample (200-length documents) in the main paper.
- Appendix A.4: I disagree with the statement that the reference-based attack is largely similar to the offline LiRA attack of Carlini et al., 2022. Two key features of LiRA, which don't feature in the reference-based attack, are its use of the logit loss (log(p/(1-p)) and fitting a Gaussian distribution to its distribution, which allows extrapolating to the high-confidence regime. The reference-based attack would be more similar if the log(1-p) terms were substracted from the loss.
- Reference-based attack: The authors acknowledge the limitation of LLM reference models: although they are not supposed to be trained on the target sample (for the one-sided hypothesis test to be valid), they likely are. I agree with the authors that this seems hard to fix. However, because the attack’s assumptions are invalid, it’s hard to interpret why and when it works. Some analysis is provided in Appendix B.2; however it would be better to explain the first time the attack is presented (in the main paper) when the attack is likely to work in spite of the invalid assumptions and why. Perhaps this intuition can also guide the choice of the reference model.
Minor :
- First paragraph of Sec. 3.2.1 – the argument needs to be completed by explaining that MIA success if linked to overfitted (Yeom et al, 2018).
- Figure 2 : the nuances of blue on the left are hard to distinguish in print, please consider increasing the contrast and the font size.
- n-gram overlap formula: Please consider adding 1-2 sentences explaining what the formula is meant to capture.
- Figure 3 – Github: is my understanding correct that on Github 40 % of the samples have 100 % n-gram overlap with the training dataset ? Please consider discussing this in the main paper and explaining that this does not mean that these non-members are members.
- Please consider fully defining the loss in the main paper using the negative log-likelihood of tokens.