LLM Dataset Inference: Did you train on my dataset?

Summary

The paper addresses the limitations of traditional membership inference attacks in identifying if specific text sequences belong to the training data of LLMs. The authors highlight the inadequacies of MIAs and propose a novel dataset inference method. This method focuses on detecting entire datasets used in model training, rather than individual strings, combining multiple MIA metrics to accurately distinguish between training and test datasets with statistically significant results and no false positives. This approach promises to enhance the legality and ethicality of training LLMs.

Strengths

1. The paper provides a compelling argument for shifting the focus from individual string-based MIAs to dataset-based inference methods. 2. The paper is very well written

Weaknesses

1. The author demonstrates the failures of MIA methods by assessing them on the training (member) and validation (non-member) splits of the Pile dataset. However, it is unclear whether the validation set is thoroughly decontaminated from the training data. The deduplication method used in the original Pile validation data is quite loose. There is a potential issue that non-member examples might still share high n-gram overlap with the member examples, complicating MIA effectiveness. 2. The proposed approach may lack sufficient novelty: it essentially builds on existing MIA methods in two main ways: (1) by combining various MIA metrics into an ensemble and (2) by extending their application to the distribution of examples. These modifications appear to be straightforward extensions of current MIA methods. 3. The proposed method requires validation data sampled from the same distribution as the test data. However, obtaining such labeled data may not be very practical

Questions

N/A

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations

Summary

Large language models are trained on a vast amount of online available data, which has lead to copyright and privacy issues (e.g. New York Times vs. OpenAI, as pointed out by the authors). There are various methods that try to identify if a given data point x was used to train a large language model. This paper presents a very systematic analysis of these methods based on the Pythia suite of models. The conclusions of the paper is, that current methods are not well-suited to determine if specific data was included in a training set.

Strengths

The authors evaluate six metrics in the main part of the paper and a very large number of additional metrics and variations of the initial six in the appendix. Furthermore, the authors test these metrics on a large number of data sets, spanning different domains (e.g. github and wikipedia, which I would consider very different). The analysis is conducted on the Pythia suite, which means that the authors had full access to all relevant information, in particular, the training data and training methodology. As a result, they had access to the ground truth. The authors also state, that they will release their code, once the reviewing process is completed and anonymisation is no longer required.

Weaknesses

It would be interesting to see, how the evaluation generalises to commercially available models, such as Gemini and GPT. This could be done e.g. with Wikipedia-Articles.

Questions

Please explain, why you did not conduct any tests with commercially available models, such as GPT-4, Claud3, etc.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations were discussed by the authors.

Summary

In this paper, the authors investigate the commonly used membership inference evaluation for LLMs and find that previous attacks primarily detect features related to temporal changes, performing poorly under real IID scenarios. To address the challenge of individual sample membership inference attacks, the authors propose a novel threat model: LLM dataset inference. The proposed method can accurately infer whether a set of data points was used in the training process or not.

Strengths

- The paper is well-written, and I really enjoy reading it. - The "Failure of Membership Inference" section is really inspiring, and it's very important to the future LLM MIA area. The proposed dataset inference method is very simple but effective. The results look very promising.

Weaknesses

- The experiments are conducted on a single series of models, rather than on various models trained with different datasets, algorithms, or even seeds. I think this is a little picky since it's not cheap to retrain a bunch of large models, but I do think it's important for reliable evaluation.

Questions

- How does the victim choose the suspect set? What if the suspect set is a mix of member and non-member data points? This might happen when the suspect intentionally trains the model on a subset of the dataset.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors mention the limitations in the appendix, which I appreciate.

Summary

The paper addresses the challenge of identifying training data in large language models (LLMs) with rising concerns over privacy and copyright violations. As it has previously been studied, traditional membership inference attacks (MIAs), which determine if individual text sequences were used in training, are often flawed due to temporal shifts in data. The authors propose a novel dataset inference method that focuses on identifying entire datasets rather than individual data points, reflecting real-world copyright scenarios where authors' works, like books, are used without permission. Their method combines multiple existing MIA metrics to identify training datasets effectively, achieving statistically significant results with p-values less than 0.1 and minimizing false positives. The paper highlights the method's robustness through extensive testing on the Pythia models and the Pile dataset, providing a more accurate approach to dataset attribution and addressing the shortcomings of prior MIAs.

Strengths

1. The paper is well-structured, with clear problem statement, explanations of the methods, experiments, and findings. Figures and tables are effectively used to illustrate key points, making the complex subject matter more accessible. 2. Their method is extensively tested on Pythia and the Pile dataset, showing its effectiveness. The use of statistically significant p-values and the absence of false positives strengthen the results.

Weaknesses

1. The concerns about the accuracy and effectiveness of existing MIA methods have been previously studied in depth in papers such as https://arxiv.org/pdf/2402.07841. There are existing publications which have highlighted the weaknesses in empirical results that detect temporally shifted member/non-member classes. 2. The computational complexity of the proposed algorithm is not addressed in this work. Authors need to provide a detailed quantitative comparison (in terms of algorithm runtime or the number of tokens processed) for their proposed method. Is this method feasible for models with more parameters (70B or higher)? 3. Although authors have studied the Pile Dataset on Pythia extensively, their experiments are just limited to this setting. There is not much insight about how this proposed method generalizes to other models and datasets. Also, I believe that the fact this method only applies to datasets (and not single strings of text) is a limiting factor. The existence of a suspect and a validation subset is not realistic in all cases.

Questions

1. Authors mention they have used 4 A6000 GPUs for experiments. How long it takes for the algorithm to run and test the Pile dataset? What is the runtime complexity for different dataset and model sizes? 2. How does the method handle cases where the IID assumption for suspect and validation sets does not hold? Are there any strategies in place to deal with non-IID data distributions? 3. What are the potential trade-offs in terms of performance and accuracy?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper addresses the problem of Dataset Inference in Large Language Models - namely, the ability to detect whether the given dataset was used by LLM developers in pre-training. First, authors demonstrate the importance and feasibility of Dataset Inference task in comparison with Memebership Inference Attacks. Namely, they show that existing MIA methods cannot derive membership when test examples are sampled from the same distribution as training data. Second, they propose novel method of dataset inference, based on aggregation of several MIA scores as features for linear classifier. According to their experiments, the proposed apprioach does not provide any false positive results, discovering subsets of LLM training dataset with high accuracy. The algorithm requires only 1000 data points from the whole dataset to discover if it was used for training. The proposed method is tested on several PILE subsets, using the Pythia model family as the base model. Authors demonstrate that the efficiency of the proposed method increases with the growth of the model size.

Strengths

+ Authors demonstrate the failure of the SOTA MIA method (top-k% score) on different subsets of the PILE dataset. Namely, they prove experimentally that this method detects distribution shift rather than actual membership inference. Besides, they show that among several considered MIA scores, no one has consistent efficiency over different domains of PILE. This analysis is essential for future research in this direction + Authors propose novel approach for dataset infenrence, based on aggregation of several MIA scores. The efficiency of the proposed method is demonstrated by testing on wide range of domains, and with several model sizes. + The method has shown its efficiency in the problematic case of the IID data: namely, when suspect and validation data are derived from the same dataset by random sampling + The overall detection quality, demonstrated by experiments, is extremely high on all tested domains.

Weaknesses

1. The proposed method requires providing suspect and valid datasets with the same distribution. In a practical situation, it is not clear how to obtain this (see Q2 to authors). 2. Authors claim that only 1000 examples are enough for Dataset Inference. Meanwhile, train and valid set are also required from the same data distribution. On practice, the method is not tested in real few-shot regime. 3. The method is tested on the single model's family, and the efficiency is demonstrated only on the largest model of the family. 4. The method is tested on PILE dataset, with the access to the validation set from the same distribution, which was not used for raining for sure. There is no experiments with less clean data or less transparent model.

Questions

Q1. It is not clear why top-k% MIA is chosen to demonstrate general MIA failure. First, according to the original paper [32], this MIA, by design, should be stable for paraphrasing; that's why it is tested on the deliberately collected dataset (WikiMia) with different *events* distribution, not just different strings. The setup of the current paper is different (and more difficult): the authors aim to detect membership of the dataset when the whole data distribution is shared between suspect and validation sets. Figure 5(a) confirms that min-k% MIA is not suitable for this setup. At the same time, other methods provide a much more informative signal. Why don't you check, e.g., perturbation-based approach, claiming that existing MIAs cannot detect membership? Q2. The proposed method requires providing IID data for check and the "equivalent" set of data for validation. Is it possible to define the notion of "IID" more strictly, for practical use? E.g., suppose the author provides a book as "suspect" data. Is the unprinted chapter of the book "IID," if it may contain novel events and characters? On the other hand, can we consider the draft version of the book as IID, taking into account that the final version was post-processed by the editor and layout designer? What if I want to check the presence of some benchmark data in pre-training of some LLM with open weights. Which "suspect" and "valid" data should I use? Q3. Is it possible to reduce the amount of used features? E.g. what is the drop of performance if the top-k% score is excluded from the feature set of the classifier? Which features are the most informative in general?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The results of the paper was not checked in realistic settings. In general, it is not clear if the method can be applied to the analysis of the existing models with grey-box access. Authors claim that their approach is extremely stable, and dies not provide any false positives; but the experiments are not enough for such claim.

Summary

This paper tackles the dataset inference problem to detect a specifically trained dataset such as a licensed dataset. Firstly, the authors claim that previous membership inference attacks (MIAs) are not successful in discriminating between members and non-members from the same distribution (iid), which is a less realistic setting. The authors propose selectively combining multiple membership inference metrics by linear regression. Experimental results demonstrate its efficacy on the PILE dataset by aggregating 52 different MIA metrics.

Strengths

- The problem in this paper is important, regarding the high risk where copyright text is used for training current LLMs. - The paper is well written, especially summarizing previous related work and methods. It’s easy to follow. - This paper criticizes the problem settings of previous papers and proposes more realistic scenarios — e.g., iid of suspect and valid dataset. - The proposed method is simple and straightforward; an ensemble of multiple attack methods outperforms a single metric

Weaknesses

- Absent of hyperparameter gammas. Depending on what gamma value was chosen, each MIA performance would vary. - It is not explored whether the proposed method is also robust to the unseen dataset beyond the PILE dataset, as well as other models except Pythia. - The title “LLM Dataset Inference: Detect Datasets, not Strings” is too broad and enables readers to misunderstand as it proposes a “Dataset inference” task and method, whereas the paper indeed proposes a mixture of MIAs to improve robustness. - Qualitative analysis of samples is absent. Is there any interesting analysis or distinguished result between attack samples (victim data) with low and high p-values?

Questions

- How did you select the 52 MIA metrics? Sharing the standard you used will be beneficial to other researchers. - In Figure 6 (b), training set deduplication results, why does the violin plot distribute like sandglass? — i.e., the distribution mass on the higher p-values is thicker than near the p-value of 0.5. Moreover, dedup and non-dedup settings are not described.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors adequately addressed its limitations and broad impacts.

Answer

Thank you very much for the rebuttal. I am looking forward to the final version of the paper and any follow-up work on this important issue.