Proving Test Set Contamination in Black Box Language Models

Thank you for your thorough analysis and constructive feedback on our paper. We appreciate the opportunity to clarify the points raised and to provide additional insights into our research. N-gram overlap is a commonly utilized measure of contamination in the literature; however, it should be noted that it acts more as a measurement tool rather than as a definition of contamination. N-gram overlap may fail to distinguish between coincidental overlap and genuine contamination in some situations, and has been observed to potentially lead to false positives under certain conditions. For example, the Stanford Question-Answering Dataset (SQuAD v2) uses background information derived from Wikipedia, and in the GPT-3 paper, Brown et. al. (2020) find high N-gram overlap for this reason, even if labels are not present in the data and no true contamination exists. Please see pg. 43 of that paper for examples of false positives in their contamination study. We believe that the definition introduced in our paper, of contamination as a statistical dependence between the model and the dataset, is precise and formal, and better captures the notion of contamination as a transfer of information between the test set and the model—not simply a correlation that appears because both the test set and pre-training data share information. While verbatim contamination of ordered data does not encompass all forms of contamination, we found that the presence of ordered test sets in pre-training data is surprisingly common. A search of The Pile, a large open-source language modeling dataset, yielded numerous instances of real-world datasets embedded with examples appearing in-order; see our top-level comment for an example. In pre-training, shuffling of the data occurs at the document level, and is not typically applied to the data within a document. Files collected from the internet would typically be treated as singular documents in dataset construction pipelines. More importantly, the use of ordering allows us to give provable guarantees of contamination, which is more difficult to achieve for other, less direct forms of contamination. Our work is the only existing contamination detection method for language models to give guarantees of this kind. Regarding concerns about sensitivity of the test to the shard count, figure 3 (left) shows that a wide range of shard counts (between 10 and 150) attain p-values below 1e-4. Once the p-value is low enough that statistical significance is attained, there is no added benefit to lowering the p-value. Therefore, the plot suggests that the test is robust to shard count, so long as the shard count is not too low (so that the t-test can still be used reasonably), and not too high (so that there are enough examples per shard to get sufficient signal from log-prob differences.) Similarly, figure 3 (right) shows that increasing the permutation count monotonically decreases the p-value, and that the p-value stabilizes beyond about 25 permutations per shard. This suggests that the test is not sensitive to the permutation count, provided that the permutation count is not too low. Our empirical results use a permutation count of at least 50. We welcome further discussion on this point to ensure we fully understand and address your concerns. Detecting contamination at the instruction fine-tuning stage would be interesting follow-up work, but is complicated by the fact that examples are commonly shuffled in this setting, and so we cannot test against a known example order. In this setting, heuristic methods may prove to be more effective. We hope this response has addressed your concerns effectively. We are grateful for the chance to discuss our work's potential, and wish to thank you again for your valuable input.

Thank you for your thorough review and valuable feedback on our work. We'd like to address the concern regarding the computational complexity of our test. It's important to note that the test is a one-time process for any given model and dataset; once the p-values are computed, there is no need for recalculation. Our findings indicate that a number of permutations beyond 30-50 per shard offers diminishing returns, as shown in Figure 3 (right). Furthermore, the test's design allows for easy parallelization. Each shard permutation can be evaluated independently, enabling the use of inexpensive commodity hardware to run the test significantly faster. Regarding the assumption of data exchangeability, this is a strictly weaker condition than the commonly held assumption of independent and identically distributed (I.I.D.) data in machine learning. Most datasets satisfy this assumption to some extent. We acknowledge the validity of our test hinges on data exchangeability. However, depending on the source of non-exchangeability, it is often the case that a dataset can be altered slightly so that our test is still valid. For example, a common source of non-exchangeability is the presence of ascending IDs (e.g. as in SQuAD and HumanEval). We can adjust the data—by either removing these IDs or permuting the examples while keeping IDs constant—to retain the test's applicability. This is discussed in more detail in the revised paper. Finally, we appreciate your suggestion to include baseline comparisons. We provide a comparison against a contamination detection method called Min-K% Prob, a state of the art heuristic method for contamination detection in language models proposed contemporaneous to our work by Shi et. al. (2023). We find that our method matches or exceeds the performance of this state of the art heuristic method. Please see the table in the top-level comment for numbers.

Thank you for your recognition of our work and for your insightful comments. This work differs from prior work in two key ways: in the definition and setting considered and in the resultant provable guarantees. We consider contamination detection as the problem of detecting a statistical dependence between the test data and model parameters, and show that we can provide provable guarantees in the case of verbatim contamination, where the full test set (with examples and labels) is embedded in the pretraining data. Prior work is primarily heuristic in nature; To our knowledge, our work is the first of its kind to provide provable guarantees of contamination for language models. For comparison against a baseline, we provide a comparison against a contamination detection method called Min-K% Prob, a state of the art heuristic method for contamination detection in language models proposed contemporaneous to our work by Shi et. al. (2023). We find that our method matches or exceeds the performance of this state of the art heuristic method. Please see the table in the top-level comment for numbers. Thank you for your question regarding the impact of model size on the performance of the test. Existing work on memorization in language models suggests that larger models memorize their training data more strongly. For example, Carlini et. al. (2023) show that both model size and repetitions in the training data increase the extractability of training data sequences. Our empirical results show that the power of our test increases dramatically in the number of repetitions (duplication rate), and we posit that our test enjoys a similar increase in power for larger model sizes, since larger models likely memorize example order more strongly. We present preliminary results on the impact of model size on the power of our test. We evaluated three models of increasing parameter counts trained on the same data mixtures as in section 4.1, on the test sets present in the pretraining data at a duplication rate of 1. These results suggest that the test performs better with larger models. | Parameter Count | Average Logarithm of P-Value | |-----------------|---------| | 355M | -1.427 | | 774M | -1.825 | | 1500M | -12.783 |

Thank you for your thoughtful review and for recognizing the importance of our work. Regarding the definition of test set contamination, we specifically address verbatim contamination, where both inputs and labels from the test set appear in the training data in order. While other forms of contamination can be studied, such as indirect contamination, we study verbatim contamination as this form of contamination is more amenable to provable detection with statistical guarantees. Your second question is regarding the omission of labels In Figure 1. We omitted labels in the figure for readability, and the actual test does incorporate the labels.