ConStat: Performance-Based Contamination Detection in Large Language Models

Summary

Authors propose a new definition of contamination: "artificially inflated and non-generalizing benchmark performance" rather than "the inclusion of benchmark samples in the training data". They develop ConStat, a statistical method that reliably detects and quantifies contamination by comparing performance between a primary and reference benchmark relative to a set of reference models. They demonstrate the effectiveness of ConStat through an evaluation of diverse model architectures, benchmarks, and contamination scenarios, and find high levels of contamination in multiple popular models.

Strengths

• A new method to detect benchmark contamination from a new perspective • Great performance on syntax and sample-specific contaminated models • Good motivation, presentation and organisation

Weaknesses

• The paper uses reference datasets that are synthetically composed, authors briefly comment on this in the limitations. More ellaboration on this is need it since it is one of the basis of the experiments. Authors could potentially estimate by human analysis what is the expected error on their approach. • Benchmark-specific contamination is complex and it is not covered with enough in-depth.

Questions

• There is a short paragraph explaining results on benchmark-specific contamination, could authors elaborate more? How does it compare to other methods? • Some statements need clarification, e.g. “MathQA is a multiple-choice benchmark that requires the model to answer immediately and therefore gives no room for this reasoning ability to shine.” ->• There is a section on exposing the limitations. Maybe authors could further comment on the limitation of experiments due to computational complexity. what does it mean that requires the model to answer immediately? Could authors further support “no room for this reasoning ability to shine”

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

• There is a section on exposing the limitations. Maybe authors could further comment on the limitation of experiments due to computational complexity.

Summary

The paper proposes a performance-based approach to detect data contamination in large language models. First, a set of reference models is evaluated on the original benchmark and a proxy of it. Next, a difficulty correction function is fitted to find the relationship between the performance from the original benchmark and the proxy benchmark. Then, the performance of a target model is evaluated on the proxy benchmark, and using the correction function, the expected performance on the original benchmark is predicted. Lastly, the difference between the actual performance on the original benchmark and the expected performance is computed and checked for significance to label the target model as contaminated.

Strengths

1. The paper is well-written. 2. The experiments are comprehensive. 3. The proposed method can be used with both open-weight and closed-weight models, which is important.

Weaknesses

1. The main weakness of the proposed method is that it identifies data contamination based on changes in performance. Performance can change for several reasons, so it cannot be solely the indicator of contamination. In fact, the method assumes that any performance increase means contamination, leading to a high rate of false positives in situations where performance improvements come from other sources. For example, this method incorrectly detects contamination when performance improves due to unsupervised domain adaptation while it is not. 2. This method can only be applied to models that are similar to the reference models, especially in terms of size, architecture, and pre-training data, as it detects contamination with respect to these reference models. This limits its generality and makes it unsuitable for models that differ from the reference set. For example, if a target model just generalizes better than the reference models, the method incorrectly identifies this as contamination. Conversely, a contaminated model can be deemed uncontaminated if it cannot translate the contamination into improved performance, e.g., when a model is contaminated with a dataset but cannot follow instructions very well. Therefore, the proposed method measures the lack of generalization relative to the reference models rather than actual contamination. In short, this method does not guarantee whether a model has seen the datasets or not. 3. Building on the previous comment, this method does not actually measure contamination; it captures situations where excessive memorization has replaced generalization. In fact, the scenarios discussed in Section 2.2 **are not types of contamination**. Instead, they are examples of excessive memorization, which the method captures. 4. The results of the proposed method are relative, not absolute. In this method, detection is only meaningful when compared to the reference models. So, if the reference models are contaminated, the target model will still appear uncontaminated.

Questions

**Question:** 1. In Table 1, what is the difference between "Shi et al. [40]" and "Shi [39]"? **Comment:** 1. Lines 11-14 and 343-345: Your method does not quantify/measure contamination.

Rating

5

Confidence

4

Soundness

2

Presentation

4

Contribution

2

Limitations

1. The outcomes cannot be interpreted individually. Specifically, detection heavily depends on the reference models and how they generalize/behave. This means that better generalization or steerability in models can be mistaken for contamination. Also, if the reference models are contaminated, this contamination can spread to the target models and go undetected. 2. The proposed method does not detect contamination, as none of the scenarios studied in Section 2.2 involve contamination. Instead, the method captures the lack of generalization or situations with excessive memorization.

Summary

The authors introduce a definition for “contamination” based on its outcome rather than its cause, unlike many previous approaches. The authors propose ConStat, a novel method for quantifying the contamination of a model on some benchmark and demonstrate that it outperforms other methodologies for detecting contamination.

Strengths

- This paper was a pleasure to read. It is both written and structured in a clear way that communicates the ideas well - The authors used ConStat to show strong contamination of some models from the Open LLM Leaderboard on specific benchmarks. This research can be used to build trust in our evaluations

Weaknesses

The authors generated a synthetic version of each benchmark they used in their experiments. They described their methodology for doing so in Appendix C. I don’t believe the authors used a sufficiently rigorous method to ensure this generated dataset was sufficiently high-quality. For example, it is possible that some of their samples did not have a high 1-gram overlap but did ask similar questions. I believe a manual review of a subset of the dataset for duplication, and perhaps other traits, would be necessary to guarantee data quality. Though I view this as a minor weakness given that their methodology was successful despite potential problems with this dataset.

Questions

1. Why did the authors not select a reference benchmark for MMLU? 2. Do the authors believe this methodology will work well when the measured task between the benchmark and reference benchmark is similar (e.g. coding) but the "form" between the benchmark and reference benchmark is significantly different, for example, multiple choice question answering evaluation vs. open-ended agentic evaluation? 3. What further research are the authors excited to see in this area? Are the authors planning on discussing this within the paper?

Rating

8

Confidence

3

Soundness

3

Presentation

4

Contribution

4

Limitations

The authors sufficiently address this.

Summary

This paper targets the problem of contamination detection of LLMs by proposing ConStat, a performance-based statistical approach. - Instead of detecting the inclusion of test samples as contamination, the authors define interpretation as "abnormal" performance on benchmarks. - ConStat builds on this definition, and leverages reference models and synthetic benchmarks for statistical testing at syntax-, sample-, and benchmark-specific levels. - ConStat is empirically verified as effective. The authors also provide comprehensive analyses.

Strengths

- The definition of contamination from a performance-based perspective is novel and interesting to my knowledge. - Based on the definition, the proposed ConStat is intuitive and looks effective based on experiments. The experiments and analyses are comprehensive and solid. - Contamination detection is timely and important in the current community. - The paper is well-written and a joy to read.

Weaknesses

- As the paper discussed in Section 6, it can only estimate the contamination relative to reference models. - It would be a little costly to construct a synthetic dataset using close-sourced LLMs each time we want to detect contamination of a specific benchmark. - This paper lacks some related works that need to be discussed. I just list a few below. [1] Yang, Shuo, et al. "Rethinking benchmark and contamination for language models with rephrased samples." arXiv preprint arXiv:2311.04850 (2023). [2] Jiang, Minhao, et al. "Investigating data contamination for pre-training language models." arXiv preprint arXiv:2401.06059 (2024).

Questions

- According to lines 1118-1120, it seems that the authors only used 1000 samples for fine-tuning. How is this number determined and will the number of chosen samples affect the performance? if so, how? - The above question applies to the number of synthetic data samples.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors offered a limitation paragraph in Section 6.

Summary

The paper introduces a new performance-based definition of data contamination, shifting the focus from the cause of contamination to its effect on performance. The paper also presents ConStat, a statistical method that detects and quantifies contamination by comparing performance on primary and reference benchmarks using a set of reference models. The effectiveness of ConStat is demonstrated through extensive evaluations on diverse model architectures, benchmarks, and contamination scenarios, revealing significant contamination in several popular models.

Strengths

1. The motivation for proposing new definition of contamination is clear and easy to understand. 2. The experiments are extensive to demonstrate the effectiveness of the proposed method. It shows that ConStat can outperform the previous methods on contamination detection and is effective for contamination quantification. 3. The paper extends the analysis to the current model families to show that these models are contaminated in different degrees for current benchmarks. And the observation basically matches other papers' findings, suggesting that these methods can be potentially used for contamination examination of LLMs in the future applications.

Weaknesses

1. The assumptions in the paper are too strong for me. I don't think the difference of performance in different benchmarks means that there would be contamination. The abnormal high performance in one dataset but normal performance in other benchmarks only imply higher possibility of contamination. 2. On line 152, the authors mentioned "additionally include an inherently uncontaminated random-guessing model". I wonder if more details could be provided about this. 3. I think the descriptions and writings in the methodology section should be re-written to provide more details and motivations for how the method is developed. The current version is somewhat hard to follow and lacks explanations. 4. Missing citations to many recent works. E.g., https://arxiv.org/abs/2311.04850 (Yang et al.), https://arxiv.org/abs/2401.06059 (Jiang et al.), etc.

Questions

See the weaknesses above.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes, the authors included a limitations sections.