Rebuttal to Reviewer AoXe
We appreciate you found our work novel, empirically strong, and valuable for future research. We have made every effort to address your concerns and incorporate your suggestions. In some cases, we provided clarifications or requested additional feedback to improve our paper further. We hope you might consider raising your score to an acceptance level if our responses adequately address the major considerations you identified as necessary for acceptance.
---
### **[Metric - TPR @ low FPR]**
Please see General Response (and Appendix B).
---
### **[Computational Complexity]**
Please see General Response (and Appendix C). We agree that considering a compute-constrained setting is a valuable suggestion. We view improving the efficiency of EM-MIA and other MIAs as an exciting direction for future work.
---
### **[Missing Experiments - Varying the number of shots $n$]**
To clarify, while each component in EM-MIA is a design choice, we mainly used the update rule for prefix scores as Line 3 in Algorithm 1 (calculating pairwise ReCaLL scores on all data $x \in D_{test}$ using each example $p \in D_{test}$ as a standalone prefix once and comparing them with the estimate of membership scores at each iteration) and the update rule for membership scores as Line 4 in Algorithm 1 (directly using a negative prefix score as a membership score), as described in Section 6.4. In this case, we do not even need to tune or exhaustively search $n$. That is the reason why we do not include a plot for the effect of $n$.
On the other hand, $n$ is an important hyperparameter for ReCaLL. You can refer to the ReCaLL paper to see how its performance changes with respect to $n$. In our experience, ReCaLL was sensitive to $n$ and the selected data points to construct a prefix. Our Figure 1 illustrates how the effectiveness of each example as a prefix varies significantly.
If you mentioned the number of shots because you are thinking about choosing a prefix consisting of data points with top prefix scores that we suggested as an alternative way to update membership scores, we hope this explanation resolves your concern. We will further clarify this in the revision.
---
### **[Missing Experiments - Varying the number of iterations]**
As explained above, each iteration updates scores without requiring additional LLM inferences, making the process fast. The number of iterations does not affect the overall computational complexity. We set the maximum number to 10, and the process usually converged after 4-5 iterations.
---
### **[Missing Experiments - Ablation on initialization]**
Please see General Response (and Appendix A).
---
### **[Missing Experiments - Reusing test examples]**
Could you please elaborate more on the ablation experiments you described? If you want a comparison after removing data points used from ReCaLL, this can be done easily by omitting the “keep_used” option in “eval.py.”
---
### **[Clarity - Pre-training data detection]**
Our method can indeed be applied to different training stages, but we focused on pre-training data detection in this work. We will clarify this in the revision.
---
### **[Clarity - Intuition behind ReCaLL]**
Thank you for this suggestion. We added a brief explanation to provide an intuition of why ReCaLL works in Section 2.4 for readers unfamiliar with ReCaLL.
---
### **[Clarity - Prefix score]**
We agree that the definition of a prefix score in the abstract might feel abstract. Providing a formal definition would require a detailed explanation of ReCaLL, which could overwhelm readers. If you have suggestions for simplifying this explanation without losing clarity, we would greatly appreciate them.
---
### **[Clarity - Table]**
We recognize the tables are dense with numbers. Following Min-K%++ and ReCaLL, we reported results for all combinations of methods, models, and lengths. Our tables include additional results due to the larger number of ReCaLL-based baselines we evaluated. We encourage readers to first focus on average scores and then dive into specific comparisons of interest.
---
### **[Clarity - OLMoMIA]**
We added equations for our OLMoMIA settings in Appendix E. Members and non-members are clustered separately. We stated this in Line 326 of the original manuscript.
---
### **[Missing References]**
Thank you for pointing out missing references. We have added them in the revised version.
---
### **[Why use the ratio of the log-likelihoods instead of the difference?]**
We experimented with both the ratio and the difference of log-likelihoods. Empirically, the ratio performed slightly better overall. Additionally, we followed the ReCaLL approach for consistency in comparison.
---
### **[EM Interpretation]**
We interpret prefix scores as latent variables that need to be predicted.
---
### **[Plans for releasing code]**
We uploaded our code as supplementary material.