Summary
The paper investigates the "radioactivity" of text generated by large language models (LLMs), focusing on the detectability of synthetic text used as training data. It introduces a novel method to reliably identify whether the outputs of a watermarked LLM have been employed to fine-tune another language model. The study reveals that watermarking in LLMs is radioactive, allowing for the detection of weak watermark signal residuals in fine-tuned models. The authors link the level of radioactivity to watermark robustness, its proportion in the training set, and the fine-tuning process. Notably, the research demonstrates that training on watermarked synthetic instructions can be detected with high confidence, even when a small percentage of the training text is watermarked. The paper contributes radioactivity detection methods for different scenarios based on model access and training data, showing how to obtain reliable p-values for watermark detection and proving the practicality of detecting radioactivity in realistic settings.
Strengths
This paper presents a novel study of the "radioactivity" of watermarked text generated by Large Language Models (LLMs), with a particular focus on the detectability of such text when used as training data for fine-tuning other models.
The paper makes a significant contribution by designing radioactivity detection methods for various scenarios based on model access (open/closed) and training data exposure (supervised/unsupervised).
Among other things, the authors effectively relate the level of radioactive contamination to key factors such as watermark robustness, the proportion of watermarks in the training setting, and the fine-tuning process, and the authors innovatively propose to utilize filtering and de-duplication for detection enhancement. The results show that even a small percentage of watermarked synthetic instructions in the training data (as low as 5%) can be detected with high confidence with a p-value of less than 10^-5.
The proposed method provides more reliable statistical guarantees for detecting whether LLM outputs are used in the training dataset than existing methods such as membership inference or active IP protection. The study also provides valuable insights on the effects of fine-tuning parameters and watermarking methods on radioactivity, which helps to deepen the understanding of the underlying mechanisms.
Weaknesses
1. The author's lack of proper consideration of the written rigor of the essay is evident here in many places where semantic or formatting errors are made. For example: (1) Uniformity of punctuation: in section 2.1, Related work (MIAs) should be formatted with the context, missing periods, similar missing periods are also found in the second sentence of the response in Figure 4 ρ = 0% in Section 5.3, and in the two bolded subheadings in Appendix C.3. (2) Uniformity of formatting: in the last question/answer pair in Figure 15 of Appendix H.7, Context content is missing line breaks; in the main text section 2.2, there has been a formula with the formula number 1, and the formula in Appendix C.1 below is numbered 2, but the formula located in the middle, under appendix C, is missing the formula number; in Table 1 "closed" in "Open/closed" needs to be capitalized. (3) Content flaws: in the first scenarios of Problem Formulation Access to Bob's model in Section 3, the tense of "open-sources" is not used properly; also in Section 3, for Definition 1, "B was not trained on D", according to the author's definition of data labeling, there is an error in D here; in Figure 2, the last sentence (after and) lacks the necessary verb. (4) Unification of singular and plural:This article suggests that the authors should standardize the singular and plural, for example, "text generatedf by **" is used in the abstract and some other places. However, there are some places where "texts generatedf by **" is used, and the same phenomenon also occurs in "data/datas".
2. The watermark detection method used in this paper is only given for specific fine-tuning cases, can the method achieve the same anomaly detection effect if the model is fine-tuned in a more complex or covert way?
3. In Section 5, the article emphasizes the consideration of a realistic scenario for watermark detection, but it does not discuss in detail how this would affect the "radioactivity" if someone attempted to remove the original watermark from the training data, which is likely to be more common in reality as well.
4. The authors mention in the paper that they rely on the proofs in previous papers, but they should give some necessary information to make the paper logically coherent, for example, in Appendix C.1, the authors cite kirchenbauer et al. For the design of LLM Watermarking, which mentions that "The logit of every token in the greenlist is incremented by δ.". However, the definition of δ is not mentioned before, but in the original paper, kirchenbauer et al. proposed the algorithm "Text Generation with Soft Red List", in which the parameter δ is quoted to modify the logit and get the probability distribution of words.