ReMoDetect: Reward Models Recognize Aligned LLM's Generations

Summary

The authors demonstrate that reward models inherently possess the capability to distinguish between human-written and machine-generated text. They propose a method for continuous pairwise fine-tuning of existing RMs, which achieves excellent results on several LLM-generated text (LGT) detection datasets and exhibits robustness against adversarial attacks.

Strengths

1. The premise of the study is interesting. Given that many LLMs are optimized based on RMs, it stands to reason that RMs encode certain features of LLM-generated text, making them a promising foundation for further training in LGT detection. 2. The experimental results presented by the authors are impressive.

Weaknesses

1. While the performance improvements are noteworthy, further clarification is needed regarding the mechanisms behind these improvements. For instance, how does the pairwise loss function proposed in this paper differ from loss functions used in other LGT detection methods? 2. The authors' experiments validate the accuracy of only one RM after continuous training using their proposed method, which is the same as in the preliminary experiments. However, the scope claimed in the paper seems to encompass all RMs. Either additional experiments should be conducted or the claim should be narrowed. 3. Although utilizing RMs as a starting point is an interesting approach, the availability of RMs transforms what would be a black-box detection into a semi-white-box detection. This difference in setup may naturally lead to performance improvements, which should be addressed. 4. If a model only undergoes the supervised fine-tuning phase of alignment, can the proposed LGT model still be effective?

Questions

1. In Table 2, can the authors report the parameter counts for the baseline methods in the main experiments? 2. In Table 2, have smaller models been tested, rather than focusing solely on large models? 3. Regarding Table 4 and 5, please provide convincing explanations for the enhanced performance against distribution shifts and attacks. 4. In Figure 5, the score distribution for human-written text shows increased variance after training, while the opposite is true for LLM-generated text. Can the authors provide an explanation for this phenomenon from the perspective of the loss function?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

We encourage the authors to provide a separate section for limitations and social impacts.

Summary

This paper proposes a method named ReMoDetect to use a reward model for model-generated text detection. Firstly, The authors find that the existing reward model can easily distinguish human-written text from language-model-generated responses. Then, the authors propose two techniques, 1) continual preference training and 2) mixed human and LLM responses, to further train the reward model for LLM-generated text detection. Experimental results demonstrate the effectiveness of the proposed reward model based LLM-generated text detection.

Strengths

- The motivation is clear, and the proposed method to use a reward model for detection is original. - Based on the experimental results, the proposed method is effective across multiple LLMs and different domains. - The paper is well-organized and easy to follow.

Weaknesses

- To evaluate a response using a reward model, it requires both the given context or prompt x and the response y. However, the prompt x is not always available in the LLM-generated text (LGT) detection problem. It would be useful to illustrate how the corresponding prompts are determined when evaluating the proposed models. And it would be great to additionally evaluate the proposed method on datasets without prompts. - It is unclear why the reward model can recognize the LLM-generated texts from human-written responses.

Questions

- Is there some correlation between the reward model accuracy and the corresponding LGT detection accuracy?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper is about a novel and effective approach for LLM-generated text (LGT) detection by making use of the reward model score. The authors observe that LGT often has higher reward model score compared to human-written texts. They then further increase the separating by fine-tuning the reward model to score LGT higher than human-written texts, and use additional LLM-rephrased human-written texts as the median preference text to assist with the learning.

Strengths

- The evaluation results are very strong on selected benchmarks, outperforming other LGT detection methods. - The analysis about robustness on unseen distributions, rephrasing attacks and input response length is solid.

Weaknesses

- The work lack qualitative analysis on examples. The cases and patterns for errors, and for improvements, are unclear.

Questions

- Why GPTZero is not evaluated on MGTBench? - While the Table 2 and 3 covers 6 models, why Table 4 only covers 4 models, Figure 4 only covers a combination of 2 models, and Figure 5 only covers 1 model?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

- The paper has inconsistency in selecting the results to report in both the main paper and the appendix. See questions for details. - The selected benchmarks are mostly scientific writing and news writing, while other commonly used benchmarks in Question Answering, Web Text and Story Generation (as defined in [1]) are not covered. [1] Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023). A survey on llm-gernerated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724.

Summary

The paper finds that reward models used in RLHF can detect texts generated by LLMs. Based on this, the paper presents ReMoDetect, a novel method that further trains the reward model using continual preference fine-tuning and a challenging text corpus rephrased by LLMs. ReMoDetect achieves new SOTA on various LGT benchmarks.

Strengths

1. The paper presents an interesting finding that RLHF reward models can make LLMs generate outputs that align too closely with human preferences, even more so than human-written texts. This motivates the authors to further fine-tune the reward model, which is a well-motivated approach. 2. The authors conduct extensive experiments showing that the proposed method achieves SOTA on various benchmarks. The ablation study demonstrates the effectiveness of each component in ReMoDetect. 3. The paper is well-written and easy to follow, with a clear storyline from motivation to proposed methodologies. The two training strategies are simple and reasonable. 4. Detecting LGT is an important research problem.

Weaknesses

1. The proposed method relies on the quality of the reward models, such as the training dataset and model parameters. A pre-trained reward model may be biased towards a specific dataset and may not generalize well to all LLMs. Poor initialization of the reward model may harm performance. 2. Reward models are also LLMs. Using such models for LGT detection involves long inference times.

Questions

1. What is the performance difference when using different reward models for initialization? For example, if we use the reward model of LLM A, and then classify the results of both LLM A and LLM B, does the LGT classifier perform better at detecting LGT of LLM A? 2. If the reward model of a specific LLM is not available and the reward model has learned some specific or undesirable preferences, how does the proposed method perform in this scenario?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The reward models are not available for some closed-source LLMs.