Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense

Summary

The paper can be split into two parts. In the first part, the authors propose a paraphrasing-based attack that circumvents various AI-generated text detectors. The authors introduce DIPPER, an 11B parameters Transformer model obtained by fine-tuning T5-XXL. By using DIPPER to create paraphrases from texts generated by LLMs, the authors show that they can successfully evade various commonly-used detectors. In the second part, the authors propose a retrieval-based generated text detector that can successfully detect instances of paraphrased generated text.

Strengths

- The paper is well-written and easy to follow. - The authors introduce DIPPER, a paraphrasing model based on T5-XXL. The authors show that DIPPER can generate high-quality paraphrases and have tested the model in a human trial included in the Supplementary materials. - The paraphrasing attacks are effective against the models tested. - The authors have an extensive discussion about the limitations of their retrieval-based system.

Weaknesses

- The paper's biggest weakness is the generated text detection via retrieval methods since many assumptions must be made for it to work correctly. - First, one needs access to all of the text generated by the model we want to test against. This is significantly limiting since the data would probably be available only to the entity that is maintaining the LLM API. Maintaining such a big database of generated text would also be expensive from a storage perspective, and computationally expensive for the retrieval phase. - Second, using a similarity score to compare the paraphrased text with retrieved texts, one must assume that the similarity between the generated text and its paraphrased variant would be high. However, the similarity score could also be large if the two texts have similar semantics. This would be true if one used something similar to the cosine similarity between two text embeddings obtained by a neural encoder [[1]]. If one maintains a significantly large and diverse database of generated text, the chance of it containing an entry that is semantically similar to a candidate text grows, leading to FP predictions. - Third, the method would only be viable for closed, proprietary models, and would not apply to open-source models, since it is impossible to collect a database of all the text generated by an open-source LLM that is hosted by multiple people and institutions. - As far as I understand, the authors have used human-generated continuations only to adjust the detection thresholds to maintain a 1% FPR (lines 171-172). If this is the case, the evaluation methodology is somewhat flawed due to the points I've previously made. In my opinion, a better evaluation methodology would be using some parts of the human-generated data to fix a low FPR, and also have human-generated data with similar semantics to the LLM-generated one in the test set. Overall, I believe that the paper is nice and explores a potential direction that could be valuable in some use-cases, but the flaws and limitations previously mentioned make me question the practical efficiency of such a detector deployed at scale in the real world. I believe the paper could be much stronger if: a) The authors would carefully build a large and semantically diverse database of generated texts and would add human-generated texts to the test dataset. The detection method would be significantly more convincing if it did not fail in this scenario. b) The authors would design some similarity metric that would result in a large score only if the candidate text is some paraphrase of the target. This could potentially greatly reduce the number of false positives. [1]: https://arxiv.org/pdf/1301.3781.pdf

Questions

- I would like to see what happens if text generated by an LLM is kept in the database and some semantically similar entries generated by humans would be present in the test dataset. - Following the first point, what would happen if you used DIPPER to paraphrase the human-generated text? Would there be higher similarity scores for the paraphrased samples if you have some semantically similar generated text in the database? - Have the authors tried any similarity metric tailored for paraphrasing? Such a contribution would make the paper significantly stronger, in my opinion.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

- The authors have extensively discussed potential limitations in the main paper and the Supplementary materials for both their DIPPER model and the retrieval-based detection systems. While the limitations regarding the computation necessary for a large-scale retrieval system can be somewhat alleviated, as discussed in section B.2 of the Supplementary, I believe the similarity-based scoring approach to be a significant limiting factor.

Summary

The authors developed a powerful paraphrase generation model called DIPPER to test the robustness of AI text detection algorithms. DIPPER successfully evaded several detectors by paraphrasing text generated by large language models. To improve detection, they proposed a defense mechanism based on retrieving similar generations from a database, which detected 80% to 97% of paraphrased text while misclassifying only 1% of human-written sequences as AI-generated.

Strengths

1. Testing the robustness of existing detectors of AI-generated text is very interesting and important. 2. A defence method is proposed to handle the paraphrasing problem discovered in this paper. 3. The experimental results are solid and promising.

Weaknesses

1. The proposed method needs the store of all AI-generated texts. I wonder whether it is practical for popular LLMs which many users and queries, like ChatGPT. 2. More detection methods should be incorporated in experiments. 3. It would be better if more datasets and tasks are used.

Questions

Is it practical for popular LLMs which many users and queries, like ChatGPT to store all their generated texts?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations.

Summary

This paper investigates the robustness of AI-generated text detection algorithms to paraphrasing. The authors train a language model to paraphrase text in an attempt to evade detection algorithms. The proposed model leverages longer contexts than existing sentence-level paraphrasers and offers users control over content diversity and reordering. After demonstrating the vulnerability of current detection algorithms to the paraphrasing model, the authors propose a retrieval-based defense in which the outputs of a language model are stored and matched against possible AI-generated text. The authors find their retrieval-based defense achieves superior robustness against paraphrasing when compared to existing watermarking and classification defenses.

Strengths

The training scheme for DIPPER is clever and well motivated (e.g. control over content re-ordering and diversity). I think the paper benefits from including both an improved attack on AI text-detection and a simple but interesting defense. Very thorough experiments. I appreciate the discussions of the limitations of retrieval-based defenses in the appendix.

Weaknesses

The authors do not appear to evaluate any non-DIPPER paraphrasers in their experiments. The claim in line 232 that "our ablations in Appendix C show that these paraphrasers have lower quality and are less compatible with the prompt as DIPPER paraphrasers" seems misleading, as DIPPER ablations are not the same as evaluations of existing off-the-shelf paraphraser models. While the paper presents strong results for DIPPER, I think the authors should either re-word this claim to be more clear or provide direct experimental comparisons to other paraphrasers. Paraphrasing attacks might not be considered effective if they degrade the original text in certain ways (e.g. introduce grammatical errors, change the "tone" or "voice" of the text) even if they preserve some notion of semantic similarity. The results in the main paper heavily emphasize the semantic similarity metric of Wieting et al. as a measure of paraphrased text quality, although paraphrasers are evaluated under other metrics (e.g. human evaluations, perplexity) in the appendix. As things stand, it is not immediately clear in the main body of the paper whether DIPPER strongly preserves the quality of the original text beyond some notion of semantic similarity. This should probably be clarified by summarizing the results of the additional quality evaluations from Appendix C.

Questions

If the underlying language model is itself capable of diverse generations, is it possible that we might see a "saturation" problem in certain topic areas? E.g. if hundreds of students use the language model to generate one-paragraph summaries of the Gettysburg address, all of which are stored in the database, will this provide a kind of "semantic coverage" of the topic such that even human-generated summaries of the Gettysburg address would be flagged as AI-generated? I think the authors' proposed retrieval defense is promising, and their proposed improvements in Appendix B.2 are interesting, but I'm not sure whether the paper effectively argues for the feasibility of retrieval-based defenses at scale. In Appendix B.1, line 680, the authors state that "At a conservative rate of 5M queries a day, [an AI text generation] database will have almost two billion entries in a year." While the authors perform experiments to validate their retrieval-based defense at a scale of 15 million entries, it is not immediately clear how this performance would extrapolate to a database three or more orders of magnitude larger (as in their hypothetical). Along these lines, it would be helpful if the authors could point to one or more analogous text-retrieval systems capable of operating at the aforementioned hypothetical scale with the precision required for AI text detection.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

The submission "Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense" investigates detection methods for text output generated by modern large language models. The contribution of this submission consist of two parts. First, the submission describes, trains and provides a state-of-the-art language paraphrasing model. This paraphrasing model is then used to critically evaluate a number of detection schemes for their robustness against paraphrasing, finding that many modern detection approaches are not robust to paraphrasing attacks that leave the semantics of the original text unchanged, but modify its wording. In a second part, the authors use this finding as motivation to describe a detection strategy based on semantic retrieval.

Strengths

This is a great submission. It that starts out with a clear hypothesis about the detection of LLM-generated text and executes its two mechanical parts - First generating the paraphraser, and then constructing the retriever, very well. In more detail, the construction of a paraphrasing data through re-alignment of paragraph-level parallel translations is novel to me and quite interesting. The authors make the reasonable choice of finetuning existing encoder-decoder models for paraphrasing based on this corpus, a choice that they experimentally validate to work well. Then, they rightfully point out that the detection of generated can be cast as a retrieval problem, which they show indeed simplifies the problem, and allows for the leverage of existing similarity search tools. Just in case, I explicitely want to point out, that I don't consider the use of existing similarity search tools a weakness, but instead a correct conclusion based on the insight of the authors that the problem can understood in this manner. In a detailed evaluation, the authors compare LLM detection based on retrieval with other approaches, finding this to be a robust choice. Finally, I especially want to highlight the immediate practical value that the paraphrasing model would have as a tool for the community. Previous papers have often only hinted at the possibility of paraphrasing, or used general-purpose APIs to attempt to provide accurate paraphrases, and this has limited evaluations of the very practical threat model of paraphrasing in the literature. With the release of this paper and the paraphrasing model provided here, the authors would provide a practical tool to the community that will suddenly makes investigation of this threat model much more feasible.

Weaknesses

I only see a few minor weaknesses, which I will point out below. These mainly orbit around questions of "why", which this submission does not always contain. * Why would these detectors break under paraphrasing? While we do observe that the detectors based on outlier detection and classifier methods break after paraphrasing, aren't some of these detectors kind of correct in their assessment that the text is not written by the original model anymore? This does not apply to all detectors, but those that claim to only detect a specific model (or model family) are not entirely incorrect? To phrase this question differently, would paraphrasing also break detectors that classify "generic" machine text, and are not specialized to detect particular models or model families? * Why is FPR low for detection?There seems currently no way to estimate the reliability of the retrieval-based detector formally and to guarantee a certain FP rate to my understanding? Is this a principal limitation, or could the method be modified to use a threshold not calibrated from existing data? Ideally, in a way that includes the size of the corpus, allowing for estimates of performance of this approach at larger scales than can be tested empirically in this work? * Is there a corpus size at which retrieval stops being meaningful, if it is based only on semantics? I could imagine, for the sake of the argument, that many school essays that summarize existing arguments about some fact or historical event on per-paragraph level, might be semantically the same. These would also be semantically similar to some output when the model was at one point queried with this topic separately? In any case, the submission stands strong based on the empirical evidence it provides, and some of these questions might be left to future work.

Questions

A few smaller questions: * What about random substring detection? To my understanding, the submission only tests queries where the back-end of the query is removed. Would a detection of a random substring from the query perfom equally well? Bonus question: What about retrieval for several substrings from different corpus entries, i.e. what if a document is constructed based on text generated by mixing and matching multiple queries to the language model API? * How large is the Par3 dataset? Could the authors briefly comment on dataset size and finetuning compute required for the paraphraser in the main body? * I found the different FPR for DetectGPT in a part of Table 1 somewhat confusing. I see that the authors want to be polite here, but to me it would be clearer to indicate that this detection method really scores 0% at the given FPR. A 20% table could be included in the appendix, although I agree with the authors that there is no practical value to detection schemes with this FPR.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

There are a number of ethical implications to storing all user-generated text on the server for retrieval, concerning data privacy. It would be great if the authors could briefly comment on this also in the main body, as this question is currently only discussed in Appendix B.1.4. One question I have, related to this discussion of ethics, (this is a bit of an aside, the paper is great without answering this), is how retrieval would interact with regulations that include the "right-to-be-forgotten", like GDPR. Could a user ask the company to delete their generations, to prevent detection? Or should it be argued that detection is "sufficient cause" to stop deletion of user data? I also think there are soom possible solutions to this question, where reconstruction of x_i from y_i can be ruled out, possibly via bloom filters, or (minimal) differential privacy?

Summary

This paper first demonstrates the vulnerability of existing AI-generated text detectors to paraphrases, and then proposes a retrieval-based method to alleviate the issue. For the first experiment, the authors trained a paragraph-level paraphrase generation system, called DIPPER, by fine-tuning an existing text-to-text model, i.e., T5-XXL. To this end, the authors compiled a dataset comprising pairs of paragraphs with two indication parameters to control lexical diversity and content reordering. The proposed retrieval-based detector assumes the LLM API providers to save all their generated contents and to offer an interface for access the database. The authors experimented with a corpus of 15 million such examples, and shows that this approach is more robust to paraphrased texts compared to existing detectors.

Strengths

- A paragraph-level paraphraser with the recipe to build it. - Demonstration of the vulnerability of existing AI-generated text detectors to paraphrases. - A simple but effective retrieval-based method for robust AI-generated text detection.

Weaknesses

- The underlying assumption of the proposed method is not likely to be realistic. We can call LLM API providers for their ethics, but there is no means to enforce them to conform to this approach. Beside the providers, nowadays there are also increasing number of publicly available strong LLMs, and we cannot regulate personal uses of LLMs and AI-generated texts. I acknowledge the discussion in Appendix B.1, but nothing is discussed in the main paper. [Through rebuttal, the authors explained some obligations of LLM providers and promised to include its summary in the main paper.] - Scalability of the proposed approach is not sufficiently evaluated. ll.297-298 states "a popular LLM API may serve millions of queries a day" and it is likely to be true, given ChatGPT acquired more than 100 million users in the first two months and more than 13 million unique users a day. Compared to this, the experiment in this paper uses up to only 15 million AI-generated texts, it is unclear whether the proposed method is feasible (memory and time in addition to accuracy) with billions or trillions of such text. I acknowledge the discussion in Appendix B.2, but nothing is discussed in the main paper. [The response reports on 100 seconds per retrieval at the client side but this is misleading since the data store and retrieval operations must locate on the server (LLM provider) side. However, I understand that the speed would not matter given the strong facility at LLM providers and high parallelizability of the retrieval task. The experiment with 15 million instance sounds not realistic but I understand that we should accepted this if the authors are not in giant tech.]

Questions

- In l.127, $L$ is defined as "unigram token overlap" but which of $p$ and $q$ is used to determine the denominator? [harmonic mean] - Input sequences in l.132 and l.180 contains ordinary tokens: "lexical", "order", "=", and ",". Did the authors use them without escaping as "<p>" and "</p>" ? If so, wouldn't it distort the embeddings for these tokens? [not escaped, following convention.] - How many training instances were used for obtaining DIPPER? [6.3 million pairs, which should be included in the main paper.] - I'd like to suggest to tidy the layout. For instance, Tables 1, 2 and Figure 5 appear in the previous page of their first mention in the main text; Figure 3 shows a table; the mention of Figure 6 appears before Figures 4 and 5; Figure 6 is embedded in an irrelevant paragraph. [the authors promised to do so.] - l.264: $L$ has already been used in l.127.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes. The authors describe the limitations in Appendix B and ethical considerations in Appendix A.

Read the responses

Thank you for your response. I'll update my review accordingly.

Thanks

Thank you for the additional clarification and experiments, I have no further questions!

Response to authors

I thank the authors for their detailed reply to my review and for their general rebuttal. I think the authors have adequately addressed my concerns regarding the technical challenges of a retrieval-based defense, and I agree that the scale of the retrieval experiments performed is reasonable given the presumptive cost of creating a novel LLM text database. I think the paper will be significantly stronger with the authors' proposed modifications to address reviewer critiques (many of which were shared by multiple reviewers, and thus would likely be raised by readers). I have adjusted my score accordingly.

Thanks for the response

Thank the authors very much for the detailed response. It well addressed my questions mentioned in my review.