LLM Detectors Still Fall Short of Real World: Case of LLM-Generated Short News-Like Posts

Author's response

We thank the reviewer for their feedback. We believe that some of the weaknesses highlighted by the reviewer stem from our lack of clarity regarding our contribution and scope, which we have clarified in the revised manuscript. As we mention in our response to other reviewers, the heart of our contribution is showing that the methodology of the prior LLM benchmarks is flawed in principle, leading to a trivial evasion attack still being effective. The LLM detectors evaluated as SotA in third-party benchmarks (notably RAID [1]), significantly underperforming and not being ready for real-world use. Specifically, 1. We show that large-scale static benchmarks, such as RAID [1], are not suited for evaluating LLM detectors in a way relevant to the real-world setting. This issue is due to: 1.1 LLM-generated text detection is inherently happening in adversarial settings, with the attacker seeking to evade detection. As such, reference static benchmarks provide a baseline not only for the LLM detector developers but also for attackers, who will learn to evade them. This adversarial setting radically differs from the general LLM evaluation practices, where benchmarks are useful to validate the model's capabilities. While we are the first ones to state this in the context of LLM detection, this difference in evaluations of ML solutions in adversarial settings has been previously highlighted, notably in cybersecurity [2]. In particular, the failure of recent benchmarks to test for temperature increase attacks led to all detectors being vulnerable to it, constituting an example of such an attack 1.2 LLM detectors' performance varies wildly depending on the task, even for closely related tasks, making large-scale benchmarks non-useful even for the best detector selection task, even of highly related topics, such as ours and "news articles" in the RAID benchmark [1], including for identical attacks, as highlighted in 4.3 "Prior Benchmarks" section. 2. We show that all the previously published benchmarks on LLM detectors are flawed in principle because they do not consider the tradeoff between the generalization to previously unseen attacks (TPR generalization) and previously unseen types of human-written texts (FPR generalization). While this tradeoff is intuitive, it has neither been stated in the LLM detectors literature nor explicitly benchmarked as a tradeoff in the past, leading to real-world issues with LLM detectors FPR generalization, notably false positives on text written by non-native writers [3]. 3. We propose a method to test LLM detectors that is useful for evaluating their real-world performance, that can be dynamically expanded, and provide code to achieve it. Specifically, we do not provide a dataset to evaluate LLM detectors on, but a method to derive an evaluation dataset from samples of texts in an application scope, given a threat model, which is radically different from prior work in the field, that instead attempted to build universal, scope and threat model-agnostic datasets. We hope that this addresses the weakness W1 stated by the reviewer. We agree with the reviewer's point in W2. However, prior work incorporating perplexity analysis and human evaluation has shown a minimal text quality degradation, even for the strongest of the attacks considered - paraphrasing [4]. As such, we did not perform the perplexity analysis for attacks we considered, but it is definitely a point to consider. We thank the reviewer and have added a point about it to the methodology and discussion section. Similarly, we agree with the reviewer that a study of human-written texts is often misclassified as LLM-generated would be highly interesting. We have anecdotal reports of neurodivergent individuals being most susceptible to such misclassification (personal communication with Prof. Dr. Rua Mae Wiliams, PhD), but to our knowledge, there is no such systemic study, and we are not aware of datasets that could be used for it. Again, we thank the reviewer for their feedback and apologize for the lack of clarity in our original manuscript. We adjusted the manuscript to respond to the reviewers' remarks and performed an additional pass to correct grammatical errors and citation usage inconsistencies in the main text. [1] Dugan, Liam et al. "RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors." Annual Meeting of the Association for Computational Linguistics (2024) [2] Arp, Daniel et al. "Dos and don'ts of machine learning in computer security." USENIX Security 31 (2022) [3] Liang, Weixin et al. "GPT detectors are biased against non-native English writers." Patterns 4 (2023) [4] Sadasivan, Vinu Sankar et al. "Can AI-Generated Text be Reliably Detected?" ArXiv abs/2303.11156 (2023)

Agree broadly with your points, but execution is lacking

Thank you for the rebuttal. In general, I'm in broad agreement with the ideas advanced in the work, and it is consistent with my own findings in this space. However, as acknowledged by you, the ideas and contributions are not particularly novel, and have been brought up before. In such a scenario, to merit acceptance, the paper must be well executed. As other reviewers have mentioned, it would be good to conduct these experiments on a large scale, either from the angle of poor generalization across human texts (you would need to collect some data for this) or just increasing the scope of detectors/LLMs considered in the paper.

Author's response

We thank the reviewer for their feedback. We believe that some of the weaknesses highlighted by the reviewer stem from our lack of clarity regarding our contribution and scope, which we have clarified in the revised manuscript. We understand the W1 remark by the reviewer that the experiment coverage could be improved by including additional human reference datasets in addition to CNN Dailymail. However, we believe that the failure of standard methodologies for LLM detector testing we demonstrate and the demonstration of the failure of detectors performing best according to third-party benchmarks is sufficient to make our point both as to the readiness of LLM detectors in the real-world use case in information operations detection, as well as general methodological issues with current state with LLM detectors testing. We agree with the reviewer's statement in W2 that the ELECTRA-RoundRobin would perform better if fine-tuned on more diverse human samples. However, the point of this experiment and Table 2 is to demonstrate a methodological flaw in prior approaches. Specifically, we show that there is a tradeoff between the generalization to previously unseen evasion attacks (TPR generalization) and previously unseen types of human-written texts (FPR generalization), which has so far never been evaluated as such in LLM detector benchmarking papers, despite leading to real-world harm [1]. Once again, the goal of ELECTRA-RoundRobin is to show that fine-tuning a classifier for detection - which is still commonly reported in LLM detector benchmarks as achieving SotA - can lead to great performance in adversarial setting at the cost of reference human dataset overfitting, which in turn is not evaluated. However, if we were attempting to develop a novel SotA detector, including diverse samples of human writing would have indeed been essential. We thank the reviewer for highlighting the issues with the quality of writing (W3) and have performed an additional pass for grammar correctness and consistent usage of /citet /citep. We believe our response to W2 also responds to question Q1. While it is possible to build better LLM detectors by increasing the diversity of human text samples, our goal is to show that the performance of LLM detectors needs to be evaluated on the human text samples in the correct subject domain. We thank the reviewer for the question Q2. We compared our results with the RAID benchmark [2] by matching their "news" category, generators, detectors, and attacks that were identical but obtained significantly different detection rates, as highlighted in the 4.3 "Prior Benchmarks" section. While we focused on generated news-like content as most socially damaging in the immediate, our method also applies to other domains, assuming a threat model is defined. While the generated reviews are likely to be sufficiently similar to share the threat model and length limitations, books are significantly longer. They will likely have more extensive prompts and undergo segmental rewrites or regenerations. In that latter case, generation and evasion procedures will need to be re-implemented in a way consistent with the considered threat model before conclusions can be drawn. While the conclusion regarding their detectability might differ from the short news-like content generation, the methodological contribution of our work will still apply - namely, the need for a scope, threat model, and the evaluation of the tradeoff in generalization on unseen evasion attacks out-of-distribution human-written text samples. Again, we thank the reviewer for their feedback and apologize for the lack of clarity in our original manuscript. We adjusted the manuscript to respond to the reviewers' remarks and performed an additional pass to correct grammatical errors and citation usage inconsistencies in the main text. [1] Liang, Weixin et al. "GPT detectors are biased against non-native English writers." Patterns 4 (2023) [2] Dugan, Liam et al. "RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors." Annual Meeting of the Association for Computational Linguistics (2024).

Author's response

We thank the reviewer for their feedback. We have adjusted the introduction to better respond to the W1 identified by the reviewer and comments from the other reviewers. Similarly, we have corrected the grammatical mistakes we could find. We respectfully disagree with the evaluation of the reviewer in the W2 that our work is limited to a single observation and lacks impact on the field, given that we believe that the methodological contribution we provide in this paper is major. Specifically, we show that the LLM detector evaluation that has been performed until now is flawed in its real-world relevance, and we propose a new testing methodology for which we validate and provide code. Specifically, we show that: - The LLM-generated text detection is inherently happening in an adversarial setting, with the attacker seeking to evade detection. As such, reference static benchmarks provide a baseline not only for the LLM detectors but also for attackers, who will learn to evade them. This setting radically differs from the general LLM evaluation, where benchmarks are useful to validate the model's capabilities. While we are the first ones to state this in the context of LLM detection, this difference in evaluations of ML solutions in adversarial settings has been previously highlighted, notably in cybersecurity [2]. We show that this vulnerability is not merely theoretical by demonstrating the effectiveness of a simple temperature-based evasion attack that has been omitted from recent LLM detector benchmarks. - The LLM detector performance varies wildly depending on the task, even for closely related tasks, making large-scale benchmarks non-useful even for the best detector selection task, even of highly related topics, such as ours and "news articles" in the RAID benchmark [1]; highlighted in 4.3 "Prior Benchmarks" section. Similarly, the tolerance to FNR and FPR varies wildly between the fields and should not be compared. Falsely accusing a student of academic misconduct has a radically different impact than not algorithmically promoting a social media post that looks too much like generated text. - All the previously published benchmarks on LLM detectors are flawed in principle because they do not consider the tradeoff between the generalization to previously unseen evasion attacks (TPR generalization) and previously unseen types of human-written texts (FPR generalization). While this tradeoff is intuitive, it has neither been stated in the LLM detectors literature nor explicitly benchmarked as a tradeoff in the past, leading to real-world LLM detectors issues with FPR generalization, notably false positives on text written by non-native writers [2]. We hope that this conceptual contribution to the field in the testing methodology alleviates the concerns about the paper's suitability for the ICLR conference stated by the reviewer in question Q1. Finally, we also understand the reviewers' suggestion in W3 to extend the evaluation of the detectors to SotA commercial API-only models. We apologize for failing to mention in our threat model that we assume such API providers will be monitoring for and blocking disinformation operations using their models, such as OpenAI recently reported doing [3]. Again, we thank the reviewer for their feedback and apologize for the lack of clarity in our original manuscript. We hope that our response fully alleviated the reviewer's concerns. To prevent any such confusion in the future, we have revised the 1.4 "Contributions" section to reflect better what we stated in the 5 "Conclusion" section and adjusted our threat model presentation to respond to W3. We have performed an additional pass to correct grammatical errors we could find in the main text. [1] Dugan, Liam et al. "RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors." Annual Meeting of the Association for Computational Linguistics (2024) [2] Arp, Daniel et al. "Dos and don'ts of machine learning in computer security." USENIX Security 31 (2022) [3] Metz, Cade "OpenAI Says It Disrupted an Iranian Misinformation Campaign" The New York Times 16.08.2024 (2024)

Author's response - 1/2

We thank the reviewer for their feedback. While the provided feedback is relevant, we believe that the weaknesses stated by the reviewer might stem from our failure to convey the core contributions of our work sufficiently clearly. Specifically, in response to the reviewer's request for a contribution summary, the heart of our contribution is showing that the methodology of the prior LLM benchmarks is flawed in principle. Specifically, 1. We show that large-scale static benchmarks, such as RAID [1], are not suited for evaluating LLM detectors in a way relevant to the real-world setting. This issue is due to: - LLM-generated text detection is inherently happening in adversarial settings, with the attacker seeking to evade detection. As such, reference static benchmarks provide a baseline not only for the LLM detector developers but also for attackers, who will learn to evade them. This adversarial setting radically differs from the general LLM evaluation practices, where benchmarks are useful to validate the model's capabilities. While we are the first ones to state this in the context of LLM detection, this difference in evaluations of ML solutions in adversarial settings has been previously highlighted, notably in cybersecurity [2]. In particular, the failure of recent benchmarks to test for temperature increase attacks led to all detectors being vulnerable to it, constituting an example of such an attack - LLM detectors' performance varies wildly depending on the task, even for closely related tasks, making large-scale benchmarks non-useful even for the best detector selection task, even of highly related topics, such as ours and "news articles" in the RAID benchmark [1], including for identical attacks, as highlighted in 4.3 "Prior Benchmarks" section. 2. We show that all the previously published benchmarks on LLM detectors are flawed in principle because they do not consider the tradeoff between the generalization to previously unseen attacks (TPR generalization) and previously unseen types of human-written texts (FPR generalization). While this tradeoff is intuitive, it has neither been stated in the LLM detectors literature nor explicitly benchmarked as a tradeoff in the past, leading to real-world issues with LLM detectors FPR generalization, notably false positives on text written by non-native writers [3]. 3. We propose a method to test LLM detectors that is useful for evaluating their real-world performance, that can be dynamically expanded, and provide code to achieve it. Specifically, we do not provide a dataset to evaluate LLM detectors on, but a method to derive an evaluation dataset from samples of texts in an application scope, given a threat model, which is radically different from prior work in the field, that instead attempted to build universal, scope and threat model-agnostic datasets. Overall, our contribution is methodological, first and foremost, for the machine learning field. We focus on a narrow application scope (generated short news-like content) and a specific threat model (moderately sophisticated attacker), and we make sure that we develop an adapted testing methodology. This deep dive focuses on getting the methodology right and differs from prior benchmarking works that attempted to cover as many models, attacks, and topics as possible; an approach that we believe, for reasons stated above, is flawed. We hope that this makes it clear to the reviewer why we disagree with their W1 suggestion to use common benchmark datasets in principle, given the methodological flaws of that approach stated above. Similarly, in the context of not presenting contributions 1-3 above sufficiently clearly in our original manuscript, we understand the reviewer's question regarding the choice of text length stated by the reviewer in W2. However, once again, for methodological reasons above, an evaluation of LLM detectors translating to real-world performance requires a choice of a specific scope. Consistently with prior literature on the potential for negative impact of publicly accessible powerful LLMs, we chose the news-like texts on social media in information operations, opting to optimize the methodology in that context rather than trying to be overly general. Once again, we do not claim to contribute a new LLM detector evaluation dataset, nor do we attempt a universal evaluation.

Author response

We thank the reviewer for their feedback. While we understand the weakness pointed out by the reviewer, we respectfully disagree. Given the existing literature on the subject of LLM detector benchmarking, we believe that our paper provides three novel conceptual contributions. - There is an inherent trade-off between detector generalization across evasion techniques and generalization across human-written texts. While this trade-off intuitively makes sense, it has neither been stated nor ever benchmarked in the LLM detection literature, which has in the past led to reports of poor performance of detectors on non-native writers [1]. - A relatively simple strategy - round-robin - was able to train LLM detectors that generalized well across new LLMs, both conversationally fine-tuned from foundational models in the training set and unrelated families, and across evasion attacks. While previously reported in the context of image-generating GANs, Round-Robin's boost to generalization capabilities in text classification with fine-tuned encoder LLMs has previously been unreported. - Large-scale static benchmarks are not suited for LLM detector evaluation due to the inherently adversarial nature of the setting and high variability of performance metrics across application domains, meaning that even in closely related settings - such as the one we considered and the "news articles" in the RAID benchmark [2], that we highlighted in 4.3 "Prior Benchmark" section. Once again, while not entirely surprising, this has not been previously reported in the context of LLM detectors and suggests large-scale benchmarks that dominated the literature until now might not be useful even for selecting an appropriate detector and suggests a need for a scope in the domain of application and threat model, as we suggest. Given that those conceptual results were dispersed in the introduction, discussion, and conclusion, we adjusted our paper to clarify this field-level contribution in the Abstract and 1.4 "Contributions" section. Responding to the questions of reviewers: Q1: Based on the results in Table 6 (Appendix), we do not observe any clear trends between Gemma-Chat (2B), LLaMA-3-8B-Instruct, and Zephyr-7B-Beta. We believe this suggests that the model size does not impact its detectability or evasion capabilities. Q2: This is an excellent question. While we believe it is out of scope for this work, the performance of the ELECTRA-RoundRobin detector suggests that applying a similar technique for both human and LLM-generated texts, increasing the diversity of the former might lead to interesting results. Unfortunately, we are unaware of any news text datasets focusing on the diversity and representation of human variability, and we will be thankful for any pointers. Again, we thank the reviewer for their feedback and apologize for the lack of clarity in our original manuscript. We hope that our response fully alleviated the reviewer's concerns. To prevent such confusion, we have revised the 1.4 "Contributions" section to better reflect what we stated in the 5 "Conclusion" section and adjusted the abstract. [1] Liang, Weixin et al. "GPT detectors are biased against non-native English writers." Patterns 4 (2023) [2] Dugan, Liam et al. "RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors." Annual Meeting of the Association for Computational Linguistics (2024).