Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop
Large Language Models (LLM) are already widely used to generate content for a variety of online platforms. As we are not able to safely distinguish LLM-generated content from human-produced content, LLM-generated content is used to train the next generation of LLMs, giving rise to a self-consuming training loop. From the image generation domain we know that such a self-consuming training loop reduces both quality and diversity of images finally ending in a model collapse. However, it is unclear whether this alarming effect can also be observed for LLMs. Therefore, we present the first study investigating the self-consuming training loop for LLMs. Further, we propose a novel method based on logic expressions that allows us to unambiguously verify the correctness of LLM-generated content, which is difficult for natural language text. We find that the self-consuming training loop produces correct outputs, however, the output declines in its diversity depending on the proportion of the used generated data. Fresh data can slow down this decline, but not stop it. Given these concerning results, we encourage researchers to study methods to negate this process.
Paper
Similar papers
Peer review
Summary
The authors test whether the re-use of already-generated examples from an LLM in further pre-training of the same model result in lower quality output. They measure the output in two ways: correctness and diversity. They use a clever, if somewhat limiting, approach to automatic evaluation: teaching the model to generate logic statements which evaluate to `True`, e.g., by the Python `eval` command, allowing them to evaluate correctness easily. To evaluate diversity they examine the number of unique statements generated as well as their diversity, measured using the Levenshtein Diversity, an averaged, normalized Levenshtein distance between all pairs of generated statements. Within this framework, they show that the re-use of previously generated statements as further training data invariably makes the model more "correct" in the sense that it generates only True statements, but also that the diversity inevitably suffers, in the most extreme cases to the point that the model generates only one output. Diversity begins to fall dramatically before the outputs become truly non-unique.
Strengths
The problem is very clearly framed and good background from the similar problem in image generation is helpful. The author's approach to make a tractable experiment by using logic statements is quite clever, and makes it easy to understand the impacts of "self-consuming" training. The authors present a solid range of experiments, focusing on generations of self-consumption, different degrees and styles of self-consumption, and--in the appendices--model size. The results are stark and convincing, at least in the context of the logic statements they evaluate.
Weaknesses
The authors are aware of some of the limitations of the work--it is possible to claim that a sufficiently large model might not suffer from these problems, but the authors have made a good attempt to address this up to a certain scale, and as they point out the cost (both in dollars and CO2) of such an experiment at large scale would be prohibitive and probably unethical. A more subtle limitation might be that the range of language tested in this paper is necessarily quite small. It would be interesting to extend the experiments to a larger language, perhaps something like SQL which is still easily verified but has a richer vocabulary.
Questions
My main suggestion is to try this work with a more sophisticated but still machine-evaluatable language, such as SQL. This is a harder problem, but one with probably more diversity, and it might help in making the results more robust to any criticism that the language was too simple. I realize that would be difficult or impossible to do for this submission, but it would make the work stronger.
Rating
10
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Response to reviewer sjdf
Dear reviewer sjdf, Thank you for taking the time to review our paper. We are grateful that you appreciate the benefits of our approach using logic expressions and the rationale behind the design of our experiments. As stated in our limitations, we acknowledge that our approach might not fully capture the complexity of natural language as also pointed out by the other reviewers. While we find your suggestion using SQL statements as a more complex yet verifiable language very interesting, you are correct in assuming that this would not be possible in the timeframe of this rebuttal. Therefore, we would like to pursue this direction in future work. Nevertheless, to address this concern along with the concerns of the other reviewers, we conducted additional experiments using a real (but manageable) natural language dataset for all four data cycles. We find a similar decline in diversity as in the logic expression experiments for all data cycles. Again, the full synthetic data cycle suffers the most from the self-consuming training loop but the other data cycles also suffer from the effects. We inspect the generated outputs manually for correctness (in terms of combinations of tokens that lead to hallucinated words not present in the original dataset) and find similar observations as for the logic expressions. This shows that our results indeed generalize past the domain of logic expressions. We added the results to our manuscript and hope those additional results can alleviate your concerns. If you have any further questions do not hesitate to ask. Sincerely, The author(s)
Response to reviewer sjdf
Dear reviewer sjdf, With the deadline for discussion approaching in a few days, we would like to ask you if we managed to address your concerns in our rebuttal or if you have any further need for clarification or suggestions. We are looking forward to your continued feedback. Sincerely, The author(s)
Dear Authors, First, please accept my apologies for the delay. The addition of the tiny Shakespeare dataset does bolster your work, and I am pleased to raise my scores slightly as a result. Best, sjdf
Response to reviewer sjdf
Dear reviewer sjdf, Thank you for your response and continued support of our work. We are pleased that our additional experiments did further convince you of our paper. Sincerely, The author(s)
Summary
This paper presents an initial study of the influence of self-consuming style training on the generation quality of LLMs. A logic expression dataset is used to avoid the inaccurate measurement of generated text quality or diversity. In the evaluation part, four types of self-consuming training are incorporated to comprehensively compare the effect of self-generated data.
Strengths
* This paper studies an interesting question, which is a valuable contribution to the further development of LLM and a deeper understanding of self-consuming training. * The paper is well-written and easy to follow.
Weaknesses
* The author missed a series of related works on self-consuming training (self-training) of LLM, and here are some examples [1][2][3]. * The experiment setup is not convincing to me. The generation quality is not equivalent to the correctness of the generated logic expression. Therefore I don't think the evaluation successfully reflects certain properties of self-training of LLMs. * Logic expressions merely form a small part of data that LLMs can generate. Experiments with only logical expressions cannot lead to a general conclusion, for instance, self-training in LLM may or may not result in XXX. [1] ReST-MCTS∗: LLM Self-Training via Process Reward Guided Tree Search [2] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [3] Reinforced Self-Training (ReST) for Language Modeling
Questions
See weaknesses.
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Response to reviewer HkTf
Dear reviewer HkTf, Thank you for reviewing our manuscript and providing helpful feedback for improvement. We would like to address your concerns below: **W1:** Thank you for raising this concern, however, we believe that there is a misunderstanding. We are not interested in the case of deliberately using synthetic data in terms of data augmentation or self-training of LLMs. Instead we investigate the emerging self-consuming loop that happens as more and more LLM-generated data is published on the internet without labeling as such. In this setting researchers do not know the source of the data and cannot carefully filter or adjust the data which is different to the setting you mention. We apologize for this confusion and added a paragraph to Sect. 2 to emphasize this difference and citing the references you mentioned, as well as carefully revising our discussion section. **W2 and W3:** Thank you for your feedback. We agree that while designing our experiments with logic expressions makes them tractable and enables us to unambiguously evaluate the results, the logic expression dataset might not fully capture certain properties of real textual data. Therefore, to alleviate your concerns together with the concerns of the other reviewers we also conducted additional experiments with a real textual dataset. We find a similar decline in diversity as in the logic expression experiments for all data cycles. Again, the full synthetic data cycle suffers the most from the self-consuming training loop but the other data cycles also suffer from the effects. We inspect the generated outputs manually for correctness (in terms of combinations of tokens that lead to hallucinated words not present in the original dataset) and find similar observations as for the logic expressions. This shows that our results indeed generalize past the domain of logic expressions. We hope this addresses your concerns and if you have any additional questions do not hesitate to ask. Sincerely, The author(s)
Thank you for your clarification. I appreciate the new experiment on natural language generation. I am raising my score accordingly.
Further response to reviewer HkTf
Dear reviewer HkTf, Thank you for acknowledging our new experiments and increasing your rating from 3 to 5. Since we have addressed your concerns and you initially found our work to be a “valuable contribution”, we would like to ask you if you have any further questions or concerns that we could address to convince you that our work is worth publishing (6+ rating). We are looking forward to your feedback and thank you for your ongoing engagement. Sincerely, The author(s)
Summary
In this article, the authors design a series of experiments to explore the impact of self-consumption training loops on large language models, in which the authors use Levenshtein diversity as an evaluation metric for data correctness and diversity. Through the experiments, the authors argue that training a large language model using data generated by the large language model leads to a decrease in the correctness and diversity of expression of the model, and analyze the corresponding challenges of current large language model training.
Strengths
1、 The effects of different data self-recycling methods, different model sizes, and the number of model iterations on language models are compared in detail. 2、 Levenshtein diversity is used as an evaluation index of data correctness and diversity. 3、 The harm of the phenomenon is analyzed.
Weaknesses
1. The literature: “AI models collapse when trained on recursively generated data” in the recursive data fine-tuning language model has given the corresponding experimental proof of the phenomenon of the collapse of the language model and the conclusion. It is not the case, as stated in the introduction, that only the use of recursive language training in the image domain leads to model collapse. This paper explores the effect of data self-recycling on training in the non-fine-tuning case from a different perspective, with limited innovation. 2. Levenshtein diversity has long been applied to measure text diversity, and is not considered NOVEL.(Ref:Beijering, Karin, Charlotte Gooskens, and Wilbert Heeringa. "Predicting intelligibility and perceived linguistic distance by means of the Levenshtein algorithm." Linguistics in the Netherlands 25, no. 1 (2008): 13-24.) 3. The number of model parameters used in the experiment is low and the type of data used is single, so the relevant conclusions obtained are not rigorous.
Questions
1. It is recommended that “output” be specified in Algorithms 1 and 2 to enhance the overall readability and comprehensibility. 2. The literature: “AI models collapse when trained on recursively generated data” has given corresponding experimental proofs and conclusions on the phenomenon of recursive data fine-tuning of language models leading to the collapse of language models. We hope that the authors will adjust the content of the article so as not to mislead the readers. 3. Figure 3 shows that using complete synthetic data to train the NGL model does not produce error data, while mixing artificial data with synthetic data produces syntactically incorrect outputs, why is that? 4、 It is suggested to give the formula of the indicator ' Average Normalized Levenshtein Distance ' to make it easier for readers to understand. 5、 Using logic expressions as training data does have some unique advantages in evaluating the semantic correctness of the output of a language model, but there are differences between this kind of data and real textual data, and I'm curious how the semantic correctness and variety of outputs in the article would be reflected on real textual data?
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
2
Response to reviewer WoA9
Dear reviewer WoA9, Thank you for your valuable feedback and suggestions that helped improve our manuscript. Below you find a response to your concerns and actions taken to alleviate them. **W1 and Q2:** It was not our intention to mislead the reader. In fact, we are familiar with the reference you are citing and already discussed it in the related work including the fine-tuning experiments. However, we added a sentence to the introduction to further clarify the literature and our contribution. Nonetheless, our contribution is still novel and valuable as the self-consuming training loop manifests in the pre-training rather than the fine-tuning step of LLMs, as the origin (human- or LLM-generated) of data collected from vast sources on the internet is not clearly determinable. Therefore, investigating this setting is important. **W2:** There appears to be a misunderstanding. We did not claim that using Levenshtein diversity is a novel approach and we cited the reference that inspired us to do so. The novelty of our experiments is using the logic expression dataset to systematically analyze the output of an LLM trained in a self-consuming way. Nonetheless, we would like to thank you for pointing us to the additional literature and we added the citation accordingly in Sect. 3.3. **W3 and Q5:** Regarding your first concern: While we agree that carrying out experiments with large scale models would be interesting we argue that such experiments are not feasible. Nevertheless, as you stated we conducted experiments with different model sizes and found no impact on the self-consuming training loop. Regarding your second concern: This is a valid criticism, also shared by the other reviewers. While our approach enables us to systematically analyze the outputs in a tractable setting, it would still be very interesting to investigate if the effects of the self-consuming training loop can still be observed for natural language. Therefore, during the rebuttal phase we conducted additional experiments using a common natural language corpus. We find a similar decline in diversity as in the logic expression experiments for all data cycles. Again, the full synthetic data cycle suffers the most from the self-consuming training loop but the other data cycles also suffer from the effects. We inspect the generated outputs manually for correctness (in terms of combinations of tokens that lead to hallucinated words not present in the original dataset) and find similar observations as for the logic expressions. This shows that our results indeed generalize past the domain of logic expressions. We added those additional results to our manuscript. **Q1:** Thank you for pointing that out. We adjusted the pseudocodes accordingly. **Q3:** Thank you for your question. The construction of a correct logic expression is not a trivial task where all tokens need to be semantically correct. Even a well trained LLM has a chance to sample a token that leads to a non correct expression. The logic expressions also grow in size during our experiments (later generations produce longer expressions) making generating a correct expression potentially harder. We hypothesize that the rapid decrease in diversity in the full synthetic data cycle leads to a collapse in diversity before this increase in size can affect the ability of the model to fully capture the problem. **Q4:** Thank you for the suggestion. We added a section to our appendix with formulas for the Levenshtein diversity as well as the diversity metrics used for the natural language experiments (see App. C). We hope this answers your questions and if you have further questions do not hesitate to ask. Sincerely, The author(s)
Response to reviewer WoA9
Dear reviewer WoA9, With the deadline for discussion approaching in a few days, we would like to ask you if we managed to address your concerns in our rebuttal or if you have any further need for clarification or suggestions. We are looking forward to your continued feedback. Sincerely, The author(s)
Response
Thank you for your efforts in preparing the rebuttal and incorporating the new experiments. After carefully reviewing your response, I have decided to raise my score accordingly.
Response to reviwer WoA9
Dear reviewer WoA9, Thank you for your continued engagement and appreciation of our rebuttal. We are glad that we were able to address your concerns and that you raised your score as a result. Sincerely, The author(s)
Summary
This paper empirically investigates the self-consuming training loop in GPT-style models. To unambiguously verify the correctness of LLM-generated content, the authors propose to use logic expressions, as opposed to natural language, to construct the training dataset. The authors conducted comprehensive experiments with four distinct data cycles, and the generated content is evaluated in terms of both correctness and diversity. The experimental findings reveal that utilizing the model's own generated data leads to a decline in diversity.
Strengths
This paper tries to reveal a critical issue in the data construction of LLMs, given the extensive use of WEB data and the challenges in distinguishing LLM-generated content from human-produced content. The authors conduct comprehensive experiments to explore the potential risks associated with directly feeding LLM-generated content into models, highlighting the limitations and risks of the self-consuming training loop.
Weaknesses
Regarding the sampling methodology and synthetic content: The experimental setup deviates significantly from real-world LLM pretraining scenarios. In practice, the vast majority of pretraining data for open-sourced LLMs originates from human-written sources such as books and web pages. Even when synthetic content is utilized, it is often generated in diverse styles (e.g., textbooks, QA) based on existing human-written web content, rather than being directly sampled from LLMs for training purposes. The authors are encouraged to consider incorporating more realistic data sources and synthetic methods to enhance the practical relevance of their findings. Concerning the logic expression dataset: While constructing datasets based on logic expressions facilitates the verification of content correctness, it also oversimplifies the underlying problem. The authors should consider the potential impact of the diversity and complexity of logic expressions on the experimental outcomes when designing their experiments. This could involve exploring the use of more complex logic expressions or incorporating elements of natural language to better reflect real-world scenarios. Regarding the base model: It is recommended that the authors conduct additional pretraining using existing open-sourced LLMs, incorporating model-generated content to further enhance the practical relevance and generalizability of their findings.
Questions
How do the initial complexity and diversity of the logic expression datasets influence the final conclusions drawn from the experiments? It would be beneficial to explore the sensitivity of the results to variations in these factors. If the new dataset is synthesized by referring to the original dataset, rather than being directly sampled from LLMs, would the diversity of the final output still exhibit a decline? This question aims to investigate the potential impact of different dataset synthesis methods on the diversity of the generated content.
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Response to reviewer nsdg
Dear reviewer nsdg, Thank you for reviewing our work and providing important feedback on our paper. We address your concerns below. **W1 and W3:** Thank you for raising this concern, however, we believe that there is a misunderstanding. We are not interested in the case of deliberately using synthetic data in terms of data augmentation or self-training of LLMs. Instead we investigate the emerging self-consuming loop that happens as more and more LLM-generated data is posted on the internet without labeling as such. Therefore, traditional web scraped data used for pretraining, including books, websites, and even scientific writing, is containing LLM-generated data with no way to clearly differentiate it from human-generated data. To account for the mixture of human- and LLM-generated data in this setting we designed the different data cycles used in our experiments. We apologize for this confusion and added a paragraph to Sect. 2 to emphasize this difference as well as carefully revising our discussion section. **W2:** Thank you for pointing this out. While the logic expression dataset enables us to design tractable experiments and to unambiguously evaluate the results, we agree that it might not fully capture the complexity of natural language. However, we argue that an experiment for many generations with large scale text corpora and state-of-the-art model sizes is not feasible nor responsible in terms of both monetary costs and CO2. Nevertheless, we conducted additional experiments for all four data cycles with a real (but manageable) text dataset to alleviate your concerns. We find a similar decline in diversity as in the logic expression experiments for all data cycles. Again, the full synthetic data cycle suffers the most from the self-consuming training loop but the other data cycles also suffer from the effects. We inspect the generated outputs manually for correctness (in terms of combinations of tokens that lead to hallucinated words not present in the original dataset) and find similar observations as for the logic expressions. This shows that our results indeed generalize past the domain of logic expressions. We added the additional results to our manuscript. **Q1:** Thank you for your question. We hypothesize that the initial diversity only has a limited outcome on the self-consuming training loop. A lower initial diversity would most likely result in less generations until a collapse occurs while a higher initial diversity could delay the effect. This is further supported by our additional experiments for natural language. While the diversity of both datasets (logic expressions and textual data) is not directly comparable it is certainly different to each other. Nevertheless, we still observe the effects of the self-consuming training loop for both datasets. **Q2:** Similar to our answer to W1, we want to clarify that we do not investigate the deliberate use of synthetic data to train LLMs. Instead, we focus on the emerging self-consuming training loop that happens as more data is published on the internet without labeling as such. As a result, synthetic data is unknowingly used in the pre-training of LLMs which is different from the setting you describe. Therefore, we do not consider other methods to generate synthetic data from the original dataset in our experiments. However, investigating how deliberately generating data from known human sources and using them within a self-consuming training loop could be an interesting research direction. We hope this addresses your concerns and if you have any additional questions do not hesitate to ask. Sincerely, The author(s)
Thank you for your efforts in preparing the rebuttal and including new experiments. After carefully considering your response, I have decided to increase my score accordingly.
Further response to reviewer nsdg
Dear reviewer nsdg, Thank you for your ongoing engagement and increasing your rating based on our rebuttals. As we are committed to further improving our work, we would like to kindly ask you to let us know if you have any further concerns or suggestions for improvement that is preventing you from recommending publication (6+ rating). Thank you for your time. Sincerely, The author(s)
Overall response by the author(s)
We would like to thank the reviewers for the insightful and constructive feedback provided on our manuscript. The reviewers found our paper **well written and easy to follow** (*HkTf*), with a **clearly framed** problem setting (*sjdf*). They also found our paper to study a **critical issue** (*nsdg*) and **interesting question**, which is a **valuable contribution to the further development of LLMs** (*HkTf*). Reviewers also found our experiments to be **comprehensive** (*nsdg*), covering a **solid range of experiments** (*sjdf*), and **comparing different effects in detail** (*WoA9*). Reviewer *sjdf* also pointed out that our **logic expressions approach is quite clever** making the **experiments tractable and easy to understand**. However, the biggest concern among reviewers was missing experiments for real textual data. Therefore, we conducted **additional experiments for all four investigated data cycles using a real but manageable text dataset** (tiny Shakespeare dataset) further extending our results and contribution. We investigated the diversity of model outputs in a self-consuming training loop using multiple diversity metrics and found a **similar decline in diversity** for text as in our experiments with logic expressions. Regarding correctness of the generated text we investigated the outputs manually. Again, similar to the logic expressions we observe a drop in correctness (combinations of tokens that lead to hallucinated words) in the first generation. By the end of the self-consuming training loop the (non-diverse) outputs are mostly correct (in terms of correct combinations of tokens to produce non hallucinated natural language) similar to the logic expression experiments. However, the decline in diversity makes the models unusable. **This shows that our findings regarding the self-consuming training loop for LLMs are indeed generalizing past logic expressions.** To reflect these findings in the paper, we extended our method (new Sect. 3.2 and adjustments to Sect. 3.3 and Sect. 3.5) and the result section (new Sect. 4.3 and App. D.5) in the main paper with all the details as well as example text outputs for different generations and data cycles in the appendix (new App. E.2). We also provide all generated text samples in the supplementary material. Additionally we provide a response to each reviewer below, addressing their other questions and adjust our manuscript accordingly. We hope we could alleviate the concerns of the reviewers and are looking forward to answering any further questions. Sincerely, The author(s)
Official comment by the author(s)
Dear reviewers, As the extended discussion deadline is quickly approaching, we would like to kindly ask you to take a look at our response to your reviews. If you have any further questions or need for clarifications, please do not hesitate to ask. If we have managed to sufficiently address your concerns, we would be grateful if you could reconsider your original assessment of our work. In any case, we look forward to your continued engagement and feedback to further improve our work. Sincerely, The author(s)
Final comment by the author(s)
Dear reviewers, AC, SAC, PC, We would like to thank everyone involved in the reviewing process for their time and engagement, especially the reviewers whose feedback helped to improve our work. Initially, reviewers agreed that our work is a **valuable contribution** (*HkTf*) with a **solid range of experiments** (*sjdf*). During the rebuttal period we provided **answers to all reviewers as well as additional experimental results** that helped **bolster our work** (*sjdf*). All reviewers appreciated our response to their concerns and raised their score as a result. We believe that we **successfully addressed their points** as no reviewer raised new concerns or had follow-up questions during the discussion phase. We are looking forward to feedback from the AC. Sincerely, The author(s)
Meta-review
The article investigates the effects of using a large language model's (LLM's) own generated data to further train itself (a "self-consuming training loop"). To accurately assess the generated content, the authors used logic expressions (which can be automatically evaluated for correctness) as training data, rather than natural language. The design makes the evaluation easy and task. They conducted experiments with four different self-consumption training cycles, evaluating the generated output for both correctness and diversity using metrics like the number of unique outputs and Levenshtein distance. The key finding is that while retraining with self-generated data can increase the correctness (in this case, generating valid logic statements), it consistently decreases the diversity of the output, The reviewers agree it is a critical problem to investigate, and the proposed approach to make a tractable experiment by using logic statements is clever. The conclusion on its own setting is also supported by the experiments results. The major concern is its generalizability. The setup in the study is significantly different from the real world LLM pre-training scenarios. First, demonstrating similar effectiveness in a more realistic domain would strengthen the work's relevance, e.g. SQL query generation. Second, the evaluation focuses solely on smaller scale data and models without comparison to established baselines. Given the known performance differences between model scales, it is unclear whether these findings generalize to larger models.
Additional comments on reviewer discussion
The reviewers discussed about its limitation of the generalizability. Reviewer sjdf still supports the paper (while he stated he might step back from rating the paper a 10) and it is certainly worth publishing. Other reviewers are still not convinced and maintained their original scores. I think the concern is valid, and the paper will be much stronger with the suggested changes.