How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse

Summary

This paper presents a theoretical analysis on language model collapse from model generated synthetic data, in a recursive training setup. In this setup, the new generation of model is trained on synthetic data generated from the last generation. From theoretical perspective, the authors prove that model generated synthetic data will cause language model collapse when it is the single data source in the recursive training setup. When combining with real training data, the authors give an upper bound of model generated synthetic data to avoid the collapse. Besides, this paper also provides some empirical results and analysis to align with their theoretical anaylsis. Overall, the conclusions and analysis presented in this paper are interesting and insightful for language model collapse related research. It is an important step to understand the impact of model generated data in LLM training.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

This paper presents a solid theoretical analysis to reveal the impact of model generated synthetic data in recursive training setup. This paper would be interesting to LLM developers and researchers to get them more insights.

Reasons to reject

Though the analysis is interesting, all the conclusions in this paper have a strong condition: recursive training setup. This condition might not apply to most of the existing LLM effort. Besides, the empirical results in this paper are based on a lightweight setup, while realistic LLM training has much larger scale. It is not clear if the same empirical results can hold on large scale setup.

Summary

The paper addresses a problem that has emerged as part of the move towards large-scale generative AI content. More specifically, the contribution is primarily of a theoretical nature but this also gets supported by some corresponding experimental work. The research question is to establish what happens to a model that gets augmented with synthetic data (or is entirely based on synthetic) data when recursively being trained on the output of the next generation of training. The take-home message is that fully synthetic data will ultimately lead to what has been coined “model collapse” in prior work. However, augmenting real data with synthetic data is shown to be less prone to the problem, in particular when the proportion of synthetic data is being controlled.

Rating

6

Confidence

2

Ethics flag

1

Reasons to accept

There are several key strengths of the paper including: (1) Topically this is a great fit. (2) The use of synthetic data to increase the pool of training data has become a very active research area, in particular since the release of GPT-2 a few years ago. This could be done for a number of reasons but often due to the lack of quality training data (and sometimes the cost associated with annotating more). My reading of the paper is that the authors foresee a future where much of the data out there is being generated by LLMs (although I do not think this is made explicit in the paper). (3) The authors provide a detailed theoretical analysis and back up their findings with (some limited) experimental work.

Reasons to reject

I have a number of concerns, some about the underlying assumptions and others in regards of the experimental work: (1) The use of synthetic data has been demonstrated to be effective in some settings (and for some NLP tasks) but not for others. However, the usual approach appears to be that of data augmentation where a proportion of synthetic data is being generated (and then filtered before being added to the original dataset). I also understand the appeal of plain synthetic data in circumstances where data privacy might be an issue (e.g. synthetic patient records). However, I wonder how realistic the assumption of continuous recursive training is. Clearly, theoretically it is something worth exploring but I am lacking the actual use case. (2) In the discussion of related work I would like to hear more about different approaches / paradigms / families of synthetic data generation and then (later) have a discussion about how that choice might affect the findings. (3) The assumptions made in the theoretical setup (Section 2.1) simplify the concept of language models quite substantially (e.g. not considering transformer architectures). How generalisable are the findings with such simplifying assumptions? (4) Similarly, the choice of experimental setup in Section 4 is also quite narrow. A fairly tiny corpus is picked on a simple prediction task and GPT-2 is being deployed. All this hints at difficulties of obtaining generalisable insights for practical applications at scale. (5) There might be some good answers to my concerns and part of this should probably be done as part of a Limitations section. Currently there is no such discussion. Some other issues: (1) It would be good to include some compelling running examples. The authors reference (Shumailov et al., 2023), and that paper appears to be doing a great job in that respect. (2) Almost all entries in the bibliography have essential details missing.

Questions to authors

Please see comments above

Summary

The paper performs a theoretical analysis of recursive training loops in language model training, i.e., where synthetic data generated by language models contaminates the data used for training future iterations of language models. Prior work has found that this phenomenon leads to "model collapse," where models' outputs become very limited as the amount of synthetic data in the training set increases. This paper builds on one of these works (Shumailov et. al. 2023), particularly exploring the setting of models for next token prediction. They conclude that theoretically, model collapse should consistently occur in the fully synthetic scenario. In the partially synthetic scenario, an upper bound on the proportion of synthetically generated data in the training set is identified to mitigate model collapse. The authors confirm these results using experiments with toy models.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

* The topic of the paper is very relevant, since the amount of model generated data on the web has drastically increased in recent years. Previous studies have explored this phenomenon empirically, but few have provided theoretical insights, which are important when deciding how to build the next generation of of language models. * The derivations seem sound, although there is a chance that I missed an error in one or more of the proofs. * There is a comprehensive discussion of related work, giving a nice overview of the recent study of model collapse * The authors generally provide good intuition for the mathematical results

Reasons to reject

* The presentation could generally use some work. The paper is dense and hard to follow. For example, in the setup in 2.1, there’s switching between terms, e.g., x_l and e_j which are multiplied in the final linear layer, and N and M for number of training samples. I can’t follow the reasoning for the use of these multiple terms, and they make section 2.1 hard to understand. * The theoretical analysis is only for a single context. It’s unclear which conclusions would change when considering the large number of possible contexts. While the authors point this shortcoming out, the absence of such an analysis makes the paper much weaker, in my opinion, since this is the main contribution. * Experimental results are only for toy settings, although it would be difficult to perform experiments on realistically-sized language models with realistically-sized datasets * The contributions above Shumailov et. al. 2023 are arguably limited

Questions to authors

* When commenting on the issue that it “is impossible that the learned model captures all the information about the original distribution”, is the issue that results from this related to the capacity of the model being trained, or to the data generated from previous models? * By canonical vectors, do you mean canonical basis vectors? They’re called one hot embeddings elsewhere, and it may be more clear to stick to one term. * After some thought, I can see why all models p^{(m)} are unbiased, but I think it would be best to explain this more explicitly to the reader

Response to rebuttal

I thank the authors for their response to my concerns. The rebuttal does contain partial answers to some of my concerns and I assume that the authors will update the paper accordingly if invited to submit a revision. My two concerns regarding (a) tiny sample size and (b) lack of realistic use case remain as before. I also note that these two points have been raised in the other reviews as well. Nevertheless I will raise my overall score slightly as I do not think that there is a systematic flaw in the work. It simply is not strong enough in my view to warrant a clear accept.

Response

Thank you for the clarifications. While some of my concerns are assuaged, my overall perception of the work remains the same. I will therefore keep my score the same.