Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences

Summary

The authors investigate the properties of self-consuming loops that arise in the training of generative models. In particular, they investigate the impact that data curation has on the iterative retraining performance of these models. The paper contains theoretical and empirical analysis of how model performance is affected by various data curation assumptions (only synthetic examples, or real data being injected at each step, or human-preference curated synthetic samples, etc).

Strengths

Originality: I believe this work is highly original, largely because it considers a new problem formulation--"what happens to self-consuming generative models when the synthetic data that they re-train one has been curated via human preferences?". To my knowledge, nobody has considered this problem, and it is an excellent and timely problem to consider. Clarity: The authors motivated the problem very clearly. Significance: many previous works have investigated self-consuming loops, but the area is still burgeoning, and right now, the area seems like "the wild west"--there are lots of papers out there with different assumptions, different results, no agreed-upon benchmarks tasks, etc. This paper is significant because it considers a more realistic setup than previous papers; it considers the setting where the web-scale data contamination happens because of human-curated data. This is an important case to consider, and represents a large step towards modeling the data contamination issue more rigorously. This more realistic setup comes with more mathematical overhead, which is challenging to deal with.

Weaknesses

Clarity: in my opinion, the presentation of statements of the theorems in the paper needs improvement in order to be useful to the community. Consider for example Theorem 2.1. The assumptions for this theorem are distributed in the section preceding it, which makes it much harder to understand and contextualize that theorem. I would strongly suggest summarizing the assumptions and key notations in the statement of that theorem (and likewise for all the other results.) An excellent model for this would be the paper that the authors cited the most, Bertrand et al's "On the Stability of ..." . That paper's Theorem 1 (from the latest arXiv version) begins--"Given theta^\star as defined in Equation (7) that follows assumptions 1 and 2....". Given that a large part of this paper's contribution is its theorems, I would be likely to raise my score, and champion this paper, if I could first see an updated draft which makes explicit every assumption in the statement of each theorem. See also the limitations section.

Questions

I'm not sure I understand how the experiment supports the theory here, please help clarify this for me--at line 250, the authors say "applying theorem 2.1, the density will converge to a renormalized Gaussian distribution restricted to the ball centered at x_* of radius r_min". But this isn't what I see in Figure 4, it looks like there are a bunch of different Gaussian balls with different densities, corresponding to different intensities. Should these gaussian balls all be the same density, or did I misunderstand something? Some misc points that didn't affect my score/judgment of the paper, but the authors should consider fixing in the next draft: - improper formatting in equation 8 (spacing/parentheses) - awkward/grammatically incorrect wording in the sentence immediately after equation (7)

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

4

Limitations

The two main things preventing me from giving a higher score are the following two limitations. I'm looking forward to hearing the author's rebuttal to these points (and hopefully seeing an updated draft, if possible). 1. Presentation of statements of theorems (what are the specific hypotheses? eg take a look at the latest version of the bertrand et al paper on the arXiv for a good way to do this, since that paper is structured in a similar way. They number their assumptions and make them more clear. See "Weaknesses" section for more details.) This is the primary limitation, from my perspective, as I think the usefulness of the paper is very limited by unclear theorem statements. 2. Proper contextualization of these results relative to the literature--namely, there's a difference between the work in Bertrand et al and Alemohammad et al, but the authors seems to be comparing them in an "apples to apples" way. Namely, the former work considers the case of iterative fine-tuning, whereas the latter considers the case of retraining from scratch. In the former case, it is strictly easier to avoid model collapse, since the model parameter update is "local", and in the latter case, the updates are "global." Specifically, on line 218 it says: "Alemohammad et al. (2024); Shumailov et al. (2023) first evidenced catastrophic degradation of the generated data in the fully synthetic loop. Bertrand et al. (2024) mitigate these conclusions in the setting where the model is retrained on a mixture of synthetic and real data and they show the stability of the process around the data distribution." I think this is a false statement, because Alemohammad et al considered re-training from scratch at each iteration, which wasn't considered in the Bertrand et al paper--but please correct me if I'm misunderstanding. And in that same vein, I think it is important to properly contextualize the present paper in the presence of that dichotomy--does this paper consider iterative fine-tuning, or iterative re-training from scratch? That is important information for the readers/the literature.

Summary

This paper studies the impact of data curation on iterated retraining of generative models. Theoretical results are derived for the convergence state of the retraining loop when using a fraction of curated synthetic data or a mixture of real data and curated synthetic data at each step. Empirical experiments on both synthetic datasets and CIFAR-10 demonstrate that the proposed approach can bias the generative model to generate samples with higher reward.

Strengths

The problem of iterated retraining of generative models using curated data is important and interesting. The authors have indeed made some achievements in this direction. The theoretical results presented in this paper are interesting and reasonable, especially in their connection to preference optimization. The toy experiments are consistent with the theoretical claims.

Weaknesses

The writing of this paper needs further improvement. Some notations are unsuitable, and certain mathematical notations are introduced without explanation. Some literature references are missing. 1. In Eq.2, $\mathcal{BT}\left(x_1, \ldots, x_K\right)$, where "BT" refers to the Bradley-Terry model, is typically used to model pairwise preferences, whereas the Plackett-Luce (PL) model is proposed for preferences involving more than two items. Eq.5: If $p_{t+1}(x)$ is not a normalized density function, the same applies to $p_{t+1}(x)$ in Eq.6. 2. There are already studies on deep generative models (GANs) that investigate the convergence state of iterative retraining on curated data from the perspective of preference [1, 2]. It should include a discussion of these works. 3. The theoretical proofs especially the part in Section 2.2 are heavily inherited from the previous work [3], which somehow makes the theoretical contributions marginal. [1] Gupta, A., Zou, J. Feedback GAN for DNA optimizes protein functions. Nat Mach Intell. 2019. [2] Yao, Y., Pan, Y., et. al. Differential-Critic GAN: Generating What You Want by a Cue of Preferences. IEEE Transactions on Neural Networks and Learning Systems, 2022. [3] Bertrand, Q., Bose, et. al. On the Stability of Iterative Retraining of Generative Models on their own Data. In The Twelfth International Conference on Learning Representations.

Questions

1. Eq.3: The term $p_{ref}$ is not explained. What distinguishes $p_{ref}$ from $p_{data}$? 2. In Theorem 2.2, deriving $\lambda < 0.5$ based on the assumption $L \varepsilon < \alpha$ and $\lambda < \frac{\infty}{2 D \varepsilon}$ appears to conflict with the claim in line 181. 3. The explanation of $r_{\ast}$ in lines 134-137 is somewhat complex. Adding some equations would clarify its meaning.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes.

Summary

This paper extends Bertrand et al. 2024’s analysis of model collapse to study settings where data is filtered (based on a particular preference model) before being used for training the next iteration of generative models.

Strengths

- The paper extends prior work studying self-consuming generative models to integrate preference learning, which is highly sensible - The paper is well written

Weaknesses

- I do not think that Equation (3) faithfully models reality. Specifically, it assumes that the $t+1$-th model is fit to a mixture of (1) real data and (2) preference-filtered data from the $t$-th model. But realistically, synthetic data should amass over time, as I believe is the case with the datasets that the authors mention in their introduction, e.g., LAION-5B. This is a point made by "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data." and I agree with them. - I might be missing something, but intuitively, Theorem 2.1 seems wrong, specifically the claim “eventually converges to the highest level set of the reward reached **at initialization**”. For the simplest possible counter example, suppose p_0 is a discrete Uniform distribution on integers 0 to 10 and the reward function is $r(x) = x$. Defining $r_* := 10$, we see that Assumption 2.1 is satisfied, including the variant where $\epsilon \geq 0$. It is hard to imagine that the iterative loop would concentrate on $10$ rather than something greater than $10$. The intuition is that sampling and filtering from the first model iteration might shift the distribution towards regions where it is possible to sample higher rewards. - I think the experiments corresponding to Theorem 2.1(Figures 4 and 5) could be improved to better connect with the maths. Specifically: (1) Provide a heatmap of the level set of $r_*$ to show what $p_*$ is. (2) As I understand, the theorem doesn’t say that the distribution’s variance collapses, but rather, that the _reward variance_ collapses. In the figures, the reward functions are unimodal, and so we see the distributions converge towards unimodal behavior. You should modify the reward functions to be multimodal to demonstrate this distinction between the distribution's variance collapsing versus the reward's variance collapsing. One way to do this might be to define the reward function as 4 of the 8 MoGs (i.e. the reward is the max of the set of negative distances to the 4 chosen centroids). Then, we should see the model concentrate on those 4 chosen centroids. - I feel like I don’t understand Theorems 2.3 or 2.4. In Theorem 2.3, the lower bound on the right hand side does not appear to depend on the model fitting iteration $t$, as best as I could tell, nor does the upper bound in Theorem 2.4. Here, I’m expecting the answer to depend heavily on the model fitting iteration. I read the adjacent discussion but didn't receive any clarity. I'm consequently not sure how to evaluate the significance of Section 2.2. Perhaps the authors could clarify? - I think there are many additional papers you might want to cite. Some may be concurrent with yours (I intentionally did not search for a preprint in order to preserve double blind reviewing), and if so, that’s fine. Here are some suggestions on several different topics: **On Model Collapse:** Beyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement. https://arxiv.org/abs/2406.07515 Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. https://arxiv.org/abs/2404.01413 **On mode collapse in RLHF:** Understanding the Effects of RLHF on LLM Generalisation and Diversity. https://openreview.net/forum?id=PXD3FAVHJT A Distributional Approach to Controlled Text Generation. https://openreview.net/forum?id=jWkw45-9AbL Red Teaming Language Models with Language Models https://arxiv.org/abs/2202.03286 Improving alignment of dialogue agents via targeted human judgements https://arxiv.org/abs/2209.14375 Aligning Language Models with Preferences through f-divergence Minimization https://arxiv.org/abs/2302.08215 **On filtering data using reward models - often known by multiple names in the RLHF literature including “Best of N” or “rejection sampling” or “reranking” in the RLHF literature:** Scaling Laws for Reward Model Overoptimization https://arxiv.org/abs/2210.10760 Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback https://arxiv.org/abs/2204.05862 There are many more

Questions

N/A

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper explores the scenario where generative models are iteratively trained on self-generated data curated by human users with some implicit reward. The key idea is that each iteration of training on the self-generated data reweights the previous distribution based on the implicit reward, which converges to reward maximization as the iteration approaches infinity. The paper also studies the scenario where the curated self-generative data is mixed with natural data and analyzes its implication on the stability of iterative training. Experiments on synthetic data and CIFAR 10 validate the insights from the theoretical analysis.

Strengths

1. This paper studies an interesting problem of generative models being iteratively trained on self-generated data curated by human users with some implicit reward. This is arguably an accurate description of what happens when new generative models are trained nowadays. 2. The theoretical framework analyzed the convergence and stability of iterative training with and without reference data. 3. Experiments on synthetic data and CIFAR 10 are interesting, especially the one on CIFAR with replay, which shows how bias amplification can be mitigated with natural data.

Weaknesses

1. Human preference can be heterogeneous, so in some cases, Eq. (2) does not hold. It would be interesting to see if the theoretical analysis can be extended to a mixture of rewards. 2. This paper didn't provide experiments on realistic datasets and large models such as LAION and Stable Diffusion (SD). Thus, it's hard to map the theoretical insights to how we should train the next generation of SD. It would be really interesting if the experiments on CIFAR were reproduced on SD to see what realistic biases are picked up, how much replay is needed to mitigate that, etc. 3. Conceptually, retraining with data curation is related to iterative finetuining in language models (e.g., rejection-sampling-based SFT). Would like to see more discussion on that.

Questions

See Weaknesses.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

I think the authors adequately addressed the limitations.

Summary

Self-consuming generative models are known to have collapse or stability problems, and the curation of synthetic data is often ignored. This paper theoretically studies the impact of data curation and proves that it optimizes the expected reward.

Strengths

1. The paper is well-written, and works on synthetic data are extremely important to the field. 2. The connection between retraining with a mixture of curated data and original data and RLHF (Reinforcement Learning with Human Feedback) with KL regularization is novel.

Weaknesses

1. The theoretical results are applied in a simplified setting, focusing on distribution and reward on $x$ only, instead of considering a joint distribution of $x$ given $y$ mimicking text-to-image generation. The theoretical results apply only to learning the distribution directly, without considering finite samples and optimization, and there is only one reward function. In recent works combining RLHF and diffusion models [1], multiple rewards are considered. How would the results hold with a (random) weighted sum of multiple rewards? 2. On the connection with model collapse: The authors motivate the results from previous literature on synthetic data leading to model collapse but do not discuss how and whether curation with a reward model will avoid model collapse. The reviewer thinks that improvements can occur when inconsistencies between text and image and implausible images are discarded through curation. However, other problems persist in model collapse, such as the inherent lack of diversity in synthetic data [2], and existing results show that human interaction with GPTs still produces data lacking diversity [3]. In this case, curation does not address the loss of diversity in synthetic data. The authors should discuss this clearly, especially since the paper is motivated by model collapse and spends considerable time discussing it. 3. Could the authors explain the contribution of Section 2.2.1? Comparing Sections 2.2.1 and 2.2.2, there seems to be no direct comparison to be made. ### Reference [1] Liang, Youwei, et al. "Rich human feedback for text-to-image generation." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024. [2] Guo, Yanzhu, et al. "The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text." *Findings of the Association for Computational Linguistics: NAACL 2024*. 2024. [3] Padmakumar, Vishakh, and He He. "Does Writing with Language Models Reduce Content Diversity?." *The Twelfth International Conference on Learning Representations*.

Questions

See weakness. One related work: [4] propose to use a correction function on the synthetic data to mitigate the model collapse. [4] Gillman, Nate, et al. "Self-Correcting Self-Consuming Loops for Generative Model Training." Forty-first International Conference on Machine Learning.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations and societal impacts are well discussed.

Thanks for the responses.

I appreciate the authors' clarification. I increased my score to "weak accept". About $p_t$. Usually, the probability is normalized to 1. If not, it should add the clarification.