Summary
The authors find that slight corruption in the conditioning of text-to-image diffusion models improves the performance. The authors theoretically analyze this empirical finding in a toy model where the goal is to learn to sample from a Gaussian Mixture and the network is piece-wise linear function. Inspired by their experimental and theoretical findings, the authors propose a technique to improve the performance of text-conditional diffusion models by adding noise to the embedding of the conditioning vector.
Strengths
* The research topic is timely. State-of-the-art diffusion models are trained on billions of image-text pairs. Many of those are potentially noisy.
* The authors propose a very simple modification to the training of text-to-image diffusion models that leads to improved performance.
* The finding that slight corruption in the conditioning helps is interesting.
* The authors validate their findings through an extensive experimental evaluation.
Weaknesses
* I believe that the authors should emphasize in the title of their paper and in the main body that they study corruption in the conditioning. Numerous works study corruption in the images themselves, e.g. see:
1) Ambient Diffusion: Learning Clean Distributions from Corrupted Data
2) Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data
3) GSURE-Based Diffusion Model Training with Corrupted Data
4) Solving inverse problems with score-based generative priors learned from noisy data
In these works, it has been observed that data corruption decreases the performance of diffusion models. I believe the authors should acknowledge these relevant works and clarify that their paper analyzes corruption in the conditioning.
* The theoretical model is not very relevant, at least not very relevant to Section 3 of the paper. In the experiments of Section 3, the authors mislabeled some of the training examples. In the theoretical model, it is assumed that the true class is given to the model since the model is using the parameters that correspond to the true class. In a sense, the theoretical model of Section 3 is more related to the proposed algorithm in Section 5. Also, the existence of multiple centers in Section 4 does not seem relevant, unless I am missing something. This is because the model has separate parameters for each center and the centers are given for each example the model sees. I went over the proof and it looks like this is exploited in the proof as well, to obtain a closed-form solution for the optimal network.
* For the reasons mentioned above, it seems to me that the reason this works is some sort of regularization. I wonder if a similar effect could be obtained by increased dropout in the conditioning or some other form of regularization. Overall, I believe that the reader of the paper doesn't develop much intuition about the origins of the experimental finding.
Questions
Apart from the weaknesses mentioned above, I have one more important question regarding this paper. Could the authors please clarify whether the corruption in Sections 3 and Sections 5 happens once for each data sample before the training or whether a new corruption takes place whenever we encounter the same data point across different epochs? To clarify, I want to understand which one of the two is happening:
1) We take a dataset, we corrupt it once, we train with the corrupted dataset
2) We use the clean dataset and every time we see a sample we corrupt it (by adding a different noise each time to its embedding or by changing each label/text to something else each time we see the same example across epochs).
If 2) is happening, it seems to me that the proposed method is closely related to regularization. I think this needs to be clarified and the effect of doing 1) or 2) should be studied as an ablation.
Limitations
The authors have adequately discussed the limitations of their work.