Guiding a Diffusion Model with a Bad Version of Itself

Summary

The paper introduces a novel conditioning method as an alternative to Classifier-Free Guidance (CFG), which allows for better control over image quality generation without compromising data diversity. This approach involves guiding the diffusion model generation with a lower-quality model (either less trained or with reduced capacity) instead of an unconditional model. The authors compare CFG with their guidance approach on ImageNet.

Strengths

* The paper is well-written and clear. * There is a thorough analysis of CFG behavior and its limitations. The toy example comparing CFG with the authors' method is particularly insightful.

Weaknesses

* The results on images do not clearly demonstrate the distribution coverage shown in the toy example. It appears that the low-quality model provides low-frequency guidance during generation, while the high-quality model focuses more on details. This approach results in sacrificing diversity for quality, contrary to what is depicted in Fig. 1. Additionally, Table 1 should include the Inception score in addition to the FID. Peculiar also the choice of omegas in Table 1; I wonder how the authors choose these values. A fair comparison between CFG and their method would be sufficient with omega={1,1.5,2,2.5,...} * In CFG, only one model needs to be trained to achieve conditioning. In this new approach, achieving greater diversity requires training two distinct models. In line 174, the authors mention: "such as low capacity and/or under-training." I would expect that the low-capacity model could function in this auto-guiding setup, but an under-trained approach might face significant generalization issues. The key question is: "How can we determine the training data size needed for the lower quality version to enhance a well-trained model?" Considering model degradation, model quantization could be a viable solution, potentially eliminating the need for an extra model. Demonstrating how an f16 model could enhance the diversity of an f32 model using this auto-guiding method would be an interesting experiment.

Questions

* In the CFG paper, a complete table of FID and IS values at various omega settings is provided (omega={1,1.5,2,2.5,3,3.5,4...}). I would like to see a similar comparison between the auto-guidance approach and CFG. * I am interested in seeing the use of a quantized model as the low-quality model for auto-guidance. * Additional examples of generated images or access to the code would be beneficial for comparing the auto-guidance method with CFG.

Rating

7

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The method requires training two diffusion models as opposed to just one with CFG. This difference is critical when scaling up the training of foundation models.

Summary

This paper presents a novel perspective on classifier-free guidance (CFG). It improves the generation quality by directing the generative model towards high-probability regions. The authors identify that this improvement stems from the quality difference between the conditional and unconditional components in CFG. Building on this insight, the paper introduces autoguidance, a new sampling algorithm that utilizes both the diffusion model and a bad version of it. Experimental results show the superiority of this method.

Strengths

1. The paper employs an intuitive toy model to support its empirical findings. The specific long-tail, tree-shaped mixture of Gaussian distributions used in this model could be beneficial for future research on generative models. 2. The empirical observations and proposed methods are coherent. The paper uncovers a new mechanism within CFG and enhances it through the proposed method. 3. The proposed method is both simple and powerful, significantly improving the SOTA generation quality on the ImageNet dataset. This new approach has the potential to inspire further related research.

Weaknesses

1. For the experiment, the paper lacks some quantitative comparison for their proposed method of text-to-image diffusion model.

Questions

1. I am still not quite clear about the necessity of the similar degradation of $p_1(x|c;\sigma)$ and $p_0(x|c;\sigma)$ empirically. (I am convinced by your synthetic experiment). Even if $p_1(x|c;\sigma)$ and $p_0(x|c;\sigma)$ suffer from different kinds of degradation, empirically the ratio of them is still possible to pull the sampling trajectory towards the high-likelihood region. If possible, could you give some toy mathematical examples about this? 2. For Figure 1(e), the author applies autoguidance to the toy model. Could you also visualize the $p_0$, $p_1$ and $p_1/p_0$ in the autoguidance setting? This will help us better visualize what the similar degradation looks like.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The paper covers limitations and societal impact.

Summary

The paper proposes autoguidance, a new method that simulates the behavior of classifier-free guidance by using a worse version of the model itself instead of an unconditional module. The authors demonstrate that inconsistencies between the predictions from the conditional and unconditional parts of CFG are responsible for some of its shortcomings such as lower variation in generated results. By using a worse version of the same conditional model, the authors show that such inconsistencies will be reduced, and sampling trajectories will converge toward samples that are closer in distribution to the data. Therefore, the paper concludes that compared to CFG, autoguidance improves quality without sacrificing diversity.

Strengths

- The paper studies an important topic. Since CFG is widely used in current diffusion models, overcoming its shortcomings will have a noticeable impact in the future. - The method is well-motivated through controlled experiments that shed light on the behavior of CFG and how autoguidance improves it in those aspects. - The experiments are well-organized and clearly demonstrate the impact of different components in autoguidance. - The paper is well-written and enjoyable to read.

Weaknesses

- More visual examples are needed to show how the diversity of generations changes as the guidance scale increases. In the final version, please include a batch of examples with a fixed condition and compare the sampling with CFG and autoguidance to better demonstrate the disentanglement between image quality and diversity in autoguidance. - The method is not readily applicable to pretrained diffusion models such as Stable Diffusion. This might limit the current use cases of autoguidance. However, this issue will likely not persist in the long run, as we may see the release of pretrained models compatible with autoguidance. Therefore, this weakness does not affect the long-term impact of the paper.

Questions

1. Can you provide precision/recall (PR) curves for your method vs. CFG? While FID considers both aspects, the PR curve shows the impact of guidance on quality and diversity more directly. 2. In addition to improvement in quality, CFG is also heavily used to improve text-image alignment. Can you provide a more detailed experiment on how autoguidance affects this aspect? From the images, it seems that some degree of CFG is still needed for optimal text-image alignment. 3. Can you provide some intuition on how to choose the guiding model besides grid-search? It seems very costly to train multiple different models just to see which one works better as the guidance model, especially if the same method is applied to text-to-image models trained on massive datasets. 4. How does the method compare to algorithms designed for increasing the diversity of CFG, e.g., [1, 2, 3]? In other words, how much of the improvement comes from increased diversity and how much comes from the fact that autoguidance provides better image quality overall (due to fewer inconsistencies between the updates)? It would be great if the authors could provide a section studying this in the final version. Currently, it is only mentioned that autoguidance is beneficial at all noise scales compared to CFG, but I believe a more detailed comparison with visual examples would strengthen the paper. [1] Kynkäänniemi T, Aittala M, Karras T, Laine S, Aila T, Lehtinen J. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724. 2024 Apr 11. [2] Wang X, Dufour N, Andreou N, Cani MP, Abrevaya VF, Picard D, Kalogeiton V. Analysis of Classifier-Free Guidance Weight Schedulers. arXiv preprint arXiv:2404.13040. 2024 Apr 19. [3] Sadat S, Buhmann J, Bradley D, Hilliges O, Weber RM. CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling. InThe Twelfth International Conference on Learning Representations.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The submission has properly discussed the limitations and social impacts.

Response to the rebuttal

I thank the authors for providing detailed answers to my questions. I believe this is a strong paper that would benefit several applications of diffusion models. Therefore, I would like to keep my score.

Congrats and Complementary Work on CFG Solutions

Congratulations on your best paper nomination! This is an excellent and innovative contribution that tackles the CFG issue by aligning both models to the same task. It’s great to see growing attention in this direction. The authors note that $p_w(x|c; σ)$ does not form a valid heat diffusion of $p_w(x|c; 0)$, which can cause issues like distorted trajectories and color over-saturation. They proposed auto-guidance method elegantly addresses these problems at high noise levels. While auto-guidance is a strong approach, there are additional methods to address these challenges beyond the noise level-dependent guidance cited. Another approach—‘Characteristic Guidance’ [1]—directly modifies the CFG formulation to fix such irregularities, from mode dropping to over-saturation. We kindly suggest acknowledging this complementary work. Doing so will help place auto-guidance more accurately within the broader literature and highlight the full range of solutions to CFG’s known drawbacks. [1] Zheng, C. & Lan, Y.. (2024). Characteristic Guidance: Non-linear Correction for Diffusion Model at Large Guidance Scale. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:61386-61412 Available from https://proceedings.mlr.press/v235/zheng24f.html.