Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models

Summary

This paper discovers that the effect of classifier-free guidance in image-generating diffusion models varies with noise levels. It proposes a very simple method of performing guidance only at intermediate noise levels, demonstrating that this method can improve sample quality, diversity, and sampling cost both quantitatively and qualitatively. The quantitative effects are shown using EDM2 and DiT models trained on ImageNet, while the qualitative effects are demonstrated using the pre-trained Stable Diffusion XL.

Strengths

**Originality** - The toy examples in Figures 1 and 2 are very intuitive and help in understanding the behavior of guidance at high, middle, and low noise levels. This interpretation of classifier-free guidance is novel. **Significance** - The method is very simple and can be easily applied to other diffusion models. - The FID and FD_{DINOv2} in Table 1 are strong. **Quality** - The ablation studies in Figures 3 and 4 effectively demonstrate the efficacy of the proposed method. - The qualitative results in Figures 5 through 8 clearly show the behavior of guidance at high, middle, and low noise levels in Stable Diffusion XL.

Weaknesses

While the paper shows good intuition and strong results, several questions remain. - Does the **optimal interval vary with the model's performance?** The results in Table 1 imply this, as the optimal intervals for EDM2 and DiT differ, and the optimal guidance weight for DiT is higher. If the optimal interval varies with model performance, the proposed method will require an **exhaustive hyperparameter search for each diffusion model**, which is inconvenient. - Will guidance at high noise levels be detrimental for **conditions denser than the class and text conditions** considered in this paper? Examples include InstructPix2Pix [A] using image conditions, ControlNet [B] using various spatial conditions, and Stable Video Diffusion [C] generating videos from images. Additionally, a text-to-image model based on the T5 encoder (DeepFloyd IF [D]), which provides richer text representation than the CLIP text encoder of Stable Diffusion XL, is also an example. [A] Brooks et al., InstructPix2Pix Learning to Follow Image Editing Instructions, CVPR 2023. [B] Zhang et al., Adding Conditional Control to Text-to-Image Diffusion Models, ICCV 2023. [C] Blattmann et al., Scaling Latent Video Diffusion Models to Large Datasets. [D] DeepFloyd, https://huggingface.co/DeepFloyd

Questions

Looking at Figures 1 and 2, the guidance scale at each interval significantly impacts the ODE trajectory. Do the authors think this tendency would also be observed in **flow matching** generative models [E] (or rectified flows [F]) that learn **straight trajectories**, compared to previous VP and VE diffusion models? [E] Lipman et al., Flow Matching for Generative Modeling, ICLR 2023. [F] Liu et al., Learning to Generate and Transfer Data with Rectified Flow, ICLR 2023.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors mention in the paper checklist that they discussed limitations in section 4, but the limitations are not clearly presented in that section. It is recommended to address the limitations in the conclusion section as well.

Summary

This paper investigates the behavior of classifier-free guidance and proposes an adjustment in its application during sampling. Instead of applying a constant weight for the guidance scale across all sampling steps, the authors suggest that the guidance should be deactivated at high and low noise scales and applied only at intermediate levels. Experimental results with EDM2 and SDXL models indicate that guidance at high noise scales is detrimental, while at lower noise levels, its contribution is negligible. Thus, the guidance interval should be incorporated as part of the sampling hyperparameters for conditional diffusion models.

Strengths

- As classifier-free guidance is an essential part of all diffusion models, the paper takes an important step to reduce the harmful effects of high guidance scales. - The proposed method is straightforward to implement and can be integrated into any model, making it valuable to the broader NeurIPS community. - The authors provide thorough experiments analyzing the impact of applying CFG over an interval. - The effectiveness of the method is also shown on toy datasets. - The paper is well-written and easy to understand.

Weaknesses

The main weakness of the work is a lack of discussion of existing works that address the issues of high-guidance scales. For example, the concept of using a non-constant guidance weight for CFG is also introduced in [1] (referred to as dynamic-CFG), and the current method of the paper is dynamic-CFG with a specific weight schedule (namely $w$ for $\sigma \in [\sigma_{lo}, \sigma_{hi}]$ and 1 otherwise). In addition, similar ideas have been explored in [1, 2, 3] in the form using a linear/cosine schedule for the guidance weight or annealing the conditioning vector to improve the diversity of generations at higher guidance scales. Moreover, half of the conclusion of the paper is already stated in [4] (i.e. guidance does nothing at lower noise levels). Accordingly, the paper would benefit from a more comprehensive related work section. [1] Sadat, S., Buhmann, J., Bradley, D., Hilliges, O. and Weber, R.M., CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling. In The Twelfth International Conference on Learning Representations (2024). [2] Chang H, Zhang H, Barber J, Maschinot AJ, Lezama J, Jiang L, Yang MH, Murphy K, Freeman WT, Rubinstein M, Li Y. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704. 2023 Jan 2. [3] Gao, Shanghua, et al. "Masked diffusion transformer is a strong image synthesizer." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [4] Castillo A, Kohler J, Pérez JC, Pérez JP, Pumarola A, Ghanem B, Arbeláez P, Thabet A. Adaptive guidance: Training-free acceleration of conditional diffusion models. arXiv preprint arXiv:2312.12487. 2023 Dec 19.

Questions

1. Is Figure 3 compiled with a fixed EMA profile for the EDM2 model or is the EMA profile also optimized along with the guidance parameters? 2. Can you explain lines 112-118 in more detail? Does that mean you always use the low and high guidance scales based on the sampling discretization? This part was not very clear to me. 3. Can you also plot Precision/Recall curves for the method to show the impact of interval guidance on quality and diversity in isolation? 4. One major advantage of high classifier-free guidance is image-text alignment. Does the proposed method have a noticeable impact in this area? For example, in Figure 7, it seems that some parts of the prompt are more strongly emphasized by the normal CFG (e.g., the sky is orange). 5. Does the optimal guidance interval change when modifying the sampling algorithm, or are the thresholds mainly model dependent?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

2

Limitations

The authors have addressed limitations and social impact of the work.

Summary

This paper explains applying guidance in a limited interval improves sample and distribution quality in diffusion models, as shown in this title.

Strengths

The author provided intuitive figures, Fig.1 and 2, helped to understand the core of this paper easily. The author provided a lot of experiments and results to support his idea.

Weaknesses

This paper provided an optimization approach in improving diffusion with guidance. The author proposed, doing guidance in the middle stage is good. However, this work does not provide a novel idea or fundamental issue in diffusion with guidance. There are more fundamental problems needed to be solved. Despite the author showed good experiments and good writing, unfortunately, the contribution of this paper is minor.

Questions

No questions

Rating

3

Confidence

5

Soundness

3

Presentation

3

Contribution

1

Limitations

N/A

Response to the rebuttal

I would like to thank the authors for taking the time to respond to my comments. Since my questions have been addressed in the rebuttal and I believe the paper represents a valuable step toward better usage of CFG for the community, I would like to increase my score from 6 to 7.