Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models

Official Response to Reviewer LhNz

We greatly appreciate the reviewer's helpful suggestions, as well as the positive assessment of the influence and quality of our work. Below, we provide detailed responses to the reviewer’s comments. ### **Lower precision at lower guidance scales** The results in Figure 8 were compiled using fixed hyperparameters for momentum and normalization, which suggests that slight adjustments to these hyperparameters may be necessary depending on the guidance scale. At very low scales, the original CFG does not exhibit oversaturation, whereas APG with the fixed parameters from Figure 8 may introduce excessive normalization. However, as the guidance scale increases, the precision and FID of CFG degrade, while APG consistently maintains its quality. This is reflected in APG achieving better FID across a range of guidance scales and a superior minimum FID overall. Thus, APG enables the use of higher guidance scales without encountering the typical issues observed with CFG. Additionally, in our experiments, we found that by reducing normalization and momentum for lower guidance scales, APG can achieve results similar to or better than CFG. ### **Motivation behind momentum** Our motivation for incorporating momentum is to filter out any common signals present across different CFG updates at various time steps. By taking a weighted running average of past updates and removing the overlapping component from the current update, we effectively retain only the new information at each step. We believe this approach helps limit the drift typically seen in CFG over multiple sampling steps. Table 2 shows that removing momentum results in a higher FID, demonstrating that momentum contributes to improved quality. We would be happy to expand on the intuition or add additional experiments on momentum in the final version.

Official Response to Reviewer mQei

We wish to thank the reviewer for the helpful comments and for finding our work novel with detailed evaluations, good presentation, and significant contribution. Please find our answers to the comments below. ### **HPSv2 and ImageReward** We will examine the suggested metrics and will gladly consider them for the final version of the paper. ### **Note on lower guidance scales** Lower guidance scales generally avoid oversaturation, but often compromise output quality and alignment with the input. Throughout our experiments, we observed that APG performs robustly across a wide range of guidance scales, as illustrated in Figure 8. Additionally, we demonstrated that APG enables the use of higher guidance scales (such as $w=15$ for Stable Diffusion XL) that are typically infeasible with CFG, achieving high-quality generations that align well with the input condition without encountering common CFG issues. Thus, we do not believe that the benefits of APG are limited to certain guidance scales shown in the paper. ### **Effect of the parallel component** As shown in Table 2, results with and without the parallel component yield similar FID scores (computed over 10,000 generated samples); however, the model with the parallel component exhibits higher saturation. This suggests that the orthogonal component primarily drives FID, while the parallel component mainly influences saturation. For theoretical insight into this effect, we refer the reviewer to our “gain” argument in Section 4 (beginning around line 186 in the original manuscript). ### **Example in figure 10** This figure primarily illustrates improvements in text quality, where "hello world" is misspelled in the CFG figure. Since the base image for CFG contains artifacts (including two extra paws around the sign), our method successfully resolves one but not all of these artifacts in this case.

Official Response to Reviewer xLYc

We thank the reviewer for the positive reception of our paper and for recognizing its strengths. We also thank the reviewer for the opportunity to clarify several points. ### **Analysis on rescaling the guidance weight** We believe that the impact of APG extends beyond merely adjusting the guidance weight. A key observation is that modifying the parallel component causes a deviation from the CFG update rule. If we express CFG as $D(x, t, y) + w(D(x, t, y) - D(x, t))$, then we can show that applying the projection alters the update to $w’ D(x, t, y) + w (D(x, t, y) - D(x, t))$ for some constant $w’ \neq 1$. This transformation cannot be achieved by simply modifying the guidance scale $w$ alone. Therefore, we argue that APG achieves more than just rescaling the CFG weight. Furthermore, in Appendix C.1, we present experiments combining APG with other methods that effectively adjust the guidance weight, specifically CADS and Interval Guidance (IG). The results indicate that combining APG with these methods yields the best FID compared to each method in isolation. This suggests that APG’s benefits are, to some extent, orthogonal to CFG weight adjustments. ### **Model for figure 8** Figure 8 was generated using the EDM2 model for class-conditional ImageNet generation. ### **The trend of the rescale weight** We observed that the norm of the updates increases during the initial steps of inference and then gradually approaches zero as denoising progresses. However, identifying a consistent pattern for this behavior is challenging, as it dynamically varies depending on the input prompt and random seed. We are happy to provide further analysis if this does not fully address the reviewer's comment. ### **Illustrations of the geometry** We would be glad to provide geometric diagrams illustrating the visual interpretation of each step in APG to help guide the reader’s intuition for our approach. If this does not fully address the reviewer's point, we welcome further discussion.

Official Response to Reviewer xLYc

Table 1 aims to illustrate the differences between APG and CFG at higher guidance scales, with results reported using $w = 4$. This choice is based on the observation that while lower guidance scales yield better FID values, higher guidance scales produce better visual quality as judged by human inspection. However, at these higher guidance scales (e.g., $w = 4$), standard CFG starts exhibiting saturation and artifacts. APG effectively addresses these issues, resulting in improved metrics. In Figure 8, we separately show that the superior metrics achieved by APG are not restricted to a specific guidance weight, i.e., APG outperforms CFG across a range of guidance values. For EDM2 [1], the paper does not recommend a specific scale but uses $w = 1.2 -1.4$ for their final models. We have observed that above this level, the saturation and artifacts begin to become more pronounced with the standard CFG. APG is designed to easily allow scales above this level (e.g., $w = 4$) while avoiding these issues. [1] Karras, Tero, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. "Analyzing and improving the training dynamics of diffusion models." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184. 2024.