Adding additional control to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-free loss-based guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer a free lunch in diffusion guidance. In this paper, we aim to develop a deeper understanding of training-free guidance, as well as overcome its limitations. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theoretically demonstrate that training-free guidance is more susceptible to adversarial gradients and exhibits slower convergence rates compared to classifier guidance. We then introduce a collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy of these techniques.
Paper
Similar papers
Peer review
Summary
This paper explores training-free loss-based diffusion guidance, providing an overview of how training-free guidance functions from an optimization perspective. It analyzes the challenges of training-free guidance, particularly regarding adversarial gradient issues and slower convergence rates. Additionally, the paper proposes two improvements: random augmentation and Polyak step size. These enhancements are validated through experiments across various applications.
Strengths
- The theoretical analysis offers valuable insights into understanding training-free guidance. - The proposed tricks, random augmentation and Polyak step size, demonstrate effectiveness.
Weaknesses
Although the paper provides a thorough theoretical analysis of why training-free guidance works and doesn’t work, the empirical evidence is insufficient to fully support the claims. The propositions are based on the assumption of Lipschitz continuity, which may not hold for complex deep neural networks, therefore, a more comprehensive quantitative analysis is necessary. Here are my detailed comments: - The experiment presented in Figure 1 is somewhat confusing. The authors aim to convey three points: (1) the loss curve will oscillate initially and decrease eventually; (2) the guidance weight needs careful selection; (3) adversarial gradients may cause incorrect generated images despite low loss. The first point requires more extensive experiments rather than just two cases. The second point is not clearly inferred from the figure. - The two limitations of training-free guidance, adversarial gradient, and slower convergence rates, lack quantitative experimental evidence. Regarding the proposed method, while benchmarking many baselines on various tasks is appreciated, several issues remain unaddressed: - The difference between UG and FreeDoM in this paper appears to be the selection of diffusion steps for the time-travel trick. However, UG also uses a “backward universal guidance” technique as mentioned in section 3.2 of the original paper [1]. - The authors did not choose classifier-based tasks, which is curious given that the paper analyzes the inferiority of training-free guidance compared to training-based guidance through this kind of task in Sec 3. Comparative experiments showing the gap between these approaches are essential. Including classifier-guided generation tasks would be beneficial to assess the significance of the two tricks’ improvements, and training-based baselines can be included in this task. - The selection of hyper-parameters is concerning. With numerous hyper-parameters involved, how are the optimal values (e.g., step size, LGD-MC Monte Carlo samples, diffusion step range for time-travel) chosen? How is it ensured that baselines are implemented optimally? - The random augmentation trick may have limitations: - While image augmentation techniques are mature, effective augmentation tricks for other domains (e.g., text, AI4Science, offline RL) may not be available, limiting the trick’s application. - Despite the claim that random augmentation does not introduce significant computational overhead, this may not hold true when using a large guidance network, as would be the case with a large, generalist foundation model for scoring samples. [1] UNIVERSAL GUIDANCE FOR DIFFUSION MODELS. https://openreview.net/pdf?id=pzpWBbnwiJ
Questions
- What is the rationale behind choosing the Polyak step size? It appears to be a post-hoc decision without strong intuition. - Since random augmentation is omitted in MDM, is the only difference between “Ours” and “FreeDoM” the Polyak step size? The loss values in Table 3 show a significant difference between FreeDoM and Ours, and clarification on this point is needed. - What's the value of $\eta$ set for DDIM?
Rating
5
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
Yes.
We greatly appreciate your insightful suggestions and comments, which have significantly contributed to the refinement of our paper. We are also thankful for your acknowledgment of our rebuttal efforts. **Comment 1:** Regarding Table 3, it is designed to evaluate the adversarial gradient issue in training-free guidance. To assess the presence of adversarial gradients, we utilized an adversarially robust ResNet-50 [33], which is trained to resist adversarial gradients. Since ResNet-50 serves as our guidance network, a large loss gap between the standard ResNet-50 and the robust variant indicates a severe adversarial gradient issue. In Table 3, the values represent the loss value. The columns denote the guidance network type, while the rows indicate the loss tested on either ResNet-50 or Robust ResNet-50. The values reported are the average over 1000 images, providing a quantitative evaluation. When using ResNet-50 for guidance, its loss on ResNet-50 is as low as that of real images from the same class (value 5.91). However, the loss for Robust ResNet-50 is significantly higher (value 6.17), suggesting susceptibility to adversarial gradients. By employing ResNet-50 with random augmentation for guidance, we observe a marked reduction in this gap (from 5.91 to 5.98), underscoring the effectiveness of our proposed method. Table 4 aims to compare the convergence of training-free guidance with training-based guidance, specifically PPAP [D]. Given that classifier guidance presents a simple scenario and all methods converge relatively fast, it tends to obscure the observations. Consequently, we have directed our attention towards the more complex task of segmentation map guidance in Table 1. The columns in Table 4 list different methods, while the rows indicate the sampling steps. The values represent the "distance" mentioned in Table 1 and are averaged over 1000 images under the same conditions as those in Table 1. Our observations reveal a significant discrepancy in the objective values between the training-free FreeDoM and the training-based PPAP [D], particularly at a lower number of steps (20 or 50 steps), which indicates slower convergence. However, when we incorporate the Polyak step size into FreeDoM, this convergence issue is substantially mitigated. **Comment 2:** As for the second concern, we are thankful for your acknowledgment of our efforts in the rebuttal. We have also conducted comparisons with training-based methods, as illustrated by PPAP in Table 1 of the PDF provided in the global rebuttal. PPAP extends classifier guidance to accommodate a broader range of conditions by fine-tuning the pretrained model on clean images to create different experts, each responsible for a specific range of noise level. The gradients from the corresponding experts are then used for guidance at corresponding noise levels. To optimize performance of PPAP, we have set the number of experts to $10$, which is the maximum reported in [D]. As indicated in Table 1, our proposed method shows performance on par with the training-based PPAP. With respect to the reviewer's request for additional experiments under class conditions, we are currently conducting these experiments. However, due to the high number of required samples and the author-reviewer discussion coming to an end, it is possible that they cannot be completed before the reviewer-author discussion period closes. We assure you that the results of these experiments will be included in the revised manuscript. **Comment 3:** Regarding the third question, the effectiveness of random augmentation is supported by Proposition 4.1. This proposition does not rely on the assumption of Lipschitz continuity or any specific distribution for the augmentation, suggesting its broad applicability across various modalities. Consequently, the theoretical foundation guarantees the effectiveness of random augmentation in AI4Science or RL without imposing any strong assumptions. As we approach the end of the author-reviewer discussion and considering AI4Science or RL represents a novel application setting for our paper, it is regrettably unlikely that additional experiments can be completed prior to the submission deadline. Nonetheless, we are committed to including these experiments in the revised manuscript and we appreciate the reviewer's understanding in this matter.
Thanks for the timely response. Considering of the theoretical contribution of this paper which helps to understand training-free loss-based diffusion guidance better, I raise the score to a 5.
Summary
This paper explores the theoretical aspects of training-free loss-based guidance mechanisms in diffusion models and improves them from theoretical findings. The first results are in explaining why the guidance strength depended on $\sqrt{\alpha_t}$ worked well in FREEDOM. Then, the authors explore the adversarial gradients, which result in a lower loss but produce generations with the incorrect condition. Specifically, they demonstrated that a time-dependently trained network with Gaussian noises is smoother compared to a general network. This smoothness can cause a slowdown of guidance. From these theoretical results, they propose to utilize random augmentation to increase the smoothness of the off-the-shelf network and improve the convergence speed with Polyak step size, which accelerates convergence by adaptive gradient step size. Experimental results show that their method can outperform loss-based guidance methods including Universal Guidance, Loss-Guided Diffusion, FreeDom, and MPGD-Z.
Strengths
- This paper explores the theoretical aspects of training-free loss-based guidance. The authors explain well-known practices in previous works with theoretical aspects, such as FreeDom. - Various benchmark datasets are utilized for evaluating their method.
Weaknesses
- My first major concern is missed baselines and related works. This work aims to improve off-the-shelf models’ guidance with the training-free based method, and [A] deals with the same configuration. [A] elucidates the design space of off-the-shelf guidance, and smoothed classifier, joint classifier guidance, and guidance schedules are explored in their work. I think that the smoothed classifier concept in [A] is very similar to random augmentation for the smoother network in this work, and similar guidance schedule schemes are explored in [A] and this work. Therefore, [A] should be discussed in this work and be compared in the experimental section. Also, I found [C] and [D]. - In the other line of work, [B] proposes the utilization of off-the-shelf models by finetuning that model to be the time-dependent network. Although [B] can be categorized into the training-required method, it would be better to discuss this kind of training-required method and compare it in the experimental section. Showing the performance gaps between training-free and training-needed methods is needed to validate the effectiveness of the proposed method. - [A] and [B] choose the imagenet classifier guidance with imagenet trained diffusion models and various diffusion models use imagenet class conditional generation as the core benchmark. However, this work does not incorporate this major benchmark in their benchmark, so it can make reader hard to recognize their absolute performance. - There are no details about the Polyak Step size, and if Algorithm 2 is all of the details of the Polyak Step Size, there have been many works to utilize normalized gradients and I felt that this is not novel.
Questions
## Typo - Line 875 in Appendix: $||f(x_2)-f(x_2)||$ -> $||f(x_2)-f(x_1)||$ - Equations are referred to as (number). I am very confused to understand whether (number) is an equation or a bullet point. How about using Eq. (number)? ## References - [A] ELUCIDATING THE DESIGN SPACE OF CLASSIFIER GUIDED DIFFUSION GENERATION, ICLR 2024 - [B] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023. - [C] ADJOINTDPM: ADJOINT SENSITIVITY METHOD FOR GRADIENT BACKPROPAGATION OF DIFFUSION PROBABILISTIC MODELS, ICLR 2024. - [D] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method, Arxiv 2023.
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
The higher number of NFEs required in training-free guidance and adversarial guidance are mentioned in the limitation section.
Summary
This paper examines the mechanisms and limitations of training-free guidance for diffusion models and develops a collection of techniques to overcome the limitations accompanied by both theoretical and empirical results.
Strengths
1. This paper performs the theoretical analysis on the training-free diffusion model from the optimization perspective, which presents the problems and drawbacks in training-free guidance. 2. This paper also proposes corresponding solutions for the two issues in training-free guidance: random augmentation for the adversarial gradient, Polyak step size for improving the convergence.
Weaknesses
1. The paper misses the recent literature in training-free diffusion model, such as RED-diff [1] and DSG [2]. 2. Although the theoretical analysis in this paper illustrates the problems in training-free guidance, the proposed solutions are not particularly novel. Similar ideas have been applied in previous works such as LGD-MC and DSG. [1] Mardani M, Song J, Kautz J, et al. A variational perspective on solving inverse problems with diffusion models[J]. arXiv preprint arXiv:2305.04391, 2023. [2] Yang L, Ding S, Cai Y, et al. Guidance with spherical gaussian constraint for conditional diffusion[J]. arXiv preprint arXiv:2402.03201, 2024.
Questions
1. How is the performance of MPGD and LGD-MC modified with the random augmentation and Polyak step size? 2. How is the proposed method compared with DPS, DSG and RED-diff?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
The authors have clearly presented the limitations in the paper.
Summary
The paper studies training-free loss-based diffusion guidance. in comparison to classifier-based guidance. First, the paper examines several drawbacks of loss-based guidance, including (1) while successful minimization of the loss can be achieved, it does not guarantee successful guidance, (2) loss-based gradients can be misaligned with the desired guidance, in contrast to the time-dependent classifier-based gradients with improved smoothness properties, (3) the superior smoothness of classifier-based guidance also means that loss-based guidance takes longer to converge, incurring more function estimations. In response to those drawbacks, the authors propose two techniques to improve loss-based guidance: (1) random augmentations to improve its smoothness, and (2) Polyak step size to further improve robustness to misalignment between initialization and the specified condition for generation. Three applications are considered for evaluation, where the proposed technique is implemented on top of FreeDoM.
Strengths
- Offers theoretical results explaining the mechanisms of loss-based guidance - Demonstrates the drawbacks of loss-based guidance, in terms of potential misalignments between its gradients and desired guidance directions, and how the smoothness properties of the employed guidance impact the rate of convergence. - Proposes two techniques to ameliorate loss-based guidance, as demonstrated on a number of applications.
Weaknesses
Nothing major, though the presentation of Section 3 leaves something to be desired. I'd need to see it after some polishing before I can be satisfied that I fully understand what's going on. See the comments below.
Questions
Technical: ======== - Section 3.2 and elsewhere: - Recommend to use terms along the lines of "misaligned gradients" rather than "adversarial gradient." - I can see it helps to leverage earlier insights from adversarial robustness, e.g., regarding the role of Lipschitz constants, but I worry that there's no benefit to confusing the behavior in question with adversarial robustness considerations. If anything, those gradients were not chosen to maximize deviations w.r.t. any particular objective, but just happen to be misaligned. - This suggestion seems to match the narrative on L166-168. - Not sure if *adversarial* on L258 is related to this discussion, but just pointing it out. - I'm not sure what's exactly meant by "non-Lipschitz" functions - A function that's, say, 200-Lipschitz or only 2-Lipschitz is still not 1-Lipschitz. - It seems that Proposition 3.2 only needs the upper bound on $\ell$, but need not make any statements about its smoothness. - Similarly, L149 can simply state that adding Guassian noise improves smoothness, which is pretty intuitive. - It would help to double-check if Proposition 3.2 follows from known standard results, e.g., randomized smoothing which seems to be what $\hat{\ell}$ is about. See Assumption A in - Duchi, John C., Peter L. Bartlett, and Martin J. Wainwright. "Randomized smoothing for stochastic optimization." SIAM Journal on Optimization 22, no. 2 (2012): 674-701. - Same goes for Proposition 4.1. - is it true that Proposition 3.2 is a special case of Proposition 4.1? - L253: Is there a way to quantify this mismatch between the condition and the input? - I can't readily place where time travel factors into the presentation and/or the experiments. Is it only mentioned in passing? Is the related analysis in the appendix only included as a bonus? Presentation: ========== Section 1: - L43: Please cite Appendix E in [25] specifically, since the paper may otherwise seem to be concerned with a different problem altogether, i.e., offline reinforcement learning. Section 2: - Suggest to flatten the section by using bold headers as used later. (Usually looks better than an absent opening paragraph.) - Section 2.1 could use some citations, also to clarify which formulation or model is adopted in this paper, and whether the contributions apply to other models as well. - L84: Please fix `fastRCNN` and include a citation. Section 3: - Proposition 3.1 - Recommend to at least provide an intuitive definition of PL. Otherwise, I'd recommend to present an *(informal)* version of the statement that's optimized for readability, deferring the formal version in its entirety to the appendix. - Figure 1: would help to explain that a wrong image was obtained in (b). - L132: Seems like a good place to start a new paragraph at "Furthermore" - L172: Recommend to cite a textbook. - L179: Promposition 3.3 - Is it necessary to reuse the symbol $g$? - $g(x_t, t)$ is L-Lipschitz w.r.t. which parameter? - It is not clear how the discretization error is defined here. Some context would help. - Is it necessary to use the symbol $h_{\text{max}}$? It seems to have nothing to do with the function $h(x_t, t)$. - Is $O(h_{\text{max}} + Lh_{\text{max}})$ the same as $O((L+1)h_{\text{max}})$? - After a second reading, it's still not clear to me how the discretization error relates to the rate of convergence. Section 4: - Algorithm 1 & 2: Suggest to highlight Tweedie's formula - L189: Please include a citation for the Polyak step size. - L191: Is it really the case that the analysis presented "completes the picture" for improving training-free guidance? If so, it would help to describe what it takes to complete the picture, then explain how the analysis answers to that. It would also help to highlight any remaining gaps or extensions for future work. - L206: Recommend to give names to those integrations defining the Lipschitz bounds. Nitpicking: ======== - Recommend to reduce redundancy: - L1: Adding additional control -> Adding / Controlling / Guiding - L19: the generation / the synthesis, as well as the creation -> generation - L32: zero-shot generalization to novel conditions -> zero-shot generalization / generalization to novel conditions - L38: employ pre-trained networks, designed for clean images [..] checkpoints for these networks pretrained on clean images are -> employ networks pretrained on clean images [..] pretrained networks are - Recommend to improve consistency in terminology and notation: - pre-trained and pretrained - clarifies the mystery and resolves the mystery - Suggest to favor technical terms: - "mystery" of guidance weights -> role of guidance weights, principles for choosing guidance weights, intricate interplay between guidance weights and the guidance function and time. - L190-191: "trick" -> approach / algorithm / heuristic - L99: Despite its intuitive -> Despite being intuitive - L100: considers -> consider - L100: with Gaussian dist. -> with a Gaussian dist. - L102: for one-dimension dist. -> for a one-dimensional dist. - L130: referred as -> referred to as - L195: incorporating them into -> passing them to - L212: emerges -> emerge - L213: efficiency -> efficacy?
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Although the authors seem to have aimed to complete the picture for training-free loss-based guidance, covering both its drawbacks and potential improvement strategies, this picture is still not clear to me. For example, it's not clear what's best possible under reasonable conditions on the loss employed for guidance, e.g., how it compares to the (time-dependent) classifier models that can otherwise be employed. I suspect this has something to do with the limitations of time travel. It would be nice to attempt to show some lower-bounds on the errors incurred through time travel.
We apologize for the oversight in not including references for Weakness 1 in our previous submission. The references for the rebuttal of Weakness 1 are as follows: [A] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps [B] Accelerating convergence of score-based diffusion models, provably
Thank you to the authors for the detailed responses. I have some follow-up comments: 1. I apologize for not fully understanding the meaning of Table 3 and Table 4 in the uploaded PDF. Could you please clarify what the values in these tables represent? I would appreciate a more detailed explanation of the settings for each table. 2. Regarding the classifier-based tasks, I appreciate that the authors chose to evaluate their method on tasks where FreeDoM and LGD-MC have shown strong performance. However, I still believe that using tasks involving classifier guidance would be more appropriate. This would allow for more direct comparisons with classifier-based methods (or even classifier-free guidance), making both quantitative comparisons and qualitative analyses easier. 3. Thank you for highlighting the presence of augmentation methods in molecule and RL domains. My concern is that it's difficult to justify the effectiveness of all augmentation methods in training-free, loss-based diffusion guidance, as the efficacy of augmentation can vary significantly across different domains.
Thanks for your responses and the efforts on the additioanl experiments. Although the theoretical contribution of this work should be acknowledged, the novelty of this paper is still limited in the reviewer's opinion. In that case, I am sticking to my score.
Thank you for your comprehensive feedback and insightful review. Your observations have been extremely beneficial in refining our manuscript. We are deeply appreciative of your recognition of our theoretical contributions and our responses to the critiques. As further detailed in the second point of our response, the methodologies we introduce are distinct from LGD-MC and DSG in their motivations and methodology, presenting complementing strategies. The experiments (Table 1 of the PDF in the global response) further indicate that the proposed methods can augment the performance of these existing works. Once again, we express our sincere thanks for your valuable comments and prompt reply.
Thank you for your thorough review and insightful suggestions. Your comments have been invaluable in refining our paper. We appreciate your positive reception of our rebuttal and are glad that our response has addressed your concerns. We are committed to carefully revising the manuscript according to your feedback.
Follow up
Thanks for responding to my comments. I recommend to move Proposition 4.1 to the preliminaries section under a new subsection, e.g. can be titled Randomized Smoothing, with adequate citations. Proposition 3.2 should follow immediately by taking $p$ to be $\mathcal{N}(0, I)$, and can be stated as a corollary.
We are immensely grateful for the time and attention you have invested in reviewing our manuscript. Your detailed comments and constructive feedback have not only highlighted areas that needed improvement but have also deepened our understanding. We want to assure you that we take your comments seriously and are currently in the process of revising our manuscript accordingly.
Decision
Accept (poster)