Official Response to Reviewer fxon
### **Motivating the method**
We appreciate the reviewer’s suggestion to better motivate the method ahead of its introduction and to move some of the theoretical discussion from the appendix to the main paper. To the best of our knowledge, accepted papers at ICLR do not get an extra page allowance, so we have to assume for the moment that we are working with limited space. However, we agree with the reviewer that the placement of the method’s motivation is important, so we have added a paragraph above Section 3.1 in our working draft to provide readers a better intuition behind the rationale of the solution before introducing the solution itself. Please find the corresponding paragraph below:
> We conjecture that the low-diversity issue is due to a highly peaked conditional distribution, particularly at higher guidance scales, which leads the majority of samples toward certain modes during inference. One way to address this issue is by smoothing the conditional distribution with decreasing Gaussian noise, which breaks and then gradually restores the statistical dependency on the conditioning signal during inference. Next, we introduce a novel sampling technique that integrates this intuition into the inference process. The superiority of this method over the above approaches is depicted in Figure 2.c.
### **Where to inject noise**
Many conditional diffusion models have an embedding layer that provides the actual continuous conditioning signal to the input of the model. This can effectively be a look-up table in the class-conditional case or a continuous functional mapping, such as the CLIP network, for text-to-image generation. This continuous embedding space is ideal for injecting Gaussian noise, and for a more unified approach across various tasks, we injected noise into each condition’s embedding. In some settings, however, such as the pose-to-image task, embedding layers aren’t normally used, as the conditioning information is directly concatenated with the input to the diffusion model [1]. We decided to experiment with the noise injection directly on top of the pose image, since it is this representation that constitutes the conditioning signal in this case. Based on the reviewer’s comment, we also tested a different pose-to-image model that does use an embedding layer and found that injecting noise to the pose embedding also successfully diversifies the model outputs. We saw a similar phenomenon for the face-ID setting, where adding noise either before or after the embedding layer in the diffusion model performed well. The takeaway seems to be that the technique is highly flexible and robust to this design choice, with the key requirement being initially breaking and then progressively restoring the model’s dependence on the conditioning signal, which can be achieved at a number of places in the model.
### **Behavior of CADS VS NFEs**
This is an excellent question. We have added a section to the appendix of the paper that explores the effectiveness of CADS with respect to different numbers of sampling steps (NFEs). The behavior of CADS vs NFEs is similar to the base sampler, and a consistent gap exists between sampling with and without condition annealing across different NFEs. Additionally, please note that as DDPM does not perform well when using low NFEs, we switched to DPM++ [2] and DDIM [3] to perform this comparison. Please find the results of this experiment in the tables below.
| | DPM++ w/ CADS | DPM++ w/ CADS | DPM++ | DPM++ |
| --- | :---: | :---: | :---: | :---: |
| **NFEs** | **FID** | **Recall** | **FID** | **Recall** |
| 25 | 7.73 | 0.67 | 17.86 | 0.37 |
| 50 | 8.56 | 0.64 | 18.64 | 0.36 |
| 75 | 8.95 | 0.63 | 18.81 | 0.36 |
| 100 | 9.63 | 0.61 | 18.90 | 0.36 |
| | DDIM w/ CADS | DDIM w/ CADS | DDIM | DDIM |
| --- | :---: | :---: | :---: | :---: |
| **NFEs** | **FID** | **Recall** | **FID** | **Recall** |
| 25 | 8.45 | 0.64 | 18.67 | 0.36 |
| 50 | 9.80 | 0.59 | 18.91 | 0.36 |
| 75 | 10.08 | 0.57 | 18.92 | 0.35 |
| 100 | 10.16 | 0.58 | 18.96 | 0.35 |
---
**References**
[1] Saharia, Chitwan, et al. "Image super-resolution via iterative refinement (2021)." *arXiv preprint arXiv:2104.07636* (2021).
[2] Lu, Cheng, et al. "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." *arXiv preprint arXiv:2211.01095* (2022).
[3] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." *International Conference on Learning Representations*. 2021.