Response to Reviewer yCsT's Comments
We thank the reviewer for their prompt response to us! We would like to confirm a few factual understandings so there is no miscommunication over the review process.
### Questionable Results
**Could the reviewer confirm that you have checked (1) our updated results in the General response and (2) the original CUT paper** (https://arxiv.org/abs/2007.15651)? Our presented images for CUT and CycleGAN in Figure 2 are **exactly the original copy** of the original paper (Horse->Zebra: Fig 3 & Summer->Winter: Fig 11). While the reviewer commented that “they are too bad compared with the results from their original papers”, we did use the same copy.
### Novelty
**Could the reviewer confirm that they have read our discussion in General Response 2 (expanded below)?** If so, we would love to receive clear pointers to additional research that we missed in the discussion. We hope to know if the reviewer agrees that CycleNet presents the first effort that explicitly enforces cycle consistency constraints in image editing using pre-trained diffusion models.
**Q: Has cycle-consistency constraint been explicitly integrated into diffusion models?**
Not yet, to the best of our knowledge.
Reviewer yCsT mentioned that:
> “The concept of **combing cycle-consistency constraint** and **diffusion model** for image-to-image translation has been widely explored by other works”
We believe there is a factual misunderstanding. To the best of our knowledge, this is the first effort that **explicitly integrates cycle consistency constraint** in unpaired image-to-image translation using pre-trained diffusion models.
We expand on what we have discussed in Section 6.3 regarding the several existing efforts to study cycle consistency in diffusion-based I2I translation.
- UNIT-DDPM ([39], Apr 2021) made an initial attempt in the unpaired I2I setting, training two DPMs and two translation functions from scratch. Cycle consistency losses are introduced in the translation functions during training to regularize the reverse processes. At inference time, the image generation does not depend on the translation functions, but only on the two DPMs in an iterative manner, leading to sub-optimal performance. This preprint does not have any public implementation, and can not be reproduced given the limited information in the paper.
- DDIB (Mar 2022): The key proposition states that exact cycle consistency is possible **assuming zero discretization error**. DDIB itself does not enforce any cycle consistency constraint:
> “DDIBs first obtain latent encodings for source images with the source diffusion model, and then decode such encodings using the target model to construct target images.”
DDIB proposed one of the first theoretic analyses on the possibility to perform unpaired I2I translation with diffusion models and is the theoretic foundation for various diffusion-based image translation models, including prompt2prompt, NullText, and CycleDiffusion.
- CycleDiffusion ([46], Nov 2022) proposes a zero-shot approach for image translation based on DDIB’s observation that a certain level of consistency could emerge from DMs. There is again no explicit treatment to encourage cycle consistency.
> “CycleDiffusion can be seen as an extension of DDIB approach to stochastic DPM” [46]
> “The method is based on the observation that cycle consistency naturally emerges from diffusion models, i.e., the same random noise leads to similar images. Therefore, there is no training objective that explicitly encourages cycle consistency.” (https://github.com/ChenWu98/cycle-diffusion/issues/16)
If there is any similar prior work the reviewers are aware of, please let us know.
**Q: Since a certain level of cycle consistency could emerge in diffusion models, do we benefit from enforcing it as a constraint?**
Yes.
We echo the comments of reviewer rhuF:
> “When we apply pre-trained DM models on I2I tasks, it is difficult or unpredictable to control the mask and attention maps to achieve the desired results.”
While exact cycle consistency is possible assuming zero discretization error, the assumption generally doesn’t hold true. Pixel-level consistency remains to be challenging when there exists image conditioning, as can be shown in the failure cases of baselines presented in our paper, and the constantly changing backgrounds in image and video synthesis tasks.
**Q: Why is it non-trivial to enforce cycle-consistency constraints in diffusion models?**
Unlike GAN-based models, in which the translation between images is end-to-end, diffusion-based methods are iterative and lack such a simple and intuitive translation cycle. Also, for text-guided diffusion models, it is difficult to design a discriminator, which is a key component to ensure cycle consistency in GAN-based models. To address this limitation, we carefully formulated the problem in Section 2 and theoretically derived the training objectives in Section 3.