SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions

Summary

In this paper, the authors propose a new module called SyncDiffusion that enables to synchronize multiple outputs from diffusion models to create coherent montage. This module guides the reverse diffusion process in each window by shifting x_t with gradients of a perceptual loss between the target window and an anchor window. The experimental results show that SyncDiffusion contributes to produce more coherent montage compared to previous methods.

Strengths

- This study adresses an important problem, namely how to synchronize multiple estimates from diffusion models to produce a single output. In particular, how to guarantee a global conherence in the output image is highly non-trivial, but should be critical for practical applications such as panorama image generation. - The main idea, which is to use gradients of a perceptual loss between target and anchor windows to ensure the global coherence, is simple and can be jointly used with various methods for combining multiple outputs from diffusion models.

Weaknesses

- The current design of the proposed algorithm relies heavily on the homogeneity of the target scene, which may limit its applicability. This is due to LPIPS used for guidance, which forces every window share a similar composition of the image. Personally, I wonder this causes the failure cases shown in Fig. 6. - The evaluation by Intra-LPIPS and Intra-Style-L is interesting to see a global coherence in a resultant image, but the superiority of the values is not clear, because it is unknown how much the values will be in real panorama images. - Somewhat related to above, the effectiveness of the proposed method is not clear. In the quantitative comparison, Mean-CLIP-S and Mean-GIQA are on par, and FID is even worse than that of MD (I understood why it happans, though). On the other hand, the user study is conducted only for evaluating the coherence, not for image quality nor prompt fidelity. - Although the authors argue that SyncDiffusion is a plug-and-play module, it is evaluated only with MultiDiffusion in the experiments.

Questions

- Please see Weaknesses. - FID of MultiDiffusion in Fig. 5 seems much different from that shown in the original paper [3] (around 30 vs ~10). Why is it? - It seems straightforward to apply SyncDiffusion to layout-to-image task by just adding an appropriate cropping process before computing the perceptual loss. This should make quantitative comparison much easier as done in MultiDiffusion or [R1]. Why did the authors focus on generating panorama images in the experiments? - [R1] "Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC," ICLR 2023. --- Given the authors' response, I updated the rating accrodingly.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

- Several limitation are described in the manuscript.

Summary

This paper proposes a diffusion synchronization module, a plug-and-play module for text-to-panorama generation. The key idea is based on the observation that predicted denoised images closely resemble the final outputs even at the beginning of the denoising process. Authors proposed to synchronize multiple diffusions through gradient descent from a perceptual similarity loss. Experiments demonstrate the effectiveness of the proposed method in generating coherent montages with good fidelity and compatibility with prompts.

Strengths

(1) This work is well motivated based on the observation that Predicted denoised images closely resemble the final outputs even at the beginning of the denoising process. In addition, it also helps local seamlessness and global coherence. (2) Evaluation metrics used in quantitative comparison experiments are elaborate. (3) The paper has high writing quality which makes it easy to understand.

Weaknesses

(1) Based on MultiDiffusion, the author introduces guidance information provided solely by gradient descent from a perceptual similarity loss. I wonder if this guidance is too weak or have they explored other guidance? (2) Pretrained stable diffusion v2 is used to generate 512x512-sized images. How can stable diffusion be utilized to generate wider images? Please provide a detailed explanation in the paper. The setting of text-to-panorama generation task should be further clarified. (3) Through the quantitative comparisons in Fig. 5, it can be observed that the setting of gradient descent weights has significant impact on the generation results. However, it is unclear how this affects the qualitative or visual effects. I wonder if the authors have experimented with other weights and is it possible that a weight other than w=20 could yield better results. Additional Comments (1) L171, L179, writing errors, Fig. 4 -> Fig. 2 (2) L191, L198, writing errors, Fig. 4 -> Fig. 3

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, authors mentioned the limitation of the proposed in its high sensitivity to prompt and additional computation overhead. They also mentioned its potential negative sociatal impact on deepfake.

Summary

The paper tackles the problem of image diffusion models for generation montages/panoramas. Their proposed method, dubbed SyncDiggusion, improves on the state-of-the-art (both qualitatively, through user studies, and in quantitative metrics) by synchronizing multiple diffusions through gradient descent using a perceptual similarity loss. This provides a guidance in the reverse diffusion process by adjusting intermediate noisy images at each step. Notably, off-the-shelf perceptual similarity losses can be interchangeably used with this method.

Strengths

1. The paper proposes a novel and interesting solution to the problem of consistent panoramic image generation using diffusion models 1. The method is original, to my knowledge 1. The paper is well written and clear 1. The paper presents adequate theoretical and algorithmic explanations 1. The paper presents a comprehensive set of both qualitative and quantitative results that demonstrate the benefits of their method over previous state-of-the-art, highlighting their contributions well

Weaknesses

1. Coherence is measured not with the current noisy color images $\{D(x_t^{(i)})\}$ but with the predicted denoised color images $\{D(\phi(x_t^{(i)}, t))\}$ (L195). What is the computational overhead of this computation? Similarly, What is the added computational overhead of this method as compared to the reported baselines? A discussion of this is omitted, although mentioned in brief in the limitations sections. 1. In turn, a discussion of the tradeoffs between consistency vs. diversity vs. computational overhead would then be beneficial.

Questions

1. Can this work be extended to other applications such as videos? 1. It was mentioned that choice of prompt is critical in this method, and can easily break it. How were the prompts for the user study chosen? Choice of prompts would greatly influence the outcome of the study, potentially in favour of the proposed algorithm.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, limitations are discussed in brief.

Summary

The paper addresses the problem of lack of global coherence in panoramas generated with diffusion models (in a tiled manner with overlapping regions). This work builds upon MultiDiffusion [1], which proposed the technique of averaging multiple predictions for an overlapping region at each denoising step to avoid edge artifacts. However, MultiDiffusion lacked long range coherence. This paper proposes interleaving an additional guidance step using perceptual similarity loss to ensure that each region is coherent with an anchor region. [1] MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, Bar-Tal et al, 2023

Strengths

- The solution of guiding the denoised image with a perceptual similarity loss for global coherence is simple yet effective - Qualitative and quantitative results are thorough and back the effectiveness of the approach in achieve global coherence - Paper is very well written, adequately discusses background, related work and limitations.

Weaknesses

- This slows down the inference process as the authors discuss in the supplementary

Questions

None

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Reply

Thanks for the authors' response. I have read it as well as other reviews. I would like to thank the authors for providing the additional experimental results. Specifically, given the user study on image quality and quantitative comparison with real panorama images, the effectiveness of the proposed method becomes much clear. It is still a bit less clear whether it is really a good idea to use LPIPS for the guidance, but exploring various guidance loss for better conherence would be one of the future work as the authors stated in the rebuttal. I will update my rating accordingly. Thanks.

Reply

Thank you for the response. The additional results are appreciated and shed further light on the effectiveness of this method, as mentioned by reviewer DATV. I have additionally read the responses to the other reviewers. Thank you for the clarification on the user study prompts, I suspected as much. My concerns with computational overhead remain however the reviewers do in fact discuss these concerns well and no one paper is expected to solve all problems. I am more confident now in upgrading my rating to a 7. Thanks.