Summary
This paper presents MVDiffusion, a new diffusion model to generate consistent multi-view images, e.g., panorama. The authors propose a novel correspondence-aware attention mechanism in order to enforce pixel-level correspondence and cross-view consistency. More specifically this mechanism is used in three modules: a generation module that generates consistent low-resolution multi-view images; an interpolation module that generates images in between; a super-resolution module. After injecting such module into Stable Diffusion Unet layers and fine-tuning the model on multi-view image dataset, the model can synthesize consistent multi-view images based on text or depth.
Strengths
1. The main idea (improving multi-view consistency through a correspondence-aware attention mechanism) is novel, simple, effective and easy to understand. The generated results are very impressive, see figure 1, 4, 5, 6 and all the images in the supplementary.
2. The paper is well-written especially the method part where the main design is demonstrated clearly, see section 4.1 and figure 3. The authors also provide the code in the supplementary.
3. The comparison with previous methods and some straightforward baselines are comprehensive and the improvements are convincing.
Weaknesses
1. The pipeline figure (Figure 2) could be further improved: this figure is supposed to make the audience understand the whole workflow without looking at the method text, however, for now, in this figure, it's a bit unclear to me how these modules are associated with each other at first glance.
2. Adding a brief section on failure cases would be great: the results in the main paper and supplementary are truly impressive, however, I'm also curious what are the common failure cases of MVDiffusion. Adding a short paragraph discussion this would be beneficial to the whole community.
3. I feel a bit confused about the panorama generation during inference time: in Line 145, it says "The generation module generates eight 256 × 256 images, and the super-resolution module upscales to eight 1024 × 1024 images". I am just wondering where's the interpolation module then? Or let me put this way, how many images are generated by the generation module and how many are generated by the interpolation module? I assume the key frames are generated by the generation module and more images in between are generated by the interpolation module?
4. How to handle the conflicts in the overlap region between two generated images? If I understand correctly, although correspondence attention works very well, it still cannot guarantee the overlap region between two generated images are *exactly* matched, so I'm just curious how does this mismatch is handled? I asked this because in the stitched 8 perspective images, it looks very consistent.
5. Three modules are proposed in this paper, it would be great if an ablation study could be conducted such that the audience could understand the influence of each individual module better, e.g., how much difference it would make if removing the interpolation module?
6. In addition to the stitch image, it would be great if showing more *densely* (much more than 8) generated multi-view images and make it an animation, such that the multi-view consistency could be better evaluated qualitatively by the audience.
Questions
It would be great if the authors could respond the points mentioned above. Thanks!
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.