GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping

Summary

The authors propose to achieve novel view synthesis from a single image. Contrary to prior work, which warps using a monocular depth estimate and then inpaints, a more flexible architecture is introduced. To do this, a diffusion model is conditioned on warped coordinates, based on the desired pose change and a monocular depth estimate. In addition, the input view (and base coordinate map) is featurized and the diffusion process is allowed to cross attend to these features, which should intuitively allow for attention between corresponding points. The method is then trained end-to-end with a denoising diffusion objective. Experiments show that this approach achieves better FID, PSNR, and LPIPS scores than baselines on RealEstate10k and ScanNet. Ablations justify the choice of warping conditioning.

Strengths

The manuscript is well written and well motivated. The related work section is comprehensive, the methods section is straightforward and easy to understand. And experimental results seem quite strong.

Weaknesses

I think the paper could benefit significantly from apples-to-apples ablations. For example, I am quite interested in a comparison between the proposed method, and the method without any depth information (I think similar to GeoGPT). Or perhaps the proposed method, and the method without the warping. The authors also claim that the cross-attention “allows the model to inherently decide which regions should rely more on its generative capability and which areas should depend primarily on the information from the input view warping.” However, they provide only a single example of this in figure 4. I think more examples would make for more convincing evidence. I would be interested in a larger scale, more robust study of this hypothesis. For example, this could possibly be automated by computing how closely the cross-attention actually matches warping. The SD warp baseline could possibly be implemented better. I believe the inpainting mask covers non-warped black pixels, resulting in many unrealistic “black borders” is in Figure 10 and Figure 5. Perhaps a more fair and robust baseline could be achieved by just expanding the inpainting mask a bit. Text2room by Höllein et al seems relevant. Perhaps the authors would consider citing it.

Questions

From the weaknesses section: could apples-to-apples ablations be conducted on components of the method? From the weaknesses section: could more evidence for “cross-attention attends to corresponding points” be presented? One major benefit of the proposed setting, as opposed to warp then inpaint, is that the method should be able to handle view-dependent effects. This may be interesting to investigate.

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations and societal impacts are included, and adequately addressed, but are not in the main body of the paper.

Summary

This paper proposes a semantic-preserving generative warping framework to generate high-quality novel views from a single image, which mainly consists of two components: - Condition the novel view synthesis on the warped 2D coords embedding from the estimated depth map. - Augmenting cross-view attention with self-attention. The proposed method eliminates the artifacts caused by error depths in the warping-and-inpainting pipeline and integrates semantic features from source views, preserving semantic details in generation.

Strengths

- Compared to existing warping-and-inpainting methods that condition on explicit depth warping, the proposed warped 2D coords embedding forms the correspondence between reference view and target view implicitly, which helps the network to be more robust to the noise in the estimated depth map without losing sematic details. - The paper is well-written, being clear and easy to follow. The technical limitations are also discussed in detail in the Sup.

Weaknesses

- The proposed embedding based on depth warping could not benefit the synthesize process when there are large camera movements or occlusions between the input view and the target view. - Although the proposed depth-warping embedding somehow reduces the influence of the noise in the estimated depth map, compared to explicit pixel warping, the implicit condition of depth embedding also lower the local preservation ability of the network when synthesizing novel views. Is the model capable of synthesizing consistent novel views which could be used to reconstruct a 3D scene ? - The comparison baseline is limited. Plucker embedding[1] is also a dense embedding capable of providing local correspondence, which supports large camera movement and occlusions. The difference between such dense embedding with the proposed depth warping embedding should be further discussed. [1] SPAD : Spatially Aware Multiview Diffusers(CVPR2024)[https://arxiv.org/abs/2402.05235]

Questions

As listed in the Weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No specific limitation and negative societal impact need to be addressed.

Summary

The paper presents a novel framework called GenWarp, which aims to generate new views from a single input image while preserving the semantic content of the original view. This is achieved by leveraging a generative process that incorporates both self-attention and cross-view attention mechanisms conditioned on warping signals. The proposed approach demonstrates superior performance compared to existing methods, especially in scenarios with challenging viewpoint changes, and exhibits good generalization to out-of-domain images.

Strengths

1. The paper introduces an approach for single-shot novel view synthesis by combining self-attention and cross-view attention mechanisms to preserve semantic details. 2. The proposed method outperforms existing techniques in generating high-quality novel views, particularly for challenging viewpoint changes. 3. GenWarp shows generalization capabilities, performing well on out-of-domain images, which indicates its applicability to various scenarios.

Weaknesses

1. The paper could further highlight its novelty by providing a more comprehensive comparison with a wider range of state-of-the-art methods, including both classical and recent approaches. Additionally, the authors should discuss any potential limitations of their approach in terms of scalability or adaptability to different types of scenes. 2. The methodology section could benefit from additional diagrams and flowcharts that illustrate the workflow and attention mechanisms in more detail. Including intermediate results and step-by-step visualizations would help readers better understand the progression from the input image to the generated novel view. 3. While the paper provides detailed instructions for reproducing the experiments, the code and data are not made publicly available at the time of submission, which can hinder reproducibility efforts. 4. The performance of GenWarp is highly dependent on the quality of the datasets used for fine-tuning, which could limit its effectiveness if high-quality multi-view datasets are not available. 5. The method struggles with generating novel views when the camera viewpoints are extremely distant, indicating a limitation in handling very large viewpoint changes. 6. The reference list could be updated to include more recent advancements in the field, particularly those published in the last year. Additionally, a more detailed comparative analysis of the strengths and weaknesses of related methods would be beneficial. 7. what is the advantage of diffusion-model-based pipeline over 3D Gaussian[A,B] or NeRF[C,D] based pipeline, can the author discuss more about existing methods? [A] Yu, Zehao, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. "Mip-splatting: Alias-free 3d gaussian splatting." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456. 2024. [B] Yan, Zhiwen, Weng Fei Low, Yu Chen, and Gim Hee Lee. "Multi-scale 3d gaussian splatting for anti-aliased rendering." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20923-20931. 2024. [C] Deng, Congyue, Chiyu Jiang, Charles R. Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, and Dragomir Anguelov. "Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20637-20647. 2023. [D] Yang, Yifan, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. "Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15901-15911. 2023.

Questions

Please refer to the weakness section.

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Summary

This paper proposes a novel single-shot novel view synthesis framework based on a pretrained T2I diffusion model. Instead of directly warping pixels between input view and novel view, an implicit approach is proposed to conduct geometry warping operation. The cross-view attention is used to eliminates the artifacts caused by error depths and integrates semantic features from source views, preserving semantic details in generation. Extensive experiments prove the efffectiveness of the proposed method.

Strengths

1. The implicit geometry warping approach is effective in addressing ill-warped problem and missing original semantic problem encountered in explicit warpping methods. 2. The cross-view attention can provide more useful information for novel view generation. 3. The qualitative and quantatitive experiments on RealEstate10K , ScanNet, and in-the-wild images show the proposed method outperform SOTA methods in both in-domain and out-of-domain scenario.

Weaknesses

1. Although the cross-view attention strategy is effective to fuse features, it may be not novel enough. There are many similar operation in video generation area. 2. The detail of finetuning the T2I model is not clear.

Questions

N/A

Rating

6

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

This method is restricted to depth-based correspondence of two views which limit its application in some scenes where depth correspondence between two views is missing.