Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis

Summary

This paper proposed a 3D Gaussian Splatting method to render novel views with sparse inputs. This paper first use a pre-trained keypoints matching network to generate dense point initializations, and propose a consistent loss between the warped binocular stereo image. Meanwhile, they introduce a opacity penalty strategy to further learn the accurate gaussian points. The proposed method achieves the state-of-the-art performance on LLFF, DTU and Blender datasets.

Strengths

1. The dense point generated from the pre-trained keypoints matching network can provide much better initialization for 3DGS with sparse inputs. 2. Using the generated binocular views to add the consistent loss can introduce more constraints in the sparse scenario. 3. The reported results seems to be much better than existing methods.

Weaknesses

1. While the reported results show that the opacity penalty strategy brings the best improvement, it's motivation is not clear. - Why add a coefficient to the opacity can guide the gaussian point to be closer to the scene surface? (L52. "guiding the remaining Gaussians to be closer to the scene surface"), especially under the setting of $\lambda=0.995$ (has a minor difference with 1.0), and are there any ablation results with different value configurations of $\lambda$. Because the coefficient only works on the opacity, the opinion of guiding the position of Gaussian to be closer to the scene surface does not convince me enough. - The statement of "the gaussian close to surface has larger gradient and the gaussian far from the surface has smaller gradient" (Line176-179) seems unconvincing. The accurate Gaussian near the surface can render accurate colors with relatively small computational loss and should have small gradients. And on the contrary, the wrong Gaussian may has a larger gradient, like the situation of "wall" in the overfitting scenario. - Such a small modification can bring nearly 2dB improvement is shocking, and the explanation and content of this part is not adequate. Adding more analysis might help make it more convincing. 2. The proposed binocular stereo consistency is actually a method to regularize the model using unseen virtual views. - And using the virtual unseen view is a demonstrated existing strategy that has been adapted by many methods like [1,2,3], and the difference of this method is using the different sampling space of virtual views (the sampling space in this paper only has horizontal translation and no rotation). - Intuitively, using more diverse sampling space like [1,2,3] may be more powerful, because it can use more unseen views from different angles to regularize the model. So what are the advantages of sampling only in the horizontal direction and are there any experimental comparisons? - The disparity-based or depth-based warping will introduce the occlusion, which has a great impact of the performance. And how to mitigate this is not clarified in this paper. - There might be an inaccuracy about the method. It seems like you want to use the backward warping to warp the right image to the left camera coordinate (Line148-160). Thus the depth needs to be pixel-aligned with the left image in Eq. (3), but you just use the depth of right image. This just confuses me a lot. 3. It seems that the dense initialization using the keypoints matching network plays an important role in the overall performance. - How about the sensitivity of the method to different matching network? - How about the performance of the method w/o dense or sparse initialization and only w/ the random initialization like DNGaussian. Because in the sparse scenario, popular SFM methods are hard to work and always fail to generate accurate points. 4. There are also some confusions in the experiment. - The results reported by FSGS differ significantly from those in the paper, and there seems to be no explanation for this in the paper. - Some visualization results (e.g., Fig. 3, Fig. 4 and Fig. 9) are not labeled with the corresponding methods, which makes comparison difficult. - Since the runtime of 3DGS-based method is relatively fast, reporting the error bar or standard deviations of multiple experiments can help to enhance the credibility of the paper. [1] SPARF: Neural Radiance Fields from Sparse and Noisy Poses, CVPR2023. [2] GeCoNeRF: Few-Shot Neural Radiance Fields via Geometric Consistency, ICML2023. [3] RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs, CVPR2022.

Questions

Refer to weaknesses for details. And due to these doubts, I tend to give the borderline and hope to see the author's response.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Have declared in the paper.

Summary

This paper proposes a new method related to the problem of novel view synthesis in a sparse input setting. The authors propose to exploit stereo consistency as a self-supervision signal in contrast to the use of priors such as diffusion which tends to produce less precise geometry. Specifically, the work proposes the use of binocular stereo consistency as a guiding signal, i.e. with a horizontal translation a disparity between two images can be calculated given intrinsics and depth, and this can be leveraged to warp the translated view to input image and assess the consistency between images. Furthermore, the authors address the issue of filtering the redundant gaussians by introducing a decay scheme based on penalising the opacity leading to pruning gaussians far from the surface of the object. The paper evaluates the proposed approach in a commonly used in literature sparse novel view scenario, using 3 common datasets. The authors present competitive results in their evaluation. An ablation of system elements/contributions is also included.

Strengths

- This paper touches upon an interesting and important topic for the community - novel view synthesis in a constrained scenario. This work is among the first ones to propose a solution based on 3D Gaussian Splatting. - The method proposed by the authors draws some inspiration from monocular depth estimation and formulates a self-supervision loss component applicable in 3D reconstruction. I find the idea novel and interesting - I appreciate the application of a relatively simple concept (correlation between depth and disparity) to the task. - I believe that the big strength of the method is the focus on precise geometry and the lack of dependence on priors from pre-trained models. - The paper clearly outlines performance improvements with respect to the previous state of the art, per-scene optimisation methods. - The work evaluates the performance of the proposed method on 3 datasets well-known in the community showing improvements in all of them. I believe it is a suitable selection used throughout the literature (following the same setup of 3,6,9 views). The ablation study is suitable for the work showing the performance impact of separate components (dense initialisation, disparity consistency loss, opacity penalty). - I find it particularly nice that the binocular stereo consistency can be added to effectively any 3D reconstruction method (provided having depth as an output). While I don't expect it to be heavily affecting dense reconstruction, I believe it would be looked into and tried to incorporate by many researchers working in sparse reconstruction. - The paper is written clearly, I didn't have any trouble following it. - The method is described in a way that should be reproducible - I believe I wouldn't have significant trouble reimplementing it.

Weaknesses

- The main weakness I see is the lack of comparison to generalisable methods [1, 2, 3, 4, 5, 6, 7]. DTU dataset which is already used in the paper is also commonly used to benchmark methods that perform a training step on a selection of scenes. While the setup of generalisable, and per-scene optimisation is favourable towards the former, I believe the proposed method would be competitive. This would potentially be an interesting comparison, both quantitatively (I would expect competitive scores), and qualitatively (varying artefacts between the two approaches), and would additionally increase the value of the benchmark. - I believe that computational performance comparison is a missed opportunity. Proposing one of the first methods for sparse novel view synthesis would benefit greatly by emphasising the speed differences between the methods. It would be a great addition to show the training and inference speed with respect to NeRF-based methods (to show improvement in quality metrics and speed), and other GS-based methods (to show probably similar speed, but increased reconstruction quality). - Both disparity consistency loss and opacity-based pruning rely on the choice of hyperparameters - namely camera shift $d_{max}$ and decay coefficient $\lambda$. It would be great if this choice was motivated and its influence on the performance analysed (including analysis of whether the same values should be applied across tested datasets). - The authors show estimated depth maps for their method with and without the use of view consistency loss. It would be good to see the depth maps from other methods (e.g. DNGaussian) - it would be interesting to see a comparison with methods that use depth priors as supervision. It would also be nice to see the comparison to ground truth depth (where not available, an interesting comparison would be the reconstruction method in a dense setup scenario - as a proxy for ground truth) - The authors mention in the checklist that results in the paper are reported as an average performance - it would be good to specify whether experiments were run multiple times, and investigate how the performance varies across the runs. [1] Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa, *pixelNeRF: Neural Radiance Fields from One or Few Images*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021 [2] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang Jingyi Yu, Hao Su, *MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo*, IEEE/CVF International Conference on Computer Vision, 2021 [3] Mohammed Suhail, Carlos Esteves, Leonid Sigal, Ameesh Makadia, *Generalizable Patch-Based Neural Rendering*, European Conference on Computer Vision, 2022 [4] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, Xiaowei Zhou, *Efficient Neural Radiance Fields for Interactive Free-viewpoint Video*, SIGGRAPH Asia, 2022 [5] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, *Is Attention All That NeRF Needs?*, International Conference on Learning Representations, 2023 [6] Thomas Tanay, Matteo Maggioni, *Global Latent Neural Rendering*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024 [7] Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Marc Pollefeys, Andreas Geiger, Fisher Yu, *MuRF: Multi-Baseline Radiance Fields*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

Questions

- How was the camera shift $d_{max}$ and decay coefficient $\lambda$ chosen? Is the same value appropriate to all datasets? I would imagine it would be increased and decreased based on the distance of the object to the camera. Also, are the values of camera shift sampled uniformly? - How would the application of disparity consistency loss to the existing methods look like? I would imagine it would be rather straightforward. Did the authors try that? - Previous methods, as pointed out by the authors, use depth priors (using pre-trained models) as the supervision. Have the authors tried adding such supervision to their approach? I understand that the method is designed to be prior-free but it would be interesting to see an indication whether this would improve the performance. - Zoom-in areas could be put side-by-side (possibly in the appendix, or another row in the figure with zoom-ins). It is a bit hard to compare small elements the authors want the reader to focus on when zooming the document - the focus areas (red boxes) are a bit far away and hard to notice small differences.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors provide a brief limitation section. I believe the issue with consistency constraints in textureless areas is a very good mention. Also, the experiment with masked training shows that the limitation is identified correctly.

Summary

This paper proposes 3D Gaussian splatting from sparse views aided by pre-trained key points matching initialization, binocular stereo constraints, and opacity regularization. Binocular stereo constraints utilize perspective projection to warp synthetic stereo images into the training images for self-supervision. Opacity regularization guides the densification/pruning process to drop non-active Gaussians.

Strengths

Good initialization of Gaussians and geometrically inspired regularization for 3DGS (stereoscopic constrains and opacity decays)

Weaknesses

1. Adding an opacity regularization does not seem enough to consider it a contribution. 2. Overstatements. The authors claim their method is prior-free, however, they use a very strong prior for dense initialization. 3. Unclear results. How is your method without any of your contributions (Table 4) 1.5 dB better than the baseline 3DGS? 4. Missing ablation studies. What is the performance if only stereo loss or only density regularization is applied? 5. I am afraid the main performance gains come from the dense keypoint initialization. 6. Inconsistent performance improvements in Table 4. Dense keypoint initialization provides ~3dB improvements, and stereo consistency and opacity regularization provides ~4dB. How is the combination of both only providing ~5dB? I understand improvements cannot be linearly added, but the gap seems unreasonable.

Questions

See weaknesses

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors provided limitations. However, the occlusions due to stereoscopic synthesis where not addressed (even though they are small due to small camera baselines).

Summary

This paper introduces a novel method for 3D Gaussian-based sparse view synthesis. Specifically, initialized from dense point clouds, the depth-warping loss and the opacity penalty strategy are introduced to obtain accurate 3D Gaussians. Extensive experiments on the Blender, LLFF and DTU dataset have demonstrated the effectiveness of the proposed method.

Strengths

1. This paper is well-writen, the idea of depth-warping loss to improve view consistency is reasonable. 2. The proposed opacity penalty strategy is novel and does make sense. Through this simple operation, a better geometry can be obtained. 3. The experimental comparison is comprehensive.

Weaknesses

1. Since depth-warping loss is common in many NeRF or 3D Gaussian-based papers, i'm not sure whether this can be viewed as a contribution. 2. Since the dense point clouds are generated from pretrained models, i do not think that this method can be claimed as prior free, please check the definition of what is prior free.

Questions

Please see weaknesses. Since the initialization is based on dense point clouds, i think it would be better to do more ablation studies on more dataset, such as the DTU and the Blender dataset. However, in general, i think this is a relatively good work and i would recommond a borderline accept for this paper. I'm glad to improve rating if the weaknesses can be addrssed.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please see weaknesses.

We will be happy to take any questions

Dear reviewers, We appreciate your comments and expertise. Please let us know if there is anything we can clarify further. We would be happy to take this opportunity to discuss with you. Thanks, The authors

Thanks for your reply

Thank you for addressing my comments. I am willing to raise the score, provided the clarifications and ablation studies are included in the final paper version.

Thanks for detailed response

Dear Authors, Thank you for the amount of information provided in response to my review. I have read your rebuttal. I find the experiment with using your additional supervision with DNGaussian particularly convincing. I saw that performance is very competitive against MuRF (except 3 views DTU but that may be due to how DTU is evaluated, i.e. same views for all the scenes). One last worry I have is whether the choice of $\lambda$ is empirically easy. The method seems to be rather sensitive to small changes. Unless the value of $0.995$ is the best or close to best for all dataset.