Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis

Summary

The paper proposes SCGaussian, a few-shot 3D Gaussian Splatting model to address novel view degeneration in sparse input scenarios. SCGaussian leverages matching priors to enforce 3D consistency by optimizing the position of Gaussian primitives along rays, overcoming challenges in monocular depth-based methods. This hybrid representation binds ray-based and non-structure Gaussian primitives, ensuring accurate scene structure. Extensive experiments demonstrate SCGaussian's state-of-the-art performance, achieving significant improvements in rendering quality and efficiency.

Strengths

1. The paper is written clearly, making it easy to follow. 2. The experiment is very comprehensive. 3. The proposed method achieves state-of-the-art performance in the task of few-shot NVS, despite not requiring sparse 3D points as input like other works did. 4. The combination of proposed optimization of the position of Gaussian primitives (section 3.3) between optimization of the rendering geometry is quite novel. The ablation studies of section 4.2 clearly demonstrate the impact of dual optimization.

Weaknesses

- The paper claims it does not require SFM points for initialization. However, it heavily relies on precomputed camera poses. When using COLMAP to obtain camera poses, SFM points are generated effortlessly as a byproduct. Alternatively, if we have camera poses and 2D correspondences, why not use triangulation to obtain 3D points? - Again, I am curious about the advantages of the proposed optimization of Gaussian primitives' positions (Section 3.3) over triangulation. Given that SVD in triangulation is significantly easier compared to optimizing a learnable depth parameter, it would be more convincing to include experiments evaluating NVS quality, generalization, optimization time, and computational resources. - The novelty of the paper is somewhat limited and may not meet the high standards expected at NIPS. - The author mentioned sparse input in large scenes, and I'm curious if the proposed method can be generalized to even larger scenes, such as driving scenes like Waymo or aerial photography data. For instance, in driving scenes, there is often a challenge of losing matching to the road area due to its lack of texture. - The details regarding the matching prior should be more comprehensive, including information such as the average number of pairs extracted, the time it takes for the extraction process, the hyperparameters used in the model, and other relevant details. - The evaluation would be more convincing with the inclusion of additional geometric metrics, such as Chamfer Distance.

Questions

All questions are present in weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations have been disscussed in the paper.

Summary

The paper proposes a structure guided novel view synthesis method for Gaussian Splatting. Experiments show that the proposed method produces clearer rendering results than other existing works.

Strengths

* The proposed method is well evaluated with various methods in neural rendering. Experiments is sufficient to support the claims. * Demonstrations show that the proposed method is simple but efficient.

Weaknesses

* The proposed method employ a pre-trained matching model to obtain matching priors. It is a strong dependency. A discussion / ablation study on the effect of its performance is necessary. * In addition, failure cases under any special scene should be discussed in the limitation. For example, the surrounding views are not always available in trafic scenes so that there are serious occlusions/dynamics in observation. How about the performance under such condition?

Questions

Please refer to the weaknesses.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Discussion over failure cases should be included.

Summary

This work presents a Structure Consistent 3DGS method using matching priors to learn 3D consistency. A hybrid Gaussian representation, including non-structure Gaussian primitives and ray-based Gaussian primitives, is introduced. Position Consistency loss between two views is adopted to achieve multi-view alignment. Extensive experiments prove the effectiveness of the proposed method.

Strengths

1. This paper is well-written and easy to understand. 2. The concept of the ray-based Gaussian primitives is interesting, which can solve the issue that 3DGS tends to increase the size of Gaussian primitives. 3. The proposed method achieves SOTA performance. The ablation study has also verified the effectiveness of the proposed method.

Weaknesses

1. The optimization of the rendering geometry in lines 195-203 is not novel, which is similar to the correspondence pixel reprojection loss in CorresNeRF [A]. 2. Which network is used for feature matching? 3. For the ray-based Gaussian primitives, do they have the shape? If not, can we call them Gaussian primitives? 3. How do the authors select the value of N? Is it defined by the results of feature matching? 4. Why does this work in line 177 define N pairs of ray-based Gaussian primitives rather than N ray-based Gaussian primitives? Since The matched pixels should have the same 3D surface points. [A] Lao, Yixing, et al. "Corresnerf: Image correspondence priors for neural radiance fields." Advances in Neural Information Processing Systems 36 (2023): 40504-40520.

Questions

Please refer to the Weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have clarified the limitations.

Summary

This paper proposes SCGaussian that utilizes a pre-trained image matcher to get the dense pixel-wise matching correspondences between sparse observing views, then regard these as the prior for 3DGS to achieve high-quality few-shot novel view synthesis. It innovatively binds Gaussians to the matched pixels and restricts their position to be along the ray, which can keep the scene geometry close to that of the image matcher predicted. Experiments on various datasets demonstrated its superior performance and efficiency compared to existing methods.

Strengths

- The ray binding is novel and interesting, which constrains the reconstructed scene geometry not too far from the matching correspondence, and thus avoids an overall structure collapse caused by overfitting. - The division of structured and non-structured Gaussians is well-motivated. - Experimental results are good in visualization and quantitative results.

Weaknesses

- As the correspondences and camera poses are all already known, why do not just initialize the Gaussians just at the intersection point? Furthermore, the introduction of a more powerful image matcher conflicts with the motivation declared in lines 145-146, as once the high-quality matching correspondences are sufficient, SfM methods can also produce high-quality point clouds. The main motivation of this paper is not convincing. - Despite the authors have conducted some experiments and tried to verify the effect of the proposed dual optimization strategy, it's unclear what is exact the setting of "w/ Matching prior" in Table 4 and "Straightforward combine matching priors" in Figure 9. Therefore, they are not convincing for me to verify the effect of the proposed strategy. - The method seems to heavily rely on the image matcher, however, nearly all evaluation datasets have strong textures, which are friendly to image matching methods, and only one extremely powerful matcher GIM [39] is used for the method. Would like to see more evaluations to prove the method's robustness when the matching correspondence is not ideal, e.g. using more kinds of matcher, and reconstructing scenes with large texture-poor regions. [39] Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In ICLR, 2024.

Questions

- The main concern is that I'm not sure whether the improved performance is just from a more powerful matching prior or the proposed designs. If most regions can be correctly matched according to GIM or other matchers, it would be very easy to build a high-quality point cloud and take it as the initialization, which can also bring a good structure prior, especially when applying to some anchor-based works like Scaffold-GS [1]. Please refute this with experiments or reasoning to prove the necessity of the proposed designs. - Would like to see more explanations for the second point of Weaknesses. - To evaluate the robustness, experiments using more kinds of matcher, and reconstructing scenes with more texture-poor regions (like DTU) are expected to be added. [1] Lu, Tao, et al. "Scaffold-gs: Structured 3d gaussians for view-adaptive rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Rating

7

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors discussed the limitation of accurate camera pose requirement. Besides, this work is built upon one specified off-the-shelf image matcher, yet does not verify its robustness for other choices.

Reply to the authors

Thank the authors for addressing my concerns. They have addressed my concerns, and I will raise my score to a borderline accept.