3D Gaussian Splatting as Markov Chain Monte Carlo

Summary

This paper proposes a novel densification strategy of 3D Gaussian Splatting (3DGS) based on the Markov Chain Monte Carlo (MCMC) sampling scheme. The authors address the ‘heuristic’ densification of standard 3DGS and adopt a distribution-aware resampling pipeline.  Consequently, they achieve higher rendering quality compared to standard 3DGS with a similar number of Gaussian primitives.

Strengths

This paper is well-written to understand. - The authors tackle the limitation of heuristic densification of standard 3DGS and suggest an MCMC sampling strategy for 3DGS. Also, the detailed analysis effectively supports their theoretical statements. - It achieves standard 3DGS in rendering quality while preserving the fast inference time with the same format of 3DGS representation. Therefore, it can be used in various applications of 3DGS without any modification.

Weaknesses

- The additional computational costs are required for MCMC resulting in the increase of training time compared to standard 3DGS. - They only suggest the resampling strategy of 3DGS. Therefore, the technical contribution is inefficient despite the detailed analysis.

Questions

- Although they have mentioned the computational time in Sec. B of the appendix, I cannot understand how much it takes compared to standard 3DGS. Can you provide the comparison in training time of this method and standard 3DGS for each scene? - Although it has been described in L184-186, I am confused about the reason why adding noise to other parameters leads to harmful results. Can you describe more details about it?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

It only tackles the cloning strategy without any other problems for achieving higher rendering quality. Thus, the technical improvement seems to be inefficient.

Summary

The paper discusses improvements to 3D Gaussian Splatting in neural rendering. Current methods rely on complex cloning and splitting strategies for placing Gaussians, which often do not generalize well and depend heavily on good initializations. The authors propose rethinking 3D Gaussians as random samples from an underlying probability distribution of the scene, using Markov Chain Monte Carlo (MCMC) sampling. They show that 3D Gaussian updates can be converted into Stochastic Gradient Langevin Dynamics (SGLD) updates by adding noise. This allows for the removal of heuristic densification and pruning strategies, replacing them with a deterministic state transition of MCMC samples. Additionally, the authors introduce an L1-regularizer on Gaussians to encourage efficient usage. Their method improves rendering quality, provides easy control over the number of Gaussians, and is robust to initialization across various standard evaluation scenes.

Strengths

1) This work is very insightful 2) Experiments are extensive and detailed

Weaknesses

/

Questions

/

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

/

Summary

Current 3DGS-based methods require carefully designed strategies such as cloning and splitting to assign a 3D Gaussian at a location. Further, they also require initializing points from SFM to generate high-quality novel views. The proposed work assumes that a set of 3D Gaussians are drawn from an underlying probability distribution, which is representative of the scene. Further, 3DGS updates are converted to SGLD updates by introducing noise. The main contributions of this work are as follows: - A fresh perspective that 3D Gaussians are sampled from a distribution and relocation strategy is compatible with MCMC samples. - Robustness to initialization. 3DGS-MCMC is not dependent on the initialization step in 3DGS. - The proposed method outperforms other NeRF-based methods and 3DGS on standard datasets.

Strengths

- **Qualitative and Quantitative Results:** The proposed method is evaluated on NeRF synthetic, Tank&temples, Deep Blending, MipNeRF360 and OMMO dataset. Quantitatively, 3DGS-MCMC outperforms NeRF-based methods and 3DGS. Qualitatively, the novel-views from 3DGS-MCMC are sharp compared to the 3DGS method (Fig. 2). This is due to the MCMC formulation proposed in this work, which allows exploration. - **High-Performance compared to 3DGS with a limited budget(L278-297):** The authors show an interesting experiment where they limit the budget for number of Gaussians during optimization. As expected, 3DGS has a significant drop in performance, whereas the performance drop in the proposed method is limited. Notably, there is a difference of 4 dB when the maximum number of Gaussians is set to 100k. - Unlike 3DGS, the proposed method is not sensitive to the initialization. This robustness to initialization is illustrated in Tab. 2. When a camera extent of $1\times$ is used, 3DGS achieves a PSNR of 22.72 dB, whereas 3DGS-MCMC achieves a PSNR of 29.64. This result show that the proposed method is robust to the initialization. Also, this substantiates the exploration claim proposed in the paper. - **Exhaustive ablation for all the key design choices:** In Tab. 3, the authors present ablation on the regularizers for Gaussians. Interestingly, in this framework, regularization of the Gaussian parameters improves the performance, whereas it is harmful in the 3DGS framework. Further, noise in the update step update allows for more exploration. This claim is further substantiated in Fig. 4. Finally, when noise is used for all the parameters, overall performance is slightly dropped.

Weaknesses

- The proposed method exhibits robustness to the initialization step. However, is there a reduction in training time between the version using SFM points and the version using random initialization? - **Missing evaluation on Scannet++[A1] dataset:** It is a large-scale dataset for indoor scenes. It will be interesting to see how this method performs on this challenging dataset. It is difficult to perform this experiment in a short duration of time. I leave it to authors to decide if they want to include in their manuscript. - **Training Time:** The proposed method generates very high-quality novel-views. However, this comes with an added training cost. 3DGS-MCMC takes 90 minutes for 1M Gaussians, whereas 3DGS can be optimized within 30 minutes. The authors mention in L467-468 that a CUDA implementation can accelerate the proposed method. [A1] Yeshwanth, C., Liu, Y.C., Nießner, M. and Dai, A., 2023. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12-22).

Questions

- L40-41: "lead to .... waste compute."Can "poor-quality renderings" be substantiated by some examples? Also, can the authors elaborate why it is a wasted compute? InstantSplat[7] reconstructs an unbounded scene with sparse views in under 40 seconds on a commercial GPU. - Does this method accurately represent the surface better than 3DGS? Can we extract a high-quality mesh from 3DGS-MCMC? Will it be better than recent works such as 2D Gaussian Splatting[A2]? [A2] Huang, B., Yu, Z., Chen, A., Geiger, A. and Gao, S., 2024. 2d Gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403.17888.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have discussed the limitations of their method in Appendix D in the supplementary material. As such, this work has no societal impact. However, some downstream applications can have a societal impact. The authors have discussed this in Appendix E.

Summary

The paper presents a simple and effective method to enhance the training of 3D Gaussian Splatting (3DGS). It offers two main contributions. First, it demonstrates that adding carefully designed noise to the Gaussian centers after each gradient step can boost the performance of 3DGS. This encourages more exploration, which is especially helpful when Gaussian centers are randomly initialized. Second, the method performs densification by replacing low opacity Gaussians with clones of Gaussians sampled through multinomial sampling of the "live" ones based on their opacity values. The parameters of the Gaussians are adjusted to minimize the impact on the rendering outcome. The method is tested on both synthetic and real datasets and shows better performance compared to the 3DGS baseline, regardless of whether the Gaussian centers are randomly or "SFM" initialized.

Strengths

- The paper tackles an important problem (optimization, cloning and splitting strategies for placing Gaussians) and introduces relevant concepts and ideas to analyze it. - The paper proposes a simple and easy-to-implement way to improve the optimization of Gaussian positions by encouraging more exploration. - The proposed relocation strategy works effectively and pairs nicely with the L1-regularizer on the Gaussians. - **Evaluation**: The evaluation is performed on various types of scenes (Nerf Synthetic, MipNeRF 360, Tank & Temples, Deep Blending, OMMO) and performance is reported with and without "SFM" initialization. Reporting the average over 3 runs and the corresponding standard deviation is a good practice for the reliability of the results. - **Performance**: The provided results show that the proposed method improves over the 3DGS baseline. More importantly, the method obtains competitive results without initializing the Gaussians with SFM points.

Weaknesses

- **Relevance of MCMC framework**: Adding noise to the parameters [1] or to the gradients [2][3] is a common practice in optimization to escape from saddle points and find local minima. Since the training relies on momentum-based optimizers, I believe the proposed method is more closely related to Perturbed Gradient Descent methods [4][5] or noise injection methods [1] [2] [3] than to MCMC methods. For a proper momentum based update, noise would typically be added to the momentum estimate instead of the parameters. This doesn't make the method any less relevant or novel. **However, it does make me question the relevance of the MCMC framework that the paper is built around. In which part of the analysis or the method is this framework necessary ?** - **The design of the noise term**: One important contribution of the paper is the design of the noise term. It would be great to have more explanations and ablations about these design choices. Why is this particular choice important for the method? Additionally, how does it compare to simpler noise schedules such as the one in [2]? - Concerning the update in equation 9 of the paper, please clarify what assumptions and approximations are needed for this derivation and in which case they are no longer valid. [1] Mark Steijkvers.A recurrent network that performs a context-sensitive prediction task. 1996. [2] Neelakantan et al.,Adding gradient noise improves learning for very deep networks. 2015 [3] Deng et al., How shrinking gradient noise helps the performance of neural networks. 2021. [4] Jin et al., How to Escape Saddle Points Efficiently. 2017. [5] Jin et al., On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points. 2021.

Questions

- Relevance of the MCMC framework (See Weaknesses). - How is the term designed and how does it affect the performance? - What assumptions and approximations are needed for the derivation of equation 9 ?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors adequately addressed the limitations in the Appendix.