4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization

Summary

The research introduces UA-4DGS, a novel approach designed to reconstruct dynamic scenes from monocular videos. UA-4DGS compensates for information loss due to large motion and self-occlusion by incorporating diffusion priors with pixel-wise uncertainty scores. It proposes an uncertainty-aware diffusion-based regularization, which selectively enhances uncertain regions while maintaining certain ones, preventing inconsistencies with training images. It also identifies and addresses the initialization problem in Gaussian Splatting for dynamic scenes, where static scene-focused Structure from Motion (SfM) techniques fail to initialize dynamic regions properly. They propose dynamic region densification to enhance reconstruction performance and memory efficiency by adding extra Gaussian points to dynamic areas.

Strengths

The paper is presented very clearly, well structured, and it is in general easy to follow and understand. UA-4DGS's effectiveness is demonstrated using the DyCheck benchmark with complex object motions. Additionally, the research shows that integrating uncertainty considerations into other NeRF regularization techniques can also improve performance.

Weaknesses

1. Writing Quality: The paper contains many typographical and grammatical errors, as well as notation misuse, e.g. L31, 37, 44, 79, and 124. I recommend a thorough review to correct these mistakes. 2. Some assumptions and claims in the paper are questionable and require further validation. For instance, the basic assumption regarding the definition of certainty ("we assume that Gaussians frequently visible in the training images have low uncertainty, while those seen less often due to motion or occlusion have high uncertainty, as they are reconstructed with lower accuracy") and the claim about under-reconstruction in dynamic regions ("under-reconstruction in dynamic regions negatively impacts the training process, resulting in an excessive number of gaussian points and slowing down inference time") need to be substantiated with additional evidence. 3. Experimental results: The experimental results presented in Figure 3 are disappointing. While the proposed UA-4DGS method seems to outperform the baselines, the results are still unsatisfactory, with significant noise and blurriness. Additionally, the comparison is based on only seven images, which is insufficient. I strongly suggest expanding the testing to include more cases to provide a more comprehensive evaluation of the model's performance.

Questions

1. In L171, why use both L1 and L2 losses at the same time? 2. Could you introduce more details about L176 - "Thus, we cache 200 images every 2000 iterations for training efficiency."?

Rating

3

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

The limitations of this paper are primarily found in the method's assumptions and the experimental demonstrations. Please refer to the weaknesses part for more details.

Summary

This paper proposes an uncertainty-aware regularization technique that uses diffusion priors to improve the reconstruction quality of underfitted areas. and a dynamic region densification technique to address the missing initialization problem on dynamic regions. Experiments verify the proposed techniques.

Strengths

1. The proposed regularization method, which applies uncertainty-aware diffusion priors on unseen views, can reduce inconsistencies with training images. 2. The proposed dynamic region densification can deal with the issue of missing initialization in dynamic regions. 3. The paper is well-written and easy to follow.

Weaknesses

1. The experimental setup is not reasonable. The goal of this paper is to reconstruct dynamic scenes from monocular videos. However, as shown in Sec. 7.2, only one part is relevant to evaluating the proposed method, while the other two involve the generalization of uncertainty-aware regularization. I think the authors should conduct more experiments on the dynamic scene datasets and move the two parts about generalization into the `Appendix`. 2. As shown in L206, the authors claim the proposed method will compare with SC-GS. However, I cannot find such a comparison in Tables and Figures. Moreover, the proposed method should also be compared with Deformable 3D-GS[1]. 3. The `Appendix` should be placed after the `References`. [1] Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., Jin, X.: Deformable 3D Gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR. (2024)

Questions

See weaknesses. More, - According to Table 2, for three components of the proposed method, *Dynamic Densification* plays the most important role (+1.3 mPSNR), $L_{\mathrm{data}}$ is the second important (+0.49 mPSNR), and $L_{\mathrm{UA-diff}}$ is the least important. (+0.26 mPSNR). I would like to know the performance of each component when used individually. - The proposed method seems to be able to change 4D-GS to any other GS-based method. I would like to know the performance of the proposed method when applying it on Deformable 3D-GS or SC-GS.

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

This paper does **not** discuss the limitations and broader impacts.

Summary

This paper tackles the problem of modeling dynamic 3D scenes from monocular videos. To tackle the more challenging dynamic regions in 4D Gaussians, the authors propose to measure the uncertainty and guide those regions with diffusion priors, while keeping certain regions unchanged. In addition, the authors propose to re-initialize the dynamic regions for better performance. Uncertainty of Gaussian points is measured by their visibility in the training set. Re-initialization of dynamic Gaussian is done by exploiting pre-trained depth and optical flow models.

Strengths

1. Complete ablation studies. 2. Improved quantitative results with respect to the baseline (4DGS). 3. Large quantitative improvements for the sparse view case.

Weaknesses

Major 1. Qualitative results are still of low quality (similar to the previous works). 2. Model training depends on several pre-trained models on different tasks, which increases the complexity of the overall method. Minor 1. Main performance improvement comes from the initialization strategy 2. Missing definition of co-visibility mask 3. Check the writing continuity of lines 190 and 191

Questions

Where is the "camera information" in line 188, is it from COLMAP?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors checked the limitation box in the checklist. However, I cannot see the limitations section in the paper nor the appendix.

Thanks for your reply

Thanks for your answers. I appreciate the additional results, but, as other reviewers pointed out, the qualitative performance is below what NeurIPS expects, making me believe this work still needs more refinement.

Thanks for the rebuttal

Dear Authors, I appreciate your efforts in the rebuttal and for providing additional demonstrations. I have carefully reviewed your reply, particularly the examples in the attached PDF. While some cases look very good, most of them appear blurry and noisy in the details. I believe the method still has room for improvement. Thank you.