Effective Rank Analysis and Regularization for Enhanced 3D Gaussian Splatting

Summary

The authors present a new regularization for Gaussian splatting (GS) method that increases the Shannon entropy of the scale parameters (it was named erank as while rendering the scale is presented as a diagonal matrix, and the aforementioned calculation turns into the effective rank), to better reconstruct 3d mesh out of the GS point cloud. The paper argues that a majority of the gaussians in previous methods are close to erank of 1, which ruins the 3d reconstruction because of needle-like gaussians that effectively add noise to the reconstructions. Given that regularization as an add-ons to other methods it improves the 3D reconstruction, as shown in the experiments on DTU dataset. Additionally, it shows in the appendix that the image rendering abilities of the GS is not impaired because of the added regularization.

Strengths

The paper dives into the attributes of the GS and impressively recognizes an issue relating to the GS ability to reconstruct 3d mesh. The authors exhibit an analysis of the amount of 1-ranked gaussians that strengthen their claim, and showed improved results on DTU dataset.

Weaknesses

There are several concerns regarding this paper, first it should include more benchmarks for 3d reconstruction, for example Tanks and Temples dataset which was used as benchmark in many other works in the field, including the works that was cited in the paper. Secondly, there are two works that the authors did not mention [1,2] and should be compared to in the DTU experiment. The first [1] has the same idea as 2DGS and was already published. It might be that their regularization solves part of the issue of the GS that this paper presents. The second [2], does not use the gaussian splatting directly for reconstruction and instead renders stereo images for depth estimate. Thus, it does not share the reconstruction error because of the needle-like gaussians. Comparing both of them will strengthen this paper. Lastly, there is no adequate justification for the erank regularization in comparison to other possibilities, e.g. linear, parabolic, or exponential loss on the size of the smallest magnitude scale. Finally, an ablation study showing why erank-regularization is superior compared to other losses is missing. [1] Dai, P., et al. "High-quality surface reconstruction using gaussian surfels." arXiv preprint arXiv:2404.17774 (2024). [2] Wolf, Y., et al.. "Surface Reconstruction from Gaussian Splatting via Novel Stereo Views." arXiv preprint arXiv:2404.01810 (2024).

Questions

see weaknesses

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

see weaknesses

Summary

3D Gaussian Splatting is a remarkable technique in novel view synthesis. However, it usually degenerates into noodle-like shapes which sometimes bring visual artifacts and inaccurate geometry. This paper analyzes the phenomenon and proposes an effective rank loss to regularize the Gaussians. The technique seems to be a plug-and-play module for 3DGS variants. Extensive experiments suggest the effectiveness.

Strengths

1. The observation and analysis are strong and the solution of using effective rank is simple yet effective. 2. The results show clear improvements over representative works including 3DGS, SuGaR, and 2DGS, which cover a broad range of scene representations. 3. The paper is well-written and easy to follow. I have no issue with the written. Most of the related works are adequately discussed. 4. The technical details are sufficient and I believe an expert in 3DGS can easily implement it.

Weaknesses

1. There is no supplemental video, which is essential for nvs tasks. 2. The improvement of e-rank for Novel view synthesis is limited. It would be beneficial to highlight how this could improve novel view synthesis. 3. Although the paper analyzes that the noodle-like artifacts are due to the training bias, they can also be created to represent high-frequency details. The paper should highlight the PSNR in Figure.1 to give readers a sense that such regularization indeed improves novel view synthesis. 4. It would be beneficial to add the number of Gaussians since enforcing the Gaussian to 2D from 1D should reduce the number of Gaussians in representing a scene.

Questions

1. How the normal is constructed for 3D Gaussian and how it is visualized? 2. What is the effect of densification on geometry reconstruction in 3DGS? In Table 1. Does 3DGS+e indicate with ERANK only or plus densification? It should be clearly ablated where the performance comes from.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Overall, the paper did a good job with minor issues required to clarify. The limitation of tuning the hyperparameter is also discussed and the potential negative effect is reasonable. **minors:** The normal visualization seems not to be normalized witn [0,1]

Summary

This paper performs a statistical analysis on 3DGS for its effective rank distribution of the learned Gaussians. It claims that most of the Gaussian learned are close to rank 1 effectively, giving needle-like artifacts in novel view synthesis and reconstruction. Hence, this paper proposes a regularization loss to discourage low range Gaussians from forming, showing improved reconstruction results.

Strengths

**Motivation** * This paper starts with a clear and strong motivation, where the Gaussian learning with 3DGS creates needle-like artifacts. This reduces both the reconstruction and novel view synthesis quality. **Method** * This paper proposes an intuitive and effective method of regularizing the effective rank of the Gaussians, to regularize on the shape of individual Gaussian. The experiment results also demonstrate the effectiveness of the proposed method on various baselines, with considerable improvement in reconstruction results and minor improvement in novel view synthesis results.

Weaknesses

I do not doubt the motivation and effectiveness of the proposed method. In fact, needle-like artifacts have been a well-known issue for 3DGS in the community. However, I feel reluctant to accept this paper due to the following reasons: 1. Although the phenomenon is well known by many, the root cause of this issue is not clear. This paper tries to explain the cause in its appendix, and gives two hypotheses: a) incorrect dilation observed by MipGaussian, b) Gaussians are more likely to densify along their short axis instead of their long axis. Based on this, I would like to point out the following issues: - This part should be put into the main paper instead of the statistical analysis. The causes of the low-rank Gaussians would be much more valuable than reiterating the phenomenon that many people already know. - This cause analysis needs to be more rigorous. The version provided in the appendix is more of a hypothesis than an analysis. - If it is because of the dilation error, is it already fixed by MipGaussian? What is the rank distribution like for MipGaussian? Can the proposed method improve the MipGaussian performance as well? If yes, then something else is still causing the issue. If not, why shouldn't MipGaussian be used instead of the proposed method? - If it is because of the densification bias, does this problem still occur without densification at all? In fact, I have personally observed many cases of needle-like artifacts that are so long that extend across the entire image. In this case, the rendering loss should be enough to prevent this from happening without anything to do with the densification. But it is not happening, why? Is there something wrong with the rendering loss supervision we have with 3DGS? I am not saying that the densification bias is not a cause, but it requires more evidence and experiment results to be proved. 2. Continuing from the previous point, the method proposed does not address the cause hypothesized. It is not fair to say that regularization methods are just bad, but there have been even simpler regularization methods used by other papers (e.g. PhyGaussian). The most common method is to regularize the ratio between the long axis and the short axis of the 3DGS. It takes only a few lines of code and works reasonably well. This paper does not mention this simpler regularization and does not compare with it as a baseline. However, I think it is very important to justify the more complicated method proposed in this paper, with either rigorous analysis as well as empirical results (preferably both). In summary, I believe that this paper lacks a rigorous analysis of the cause of the low-rank Gaussian formation, and the proposed method lacks novelty and performance comparison against the simple naive regularization used in other papers.

Questions

To summarize my opinions mentioned in the weakness section into questions, it would be: 1. what is really the cause for the low-rank Gaussian formation and how does the proposed method address it? 2. why is the proposed method better than the simple regularization on the ratio of the long/short gaussian axis?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

The paper does not have potential negative societal impact and there is no significant limitation of the method that needs to be addressed other than the ones mentioned in the previous sections.

Summary

The paper identifies a common problem in 3D Gaussian splatting where Gaussians converge to needle-like shapes, with its variance mostly contained in one axis. These needle-shaped Gaussians can cause artifacts and inaccurate surface reconstruction. To quantify this phenomenon, the paper uses the concept of effective rank, a continuous generalization of the rank of a matrix. During scene reconstruction, while 3D gaussians may begin with an effective rank of 3, they tend to converge to an effective rank of 1 after several thousand iterations of training. Based on effective rank, the paper proposes a regularization method to reduce needle-shaped Gaussians. On the DTU dataset, the authors show that 3DGS with rank regularization causes better chamfer distance and PSNR metrics.

Strengths

The paper introduces a method to prevent needle-like Gaussians in 3DGS, an important challenge in view synthesis, and presents a clear way to quantify the problem. The authors propose a new regularization term that is simple and can be used alongside other Gaussian splat variants such as 2DGS and SuGaR. On the DTU dataset, the authors report better chamfer distance and PSNR metrics than prior work. Overall I found the paper very interesting and its claims well justified. Because of the widespread popularity of Gaussian Splatting, the method can be useful to both researchers and 3D designers.

Weaknesses

While the paper introduces a new way to quantify needle-like Gaussians, this is not the first paper to propose a regularization term for them. PhysGaussian [A] introduces an anisotropic regularizer to prevent skinny Gaussians. This regularization term has already been implemented in Nerfstudio’s Splatfacto method [B]. This seems like the most relevant and important comparison to the proposed method, and is completely missing from the paper. This missing comparison is the main reasoning behind my score. In addition, I wish that the paper did a deeper investigation into what causes spikey Gaussians. I found the discussion in Appendix A.4. very fascinating, and think it could be very valuable to the community if elaborated on. Maybe a simple experiment validating the hypothesis could strengthen the paper. Minor Comments: I found that Fig. 1 was hard to parse. I think it would help to include labels for the figure’s images. [A] https://arxiv.org/pdf/2311.12198 [B] https://docs.nerf.studio/nerfology/methods/splat.html

Questions

What happens if you do not regularize erank(G) < 2? Do Gaussians with erank(G) > 2 hurt reconstruction quality? What are x and o in Fig. 8?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes the authors have discussed the limitations and potential societal impacts.

Official Comment by Reviewer 7LDG

Thank you for your response. I appreciate that most of my questions have been addressed. After reading the author rebuttal and considering the other reviews, I’d like to share my thoughts. The proposed method is simple yet effective and can serve as a plug-and-play module for various GS variants, potentially broadening its impact. I also agree with Reviewer w1Sn that there should be more analysis on the root cause of the needle-like artifacts, as this could enhance the paper’s impact. I encourage the authors to include some of this analysis in the main paper. Additionally, it would be beneficial to emphasize that the regularization, while reducing needle-like artifacts, does not compromise the reconstruction of high-frequency details. I also noticed, as pointed out by Reviewer biCn, that some important baselines like PhysGaussian were missing in my initial feedback. I appreciate that the authors have since provided these comparisons. Overall, while some minor revisions are needed, I believe the authors are well-equipped to address these issues, as demonstrated in their rebuttal, including the comments and tables presented. Therefore, I have no concerns with the majority of the work and would like to maintain my original rating.

Thanks

Thank you for your response. I appreciate the new experiments on PhysGaussian, and I see how their method can still cause low-rank Gaussians from forming. Along with the other reviewers, I agree that a deeper investigation into the cause of spiky Gaussians is important for the paper, and I hope that the new analysis is included in a revised version of the paper. Based on the rebuttal, I will change my score.

Comparison with recent results would improve the paper.

The authors provided answers to our concerns regarding the T&T dataset, and the justification for using the entropy like regularization compared to other possibilities. Regarding the recent papers to compare with, such evaluations would definitely improve the paper.