LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

Summary

LightGaussian introduces a three-stage technique to efficiently reduce the number of Gaussian primitives. In the first stage, redundant Gaussian primitives are pruned based on global significance, rather than opacity. The second stage involves SH distillation, which utilizes data augmentation from synthesized pseudo views. Finally, the third stage employs Vector Quantization of the SH coefficients. By incorporating these three approaches, LightGaussian achieves a remarkable 15x compression while maintaining competitive rendering quality.

Strengths

1. LightGaussian introduces a novel pruning strategy based on global significance, along with SH Distillation and Vector Quantization techniques, to effectively reduce both primitive redundancy and feature redundancy. 2. LightGaussian functions as a plugin and can be utilized in any GS representation model. 3. The authors conducted a comprehensive ablation study to demonstrate the effectiveness of the proposed method. 4. This method efficiently reduces the number of Gaussian primitives, leading to a significant improvement in rendering speed. 5. The paper is well-written and easy to follow.

Weaknesses

1. **Lack of necessary experimental analysis.** Although the authors compare LightGaussian with Compressed 3D-GS and Compact 3D-GS in Table 1, they fail to provide any analysis in the experimental results section, which is quite peculiar. Moreover, most of the experimental analyses focus on NeRF, which seems unnecessary. Furthermore, in Table 1, it is difficult to distinguish the visual quality performance and compressed capacity between LightGaussian and Compressed 3D-GS, and, strangely, there is no qualitative number provided for the FPS of Compressed 3D-GS. 2. **Lack of novelty.** This method appears to be a post-processing approach of 3D-GS and lacks essential characteristics, unlike Scaffold-GS [1], which proposes a hierarchical 3D Gaussian representation and only stores the information of anchors. It would be beneficial for the authors to compare LightGaussian with Scaffold-GS and its subsequent work, HAC [2], in terms of both experimental results and analysis. 3. **Poor layout formatting.** In Figure 7, the author incorrectly typeset the paper, leading to a compromised reading experience. By the way, the comparison in Figure 6 is not clearly distinguishable, as the eight images appear to be nearly identical. [1] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [2] HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression

Questions

1. Please provide a more detailed analysis of compression GS methods, such as Compressed 3D-GS and Compact 3D-GS, in the experimental section. In addition to comparing FPS, it would be valuable to include a comparison of training times as well. 2. It would greatly enhance the paper if the authors compare LightGaussian with Scaffold-GS and HAC in the experimental section for a more comprehensive evaluation.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discuss the limitation in Section 5.

Summary

The paper focuses on compressing 3D Gaussian Splatting (3D-GS) models by mainly focusing on reducing the number of Gaussians and compressing the feature size of Gaussians. With three key steps: gaussian pruning and recovery, spherical harmonics distillation, and vector quantization, the paper achieves ~15x reduction in disk usage and ~2x faster rendering speeds with minimal loss in rendering quality. The effectiveness is verified on Mip-NeRF 360, Tank and Temple, and NeRF synthetic datasets.

Strengths

* Compacting the storage cost and rendering speed of 3DGS with minimal loss of rendering quality is an interesting task and has a lot of applications such as rendering on VR devices. * The paper proposed a heuristic metric to evaluate the importance of each Gaussian, to help identify the less important Gaussian which are then pruned out to reduce the storage cost and accelerate the rendering speed. * The paper used knowledge distillation after reducing the dimension of the spherical harmonic features to recover the rendering quality. * The paper did detailed ablation studies on the three key steps and the sub-steps, plus different design choices and hyperparameters, which clearly shows the effectiveness of each components.

Weaknesses

* The proposed vector quantization step lacks novelty, for example both [34] and [36] used vector quantization to compress Gaussian attributes, with minor differences in the method. * In comparison to the prior 3DGS compression works, the improvement is limited. As shown in Table 1, the improvement on the Mip-NeRF 360 dataset is not clear compared to [34], and the results are worse than [34] on the Tank and Temple dataset in terms of both rendering quality and model size. Besides, why the rendering speeds of [34] are not available? * The method requires at least two finetuning steps (Gaussian co-adaptation and VQ finetuning) for the compression, beyond the pre-trained 3DGS model, and each finetuning step requires the same iterations as pre-training. Prior work for example [36] can compress 3DGS in an end-to-end manner, and achieve similar performance. How long does the method take to compress one 3DGS model, and in comparison with the 3DGS compression baselines? In conclusion, the paper is not significantly better than the 3DGS compression baselines. The proposed two novel compression methods such as the Gaussian pruning by a heuristic metric and the knowledge distillation do not show considerable improvement over the baselines in terms of rendering quality, rendering speed, and storage cost.

Questions

Please address the issues I pointed out in the 'weakness' section. Furthermore, I would appreciate clarification on the following questions in the authors' rebuttal. * Can authors provide the rendering speeds of [34] which are missing in Table 1? * In Table 2 are the reported FPSs for training (finetuning) or testing? If for testing why the FPS changes after some steps that should only affect the training procedure, for example from step [4] to [5]. Are they just noises? * In the experiments do you use a universal pruning ratio in the Gaussian pruning step, or is it scene-specific? * (Minor) In Algorithm 1 line 12 why use G' rather than G as the teacher? As the rendering quality of G' is slightly worse than G. Is it because G' is faster to render?

Rating

6

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors provide a discussion of limitations in the paper.

Summary

In this paper, the authors delivered a compact 3D Gaussian representation, i.e., LightGaussian for novel view synthesis. There are three technical contributions. Firstly, the authors present Gaussian Pruning and Recovery that measure the significance of each Gaussian to the view quality and then prune 3D Gaussian that has minimal impact on the visual quality. Secondly, the authors propose Spherical Harmonics (SH) distillation that condenses the information from higher-order coefficients to a more compact form. Thirdly, a Vector Quantization step is presented to reduce the redundancy in spatial and lighting representations.

Strengths

i) The technology in this paper is very comprehensive, including the three most mainstream methods of compressing deep learning models, i.e., pruning, distillation, and quantization. Thus, this paper has great engineering significance and can inspire subsequent works of compressing 3DGS from three points of view. ii) The idea of Gaussian Pruning & Recovery and SH distillation is novel. Computing the score for each 3D Gaussian to represent its significance is fancy and reasonable. The insight of designing Eq. (3) as the score function is interesting. As for the distillation, few people pay attention to the learnable numbers, especially the SH coefficients. Distilling the high-order coefficients with pseudo views also provides a new idea of how to reduce the attributes / learnable parameters of 3D Gaussian. iii) The proposed LightGaussian achieves a better balance between the reconstruction quality, storage usage, and inference speed. To be specific, in table 1, LightGaussian yields improvements of 0.2 dB in PSNR, 81 fps in inference speed, and 6 MB in storage memory than the state-of-the-art method compact 3DGS on the Mip-NeRF 360 datasets. Meanwhile, LightGaussian achieves faster speed and small model size on the Tank and Template datasets with only a 0.2 dB drop. iv) The experiments are sufficient. The results in Table 2 clearly demonstrates the effect of each component. The experiments in Table 3, 4, and 5 study the variants of the three compressing techniques. The authors consider very comprehensive situations in these ablation study experiments. v) The static webpage and video demo in the supplementary are very attractive and Exquisite, which save a lot of time for readers to understand what the authors did in this work.

Weaknesses

i) The story line is not coherent and the motivation may need more explanation. In particular, the three technical contributions, i.e., Gaussian Pruning and Recovery, SH Distillation, and Vector Quantization seem like three independent works. Combining them together without clear motivation make this paper like a technical report. Although I appreciate the work the authors have done, it would be better to improve the writing by establishing a coherent story line and adding clear motivation for the proposed methods. ii) It is better to add discussion and fair comparison with previous 3DGS compressing works such as compressed 3D-GS [34] and compact 3D-GS [36]. For example, what is the difference between the compress techniques? And what about the performance with the same baseline model? iii) Some technical details may require more explanation, for example, why designing Eq. (3) like this as the contribution score of each 3DGS? It is interesting to know the process and key insight of the designing process. How you think about this question? iv) For the main results in table 1, why the Compressed 3D-GS does not have the inference speed? And why LightGaussian is better than SOTA on the Mip-NeRF 360 dataset and performs worse on the Tank and Template datasets? It is better to add analysis and discussion here to explain this performance gap.

Questions

I have a question for the presented SH distillation step Some existing works just use an view-independent component (also low-order SH coefficients) to represent the color of 3D Gaussian [1]. Did you ever try this method? It is interesting to have a discussion between the SH distillation with this option. [1] Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis. In ECCV 2024.

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

Yes, the authors have analyzed the limitations and broader impact of the method in section 4.4 of the main paper.

Summary

This manuscript presents a pipeline to drastically reduce the size of pretrained 3D Gaussian splatting models in a way that preserves novel view image fidelity and increases rendering speed. This pipeline consists of three parts: 1) pruning based on an introduced global significance score followed by fine-tuning, 2) reducing the spherical harmonic dimension of the Gaussians and using knowledge distillation to recover the lost specular information, 3) vector quantization of the spherical harmonics of gaussians below a certain global significance score threshold. The global significance score per Gaussian is accumulated over all pixel rays that that intersect that Gaussian and is the sum of the opacity of the Gaussian multiplied by a normalized volume of the Gaussian. Experimental results show a marked decrease in the model size and increase in rendering speeds while image quality is only slightly degraded.

Strengths

S1: This manuscript is well written and easy to follow. Details are provided that should allow for the ability to replicate experiments. S2: The current size of 3D-GS models makes them unusable in low resource settings — such as VR/AR — and compressing these models will open up many new advancements in downstream applications. The proposed pipeline demonstrates great performance on real world scenes and will be a benefit to researchers working in resource constrained settings.

Weaknesses

**Main weaknesses:** W1: I’m surprised that contribution per pixel ray is sufficient for an effective the global significance score. It seems like a per Gaussian score would need to account for both the Gaussian’s contribution to the input image pixels AND the location of those pixels in 3D space. A Gaussian observed by multiple cameras should hold more importance as it will be more likely to faithfully reconstruct the 3D scene than a Gaussian only observed by a single camera. W2: I’m also surprised that the global significance score doesn’t need to account for the exponential drop in contribution to the pixel color due to alpha blending from Gaussians in front of the Gaussian whose score is being computed. W3: Does the global significance score get updated for all Gaussians along a pixel’s ray or is it updated for only those Gaussians that contribute to the pixel color? I suspect it is the latter, but this should be clearly stated in the manuscript. W4: It’s surprising to me that recovering the higher degree spherical harmonic representation in the lower spherical harmonic degrees via knowledge distillation and sampled views is needed to maintain high image quality after spherical harmonic reduction. I’d expect that recovering the color via fine-tuning in the just the training views should work quite well and knowledge distillation and sampled views should be ablated against this. **Cosmetic issues:** Eq. 4: Volume should always be positive, so there shouldn’t be a need to take the max of normalized volume and 0. Line 200: “network” should be “model”, 3D-GS models aren’t networks.

Questions

I’m rating this manuscript as “Accept”. I encourage the authors to engage with the weaknesses I’ve listed and will consider raising my score if they are addressed in the rebuttal.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

This manuscript has adequately covered the limitations of the proposed pipeline.

Response to the author rebuttal

Thanks for your reply and effort. All my concerns have been addressed. I keep my original score.