Recently, 3D Gaussian Splatting (3DGS) has become a promising framework for novel view synthesis, offering fast rendering speeds and high fidelity. However, the large number of Gaussians and their associated attributes require effective compression techniques. Existing methods primarily compress neural Gaussians individually and independently, i.e., coding all the neural Gaussians at the same time, with little design for their interactions and spatial dependence. Inspired by the effectiveness of the context model in image compression, we propose the first autoregressive model at the anchor level for 3DGS compression in this work. We divide anchors into different levels and the anchors that are not coded yet can be predicted based on the already coded ones in all the coarser levels, leading to more accurate modeling and higher coding efficiency. To further improve the efficiency of entropy coding, e.g., to code the coarsest level with no already coded anchors, we propose to introduce a low-dimensional quantized feature as the hyperprior for each anchor, which can be effectively compressed. Our work pioneers the context model in the anchor level for 3DGS representation, yielding an impressive size reduction of over 100 times compared to vanilla 3DGS and 15 times compared to the most recent state-of-the-art work Scaffold-GS, while achieving comparable or even higher rendering quality.
Paper
Similar papers
Peer review
Summary
This paper presents Context-GS, a method designed to reduce the memory overhead of 3D Gaussian Splatting (3DGS). Inspired by context modeling in image compression, the authors introduce a similar concept into Scaffold-GS, which uses anchors to predict 3D Gaussian distributions. The method encodes anchors at a coarse-to-fine level, significantly enhancing storage efficiency. Experimental results on real-world datasets demonstrate that the proposed method achieves a high compression ratio while maintaining comparable fidelity.
Strengths
1.The method is novel, integrating the concept of context modeling from the image compression domain. 2.The results are strong. It achieves significantly better compression while retaining high rendering quality, comparable to the original Scaffold-GS and clearly superior to other counterparts. The evaluation is comprehensive, and the paper conducts thorough ablation studies that fairly analyze the actual compactness. 3.The overall presentation is easy to follow.
Weaknesses
1. The main concern is that the proposed main components have a very minor effect on performance. As shown in Table 2, the primary contribution to compression is adopted from Compact-3DGS, while the proposed "HP" and "CM" components reduce the memory by only up to 4 MB. 2. The proposed method appears complicated and specifically customized to Scaffold-GS, which limits its extendability. One of the most appealing features of 3DGS is its compatibility with many graphics engines. However, the use of neural networks and view-dependent Gaussian prediction undermines this advantage.
Questions
The teaser figure is hard to understand and does not effectively convey the main idea. Upon reading the caption, my initial impression was that (b), (c), and (d) are from the proposed Context-GS. However, the caption stating "(c) verifies the spatial redundancy" and the main text at L48 stating "spatial dependency has been significantly reduced" confused me about what exactly is being reduced. Is one of the figures taken from Scaffold-GS? It would be more effective to first provide a reference scene so readers can understand what is actually being reduced. Additionally, the points are too small and chaotic, making it difficult to convey the ideas effectively. Minors: - L21: "quired" should be "queried." - L29: "Neural Gaussian"? It is unclear why this is referred to as "Neural Gaussian" instead of "3D Gaussian." 3D Gaussian typically does not involve neural networks. The paper might define "neural" as meaning differentiable, but this could confuse some readers into thinking it is related to neural networks.
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
The paper discusses limitations in encoding and decoding costs, but the primary limitation appears to be highlighted in the weaknesses section. The proposed method also seems difficult to apply to more generic Gaussian splatting techniques. One of the authors' main motivations is that "these papers mainly focus on improving the efficiency of a single neural Gaussian and neglect the spatial redundancy among neighboring neural Gaussians." However, this motivation does not resonate with me, as the ablation study does not indicate significant improvements from considering spatial redundancy.
Thanks for your reply! Attached more results on different backbones
Dear Reviewer 8ybR, Thank you for your valuable feedback, which has significantly improved our submission. Regarding your concerns about customizing to Scaffold-GS, we are conducting experiments on various backbones, including vanilla 3DGS. Preliminary results, attached below, **show huge improvements** over **the most recent SOTA** methods on **different backbones**. We will provide more detailed results soon. Best regards, Authors --- **Table R1**: Performance of the proposed method on the vanilla 3DGS backbone. | Measured on Bilbao (Bungernerf) | PSNR (dB) | SSIM | Size (MB) | | --- | --- | --- | --- | | Compressed3D (CVPR'24) | 25.81 | 0.8403 | 49.32 | | Ours | *27.77* | *0.8845* | **13.39** | **Table R2**: Performance of the proposed method on the Compact-3DGS backbone (CVPR'24 Oral). Unlike the vanilla 3DGS, Compact-3DGS uses a small MLP to predict the colors of 3D Gaussians. | Measured on Bilbao (Bungernerf) | PSNR (dB) | SSIM | Size (MB) | Decoding time (s) | | --- | --- | --- | --- | --- | | Compact-3DGS (CVPR'24 Oral) | 25.12 | 0.8581 | 51.15 | 613 | | Ours (low bpp) | *25.58* | *0.8613* | **13.36** | **16.12** | | Ours (high bpp) | **26.28** | **0.8668** | *14.85* | *21.52* | _____ Dear Reviewer 8ybR, Attached are the updated full results on the BungeeNeRF dataset, demonstrating the generalizability of our method on different backbones. On different backbones, we **all achieve huge improvements**. **Table R1**: The performance of the proposed method on the vanilla 3DGS backbone. | Method | Backbone | PSNR (dB)| Size (MB) | | ---- | --- | --- | --- | | 3DGS | `3DGS` | 24.87 | 1616 | | Compressed3D (CVPR'24) | `3DGS` | 24.13 | 55.79 | | Ours | `3DGS` | **25.06** | **14.36** | **Table R2**: The performance of the proposed method on the backbone used by Compact3DGS. | Method | Backbone | PSNR | Size | | ---- | --- | --- | --- | | Compact3DGS (CVPR'24 Oral) | `3DGS+tiny color mlp` | 23.36 | 82.60| | Ours | `3DGS+tiny color mlp` | **25.83** | **13.93** |
Summary
The paper aims to compress Gaussian Splatting-based neural rendering representations. To achieve higher representation performance with a smaller size, the paper proposes hierarchical anchors, where coarser level anchors work as context to achieve a higher compression rate. Additionally, the paper introduces hyperprior coding for improved performance.
Strengths
**Novelty** The main idea is novel within the 3DGS framework. **Performance** The performance improvement is significant. **Experiments** The ablation study has been conducted thoroughly, making it easy to understand the contribution of each part in improving the rate-distortion curve.
Weaknesses
**Mathematical Notations (minor)** The meaning of ^ is unclear. In Eq. 5, $\hat{x}$ denotes that $x$ has been quantized. However, in Eq. 6 and line 156, $\hat{V}$ does not seem to refer to a quantized set of anchors.
Questions
Line 167: Is “senses” a typo of “scenes”? Lines 201-202: It is mentioned that the number of levels was set to three. Did you run experiments with different numbers of levels, and how did they affect the performance? Tab. 5 (appendix): Does “coding” refer to the hyperprior coding used in the main paper? If so, could you explain why “w/o encoding anchors” results in similar or smaller sizes compared to “w/ encoding anchors”?
Rating
7
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Limitations
The paper provides limitations of the proposed method.
Thanks for your reply!
Dear Reviewer gvyt, **We sincerely thank you for your support and recognition of our work.** Regarding the concerns raised by other reviewers about the dependency on Scaffold-GS, we are pleased to share our new results demonstrating the effectiveness of our method on other backbones, such as vanilla 3DGS. By utilizing our proposed end-to-end entropy optimization and context model, we achieved impressive performance on these backbones, as shown in the attached tables. We will release the models based on both Scaffold-GS and vanilla 3DGS upon acceptance. Thank you again for your time and valuable insights. Sincerely, Authors ___ The results of the following tables are measured on BungeeNeRF dataset. **Table R1**: The performance of the proposed method on the vanilla 3DGS backbone. | Method | Backbone | PSNR (dB)| Size (MB) | | ---- | --- | --- | --- | | 3DGS | `3DGS` | 24.87 | 1616 | | Compressed3D (CVPR'24) | `3DGS` | 24.13 | 55.79 | | Ours | `3DGS` | **25.06** | **14.36** | **Table R2**: The performance of the proposed method on the backbone used by Compact3DGS. | Method | Backbone | PSNR | Size | | ---- | --- | --- | --- | | Compact3DGS (CVPR'24 Oral) | `3DGS+tiny color mlp` | 23.36 | 82.60| | Ours | `3DGS+tiny color mlp` | **25.83** | **13.93** |
Summary
In this paper, the authors propose ContextGS to reduce spatial redundancy among anchors using an autoregressive model. Specifically, the authors divide anchors into three levels, performing entropy coding from the top (coarse) level to the bottom (fine) level. Anchors from coarser levels are utilized as context to assist in the entropy coding of anchors at finer levels. Experimental results show that the proposed method achieves a size reduction of 15 times compared to Scaffold-GS and 100 times compared to 3DGS.
Strengths
1. The authors have explored the correlation between anchors and, for the first time, introduce autoregressive entropy models for spatial prediction of anchors. 2. The entire pipeline is trained end-to-end with joint rate-distortion optimization, supporting multiple bitrates by adjusting λ.
Weaknesses
1. The size reduction brought by entropy model is limited. The authors claim that the proposed method can achieve a size reduction of 15 times compared to Scaffold-GS. However, Table 2 in the ablation study indicates that the size reduction primarily stems from entropy coding and masking loss (from 183.0 MB to 18.67 MB), while the main contribution of the paper, the hyperprior and anchor level context model, results in approximately a 25% bitrate reduction (from 18.67 MB to 14.00 MB), which is relatively modest. 2. The description of the anchor division method is somewhat confusing, and the equations are difficult to understand. 3. The authors design a learnable hyperprior vector for each anchor as an additional prior. However, this approach may introduce additional spatial redundancies between anchors.
Questions
1. It’s better for the authors to make furhter clarification regarding how anchors are divided into different levels. Are anchors that have the same position after quantization selected for a higher level (according to lines 154-155)? But Figure 2(b) contradicts this, as v_3^{k-1}, which is far from the quantization center, is selected as the anchor for upper level. 2. I wonder about the effects of the learnable hyperprior feature z_i and would like to see the performance without it.
Rating
5
Confidence
4
Soundness
3
Presentation
1
Contribution
2
Limitations
This paper seems to lack discussions of the limitations or the potential negative societal impact.
New results to alleivate your potential concerns and looking for your reply
Dear Reviewer otkA, Thank you very much for your insightful review and valuable suggestions. We have carefully considered your feedback and addressed your comments thoroughly in our rebuttal. `We believe the clarifications and improvements we've made effectively address the concerns you raised`. **If you find that our responses have satisfactorily resolved the issues, could you please consider adjusting your rating accordingly?** **If you still have concerns, we are more than happy to provide any additional information or discuss further if needed.** ---- ### Additional Highlight on Performance if you still have concerns We want to highlight that **a strong end-to-end entropy baseline is also our contribution** since there is no standard implementation and **different papers have different entropy models**, e.g., Huffman coding in Compact3DGS, and entropy and run-length encoding in Compressed3D. Our proposed entropy framework is **much stronger** and can be applied to different 3DGS backbones, **not only limited to ScaffoldGS**. ### More results on different 3DGS backbones We further conducted experiments on the vanilla 3DGS, and the modified 3DGS backbone used in Compact3DGS (CVPR'24). The results on the challenging benchmark `BungeeNeRF` are as follows **Table R1**: The performance of the proposed method on the vanilla 3DGS backbone. | Method | Backbone | PSNR (dB)| Size (MB) | | ---- | --- | --- | --- | | 3DGS | `3DGS` | 24.87 | 1616 | | Compressed3D (CVPR'24) | `3DGS` | 24.13 | 55.79 | | Ours | `3DGS` | **25.06** | **14.36** | As shown in Table R1, under the same backbone, we achieve **0.9dB** PSNR improvements with a size reduction of **~3x** compared with the most recent SOTA on the same backbone. We use **less than 1%** bitrate and achieve better PSNR than the vanilla 3DGS. **Table R2**: The performance of the proposed method on the backbone used by Compact3DGS. | Method | Backbone | PSNR | Size | | ---- | --- | --- | --- | | Compact3DGS (CVPR'24 Oral) | `3DGS+tiny color mlp` | 23.36 | 82.60| | Ours | `3DGS+tiny color mlp` | **25.83** | **13.93** | Compact3DGS uses a slightly modified backbone from the vanilla 3DGS, incorporating a small MLP for 3D Gaussian color prediction. As shown in Table R2, our method significantly outperforms the latest SOTA method. Compared with Compact3DGS (CVPR'24 Oral), we achieves **2.5dB** improvements with a **~5x** size reduction. ___ Because **half of the discussion period has passed**, please feel free to raise any concerns so that we can better address any potential misunderstandings. Thank you again for your time and thoughtful consideration. Best regards, Authors
thanks for the repsonse
Thanks for the detail response and clarification, part of the concerns have been addressed, I have increased the score.
Thanks for your update!
Dear Reviewer otkA, Thanks for your update and support! We appreciate the time and effort you have dedicated to reviewing our manuscript. Best regards, Authors
Summary
This paper proposes ContextGS, a compact 3D Gaussian Splatting (3DGS) framework that requires only a minimal amount of storage size while demonstrating high rendering quality. Upon the neural Gaussian-based 3DGS framework Scaffold-GS, the authors construct a multi-level anchor structure to reduce the spatial redundancy and adopt context modeling proposed in image compression tasks. Consequently, it achieves a 15$\times$ compression ratio compared to Scaffold-GS and a 100$\times$ compression ratio compared to 3DGS. Despite the minimal storage size, it outperforms the existing compact 3DGS approaches in rendering quality.
Strengths
- The quantitative evaluation shows that this method outperforms in rendering quality compared to existing compact 3DGS frameworks, including Scaffold-GS and HAC while requiring minimal storage usage. - The proposed entropy modeling of neural Gaussian features further reduces the storage size with minor degradation of rendering quality.
Weaknesses
Although it achieves high compression performance, there are several critical points to concern. - There are limited technical contributions to their method. The concept of multi-level (or multi-resolution) anchor structure has already been proposed in Scaffold-GS and context modeling of neural Gaussians has been introduced in HAC. There are only minor technical contributions compared to the previous work. - It requires noticeable encoding and decoding time resulting in a bottleneck in the practical application of 3DGS. Especially, the encoding/decoding time of 'w/ APC' requires more than 40 sec for a single scene. Also, Scaffold-GS needs a per-view decoding process for view-adaptive rendering, resulting in slower rendering speed than 3DGS.
Questions
- What does the ‘mask’ of Scaffold-GS in Table 4 mean? To my knowledge, there is no masking parameter in Scaffold-GS. I am confused that the masking strategy of Lee et al. is applied to Scaffold-GS. - In L271-L272, the authors have argued that this method has fewer anchors due to the use of masking loss. Please clarify this part. - As they have mentioned in L392-394, there exists additional computational costs for entropy minimization during training. I wonder about the exact training time for this method compared to Scaffold-GS and HAC. - Also, the exact quantitative evaluation of rendering speed is needed to support the faster rendering time of ContextGS compared to Scaffold-GS as described in L271-L272. Please provide a comparison of rendering speed to prove the argument. - The multi-level partitioning strategy is similar to the multi-resolution structure of Scaffold-GS. Please clarify the difference between the multi-level strategy of this paper and Scaffold-GS. - Does the encoding anchor in Table 5 denote that APC in Table 4?
Rating
4
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
The additional computational costs cannot be alleviated for achieving a small storage size. Also, the format of this representation does not fit to standard 3DGS, thus the existing applications such as real-time interactive renderers cannot be used. Therefore, it has disadvantages to be used as a practical 3DGS representation.
Additional results on different 3DGS backbones
Dear Reviewer fG6F, Thank you very much for your insightful review and valuable suggestions. We have carefully considered your feedback and addressed your comments thoroughly in our rebuttal. We believe the clarifications and improvements we've made effectively address the concerns you raised. **If you find that our responses have satisfactorily resolved the issues, could you please consider adjusting your rating accordingly?** **If you still have concerns, we are more than happy to provide any additional information or discuss further if needed.** To further alleviate your concerns in the limitation part, we evaluated the proposed method on both the Compact-3DGS and vanilla 3DGS backbones. The results are presented in the following tables: | Measured on Bilbao (Bungernerf) | PSNR (dB) | SSIM | Size (MB) | | --- | --- | --- | --- | | Compressed3D (CVPR'24) | 25.81 | 0.8403 | 49.32 | | Ours | *27.77* | *0.8845* | **13.39** | **Table R2**: Performance of the proposed method on the Compact-3DGS backbone (CVPR'24 Oral). Unlike the vanilla 3DGS, Compact-3DGS uses a small MLP to predict the colors of 3D Gaussians. | Measured on Bilbao (Bungernerf) | PSNR (dB) | SSIM | Size (MB) | Decoding time (s) | | --- | --- | --- | --- | --- | | Compact-3DGS (CVPR'24 Oral) | 25.12 | 0.8581 | 51.15 | 613 | | Ours (low bpp) | *25.58* | *0.8613* | **13.36** | **16.12** | | Ours (high bpp) | **26.28** | **0.8668** | *14.85* | *21.52* | The results **show significant improvements** over **the most recent SOTA methods** on **different backbones**, strongly demonstrating **the significance of the proposed method as a general framework**. Thank you again for your time and thoughtful consideration. Best regards, Authors
Thanks for your reply. More repsonses are provided.
Dear Reviewer fG6F, Thanks for your response and we are happy to provide additional information for your concerns. ### "The key idea to compress Gaussians (entropy minimization w/ hyper-prior for anchor-based Gaussians) has been explored in the previous approaches." 1. As shown in the title of our paper, the main contribution we claimed is that we are **the first to apply the context model to the 3DGS**. 2. As shown in our previous comment, our method is **not limited to anchor-based Gaussians**. On **vanilla 3DGS**, we can **achieve 0.93dB improvements with a ~3x size reduction** compared with the CVPR'24 Oral work. 3. We have shown **significant improvements** of the proposed context model. On the proposed strong entropy baseline (even stronger than the SOTA of ECCV'24), a \~25% improvement is huge. (For your reference, the **accumulated improvements** on recent entropy-based compression is only **\~11% over 3 years**.) 4. **For the two papers you mentioned in the initial reviewer**: The ScaffoldGS does not introduce hyperprior since it does not use entropy coding. HAC is a concurrent work and has significant differences in the hyper-prior design. - Specifically, HAC was submitted to Arxiv on **March 21st** (visible even later) and was accepted on **July 2nd**. Our work submitted the abstract to NeurIPS on **May 15th**. This is a **concurrent work** according to the NeurIPS guidelines. - > "Papers appearing **less than two months** before the submission deadline are generally **considered concurrent** to NeurIPS submissions. Authors are **not expected to compare to work that appeared only a month or two before the deadline**." from the **NeurIPS 2024 guideline**. - Even though HAC is a **concurrent** work and is only an **Arxiv** paper at our submission, we still included it for comparison for completeness. We have significant differences in both **motivation** and **model design**, leading to a significantly better performance. 5. Could you please **provide references for the papers that share the same core idea** besides HAC? As far as we know, there is no previous work that uses a similar idea with us. 6. Our novelty is **acknowledged by all other reviewers**, e.g., `"for the first time, introduce autoregressive entropy models for spatial prediction of anchors."` by Reviewer otkA, `"The main idea is novel within the 3DGS framework."` by Reviewer gvyt, and `"The method is novel, integrating the concept of context modeling from the image compression domain."` by Reviewer 8ybR. ### Additional computational costs 1. With significant performance improvement, **slightly increased** training overhead **shall not be a reason to reject a paper**. Many papers even do not report their training time, e.g., [a] accepted by ECCV'24. > [a] End-to-End Rate-Distortion Optimized 3D Gaussian Representation, ECCV'24 2. Our training speed is in fact similar to Compact3DGS (CVPR'24), i.e., around \~1 hour for a city-level scene. However, as shown in the comment, we can **achieve 0.46dB improvements with a ~3x size reduction** compared with it. **Table R1: The performance of the proposed method on the same backbone with Compact-3DGS.** | Measured on Bilbao (Bungernerf) | PSNR (dB) | SSIM | Size (MB) | Decoding time (s) | | --- | --- | --- | --- | --- | | Compact-3DGS (CVPR'24 Oral) | 25.12 | 0.8581 | 51.15 | 613 | | Ours (low bpp) | *25.58* | *0.8613* | **13.36** | **16.12** | ### "Encoding time has not been addressed" Encoding time **is included in the training time** since our model directly outputs the bitstreams. If take it out from the pipeline, our encoding time is ~20s for a large-scale city-level dataset. If needed, we will report the detailed encoding time later for comparisons. ### Decoding time **As far as we know, almost all the SOTA works for 3DGS compression dependent on coding techniques. Could you provide reference papers that achieve SOTA performance w/o coding techniques?** Besides, the results in our rebuttal demonstrate that our decoding time is faster than other SOTA works. ____ If you find that our new responses have resolved your concerns, could you please consider adjusting your rating accordingly? **Thank you again for your time and please do not hesitate to reply to us if you still have any concerns.** Best regards, Authors
We can achieve much better PSNR using less training time compared with HAC
While we still hold the opinion that **slightly increased training overhead shall not be a reason to reject a paper**, to further alleviate your concern regarding the training time, we do some preliminary exploration of reducing the number of total training iterations. The results are shown in Table R1. We can achieve **much better performance than HAC using less training time**. Even using a similar time, we can achieve similar performance with the ScaffoldGS **Table R1**: The performance, size, and training time of different methods on Rome/BungeeNerf. | | PSNR (dB) | Size (MB) | Trainining time | | --- | ---| --- | --- | ScaffoldGS | 26.25 | 184.34 | ~25 mins | HAC (ECCV'24, concurrent work) | 25.68 |19.30| ~40 mins | Ours (20k iterations) | 26.35 | 14.56 | ~35mins | Ours (15k iterations) | 26.10 | 15.99 | ~25mins | The encoding time (already included in our training time) is ~15s, as a comparison Compact3DGS uses ~50s and HAC uses ~32s. _____ **We believe the clarifications we have made can effectively address all of the concerns you raised**. If our responses have resolved your concerns, could you please consider adjusting your rating accordingly? If not, as the discussion period is coming to an end, could you let us know of any remaining concerns? Thank you for your time. Best regards, Authors
Most questions are addressed.
Most of my concerns have been addressed. However, I'm uncertain whether a 4MB increase represents a significant improvement, given that the original Gaussian data is typically several hundred MB. Additionally, the proposed method appears to heavily rely on the concept of anchors, which may make it somewhat customized to Scaffold-GS. I am still unclear on how this method can be applied to 3DGS. Nevertheless, I would like to raise my score.
I appreciate the authors' rebuttal. While I share some concerns raised by other reviewers, such as the dependency on Scaffold-GS (even though the authors provided outperforming results, this only addresses half of the proposed method, as one of the two main methods was Scaffold (anchor)-dependent), I believe the paper's strengths, particularly in performance, outweigh these concerns and weaknesses.
I appreciate the authors’ efforts during the rebuttal period. I am pleased that your responses have addressed most of my concerns. Despite the rebuttal, I have decided to maintain my rating due to the lack of technical novelty and evaluations. The key idea to compress Gaussians (entropy minimization w/ hyper-prior for anchor-based Gaussians), has been explored in the previous approaches. Moreover, the proposed multi-resolution strategy needs more evaluations to show its effectiveness. The limitations for the additional computation costs still remain. The longer training time (~60min) is clearly large, not similar, compared to previous approaches (Scaffold-GS: ~25min / HAC: ~40min). Also, the decoding time is much larger than previous methods that do not use entropy coding. Moreover, the encoding time has not been addressed in the rebuttal.
Decision
Accept (poster)