QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos

Summary

This paper introduces a novel technique, Quantized Efficient Encoding (QUEEN), to achieve streamable free-viewpoint videos. Unlike methods that directly optimize per-frame 3D-GS given multi-view videos, QUEEN learns the residuals of Gaussian attributes between continuous frames by decoding learnable latent embeddings for each 3D Gaussian. Additionally, a quantized decoder technique has been adopted to significantly reduce the size of latent embeddings (except for the embedding of 3D positions), with only a slight or no decrease in rendering performance. As the authors observe that position residuals are sensitive to compression, they introduce a differentiable L0 sparsity penalty (hard concrete gate with learnable parameters) instead of quantizing the position embeddings. Finally, to improve training efficiency and reduce storage requirements, the values of the gates for each frame are initialized using the 2D viewspace Gaussian gradient difference between neighboring frames. Extensive experiments have shown that QUEEN achieves the best training time, model size, and rendering performance compared to state-of-the-art online free-viewpoint video methods.

Strengths

The paper introduces a novel framework, Quantized and Efficient Encoding (QUEEN), which leverages 3D Gaussian Splatting (3D-GS) for streamable free-viewpoint video (FVV). The combination of 3D Gaussian and the quantized optimization is both novel and original. A comprehensive performance analysis of quantization and the sparsity loss has been conducted to validate this technique's design choices and effectiveness. This work provides several key insights for compressing a sequence of 3D Gaussians, achieving the best trade-off between quality, storage, and speed through quantization and sparsification. Compared to state-of-the-art offline and online methods, QUEEN significantly outperforms in training speed, storage efficiency, and rendering speed, while its rendering quality remains comparable to offline methods.

Weaknesses

1. Section 3.2 is difficult to follow due to the absence of preliminary information. There is a lack of detail regarding the quantization process. Specifically, the exact implementation of the quantization that results in more than a 10x size reduction is not provided. A detailed ablation study of the quantization type and the length of latent embeddings would be beneficial. Additionally, since the authors do not claim they will release the code for QUEEN, it is challenging for other researchers to evaluate and validate QUEEN's performance. 2. The ablation results presented in Table 2 are confusing and lack crucial analysis. For instance, it is unclear why the baseline achieves the highest rendering FPS without attribute quantization on the Immersive Dataset, while the same quantization improves the rendering FPS on the N3DV dataset, which appears contradictory. Furthermore, there is no clear explanation for why quantization improves rendering PSNR but results in slower training speeds. Additional analysis is needed to clarify these observations.

Questions

1. Please clarify and add more details about the residual quantization in Section 3.2 to make it easier to follow. It would be beneficial to include an ablation study comparing different quantization methods. Additionally, highlighting any specific technical contributions of this work compared to other quantization approaches would improve the readability and comprehension of this section, or this is just a novel application of quantization technique? 2. Please provide a more detailed analysis of the ablation study results to better understand the key design choices in QUEEN’s framework. This additional analysis will help elucidate how these design choices contribute to the performance and effectiveness of the proposed method.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations of QUEEN as well as the societal impacts in their supplementary material.

Summary

This paper presents a novel method for free-viewpoint video learning and streaming based on 3D Gaussian splatting (3DGS). The proposed method learns the attribute residuals of the raw Gaussian points in a frame-by-frame fashion, and the learning process is incorporated with both latent quantization and sparsity regularization. In addition, dynamic contents are identified and separated through Gaussian viewspace gradient difference in order to further accelerate computation. Compared to existing works, the proposed method is able to reduce the memory cost of FVV by a large margin, while preserving high-quality representation capability.

Strengths

* This paper is overall a solid submission. It addresses the challenging problem of FVV encoding and compression. The proposed method is able to a high compression rate while ensuring reconstruction quality. * The authors conduct comprehensive comparisons against state-of-the-art methods. The experiments are convincing. * The paper is overall well-written and easy to follow.

Weaknesses

* The idea of exploiting temporal redundancy via encoding attribute residuals is not new. Although it has not been explored in the context of 3DGS, similar ideas have already been studied in NeRF-based representation, such as [75]. * The proposed method appears to be highly sensitive to hyperparameter tuning, as indicated in Section B, Tables 9 and 10. The authors find it necessary to adjust hyperparameters for different datasets. For instance, the learning rate for position residual learning in Immersive (0.0005) is approximately twice as high as that in N3DV (0.00016). Other hyperparameters, such as gating and quantization parameters, also exhibit significant differences across these datasets.

Questions

None

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitation and potential social impact Sec.C and Sec.D of the supplemental document, respecitively.

Summary

This paper proposes a framework called QUEEN, based on Gaussian splatting, for compact free-viewpoint videos. The data size is reduced to around 0.7MB per frame while achieving fast training speeds. QUEEN encodes the Gaussian attribute changes between consecutive frames. Specifically, it uses latent codes to embed the attribute residuals and employs a learnable gating scheme to update only the Gaussians that exhibit movement. The authors also introduce methods such as gate initialization and point cloud initialization for improved performance. The results are promising.

Strengths

QUEEN updates all Gaussian attributes and achieves better performance than those that do not. The position residual gating is introduced to reduce the computational complexity caused by full updating, which is efficient. The entire framework is sound. The authors also provide comprehensive experiments to demonstrate its effectiveness and justify the choice of the combination.

Weaknesses

This experiment was conducted on Neural 3D Videos and Immersive Videos datasets, which feature forward-facing scenes with limited viewing angles. When we talk about free-viewpoint videos, we also expect large freedom for the novel viewpoints. However, this paper does not demonstrate such scenarios, such as 360-degree rendering for dynamic objects. The authors claim that QUEEN is intended for streaming FVVs, but the experiments do not evaluate or demonstrate the "streaming" feature. There is not information that introduces how does the entropy coding work on the quantized integer latents.

Questions

The reviewer suggests that the authors make their claims more accurate, as mentioned in the weaknesses section. Some notations are confusing. \mathbf{l}_i denotes the quantized integer latents, while \mathbf{l}_{p_i} is the learnable pre-gated residual. In line 223, \alpha is learnable parameter. what is the relationship between \alpha and g_i ?

Rating

5

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

In the limitations section, the authors mention that per-frame training faces challenges in reconstruction capability. It would be helpful if the authors could provide a more detailed explanation.

Summary

This paper introduces QUEEN, a framework designed to enable fast encoding (training) and decoding (rendering) for online free-viewpoint video (FVV) streaming using 3D-GS. Achieving high frame generation quality alongside real-time seamless streaming and rendering is challenging due to the intensive computation and large data volumes involved in this process. Towards this, QUEEN first proposes to apply all attribute residuals, unlike the state-of-the-art works that only use a subset, thereby maintaining high video quality. QUEEN also incorporates several components such as attribute residual quantization and position residual sparsity to accelerate the encoding and rendering process and reduce model size. Additionally, QUEEN leverages inter-frame redundancy, an important feature missed by previous works, to further enhance FVV generation efficiency. The evaluation demonstrates that QUEEN can reduce training time to only 5 seconds and achieve a rendering speed of around 350 FPS with a model size of merely 0.7 MB, all while maintaining near-optimal video quality.

Strengths

+ The paper clearly discusses existing challenges in FVV, presents the proposed solutions logically, and details the evaluation results clearly. + The methodology is robust and the proposed components are thoroughly evaluated. The evaluation results demonstrate substantial improvements in model size and performance.

Weaknesses

- Experimental Settings: The paper lacks details on some crucial experimental settings. For example, the number of bits used for attribute residual quantization is not specified. Similarly, for sparse position residuals, the precision is mentioned as full precision, does that mean fp32? For such case, could using bfloat16 (which reduces model size by half) solve the memory footprint issue without necessarily employing sparsity techniques? - Evaluation Analysis: The analysis of evaluation results is insufficient. For instance, Table 2 (which is not referenced in the text), shows that introducing quantization and sparsity increases the quality (PSNR). Intuitively, these techniques should reduce quality, so an explanation is needed. Additionally, despite introducing extra processing steps (such as quantization, entropy encoding, etc.), training and rendering times do not increase. Detailed analysis/explanation on these evaluation results would be very beneficial. - Perception of Quality: While numerical metrics like PSNR, SSIM, and LPIPS quantify the video quality, they may not fully reflect user perception of quality. A user study, though challenging, would significantly strengthen the paper.

Questions

- Can you provide more details on the experimental settings for attribute residual quantization and sparse position residuals? Specifically, what number of bits is used for quantization, and is fp32 used for sparse position residuals? - Can you explain why introducing quantization and sparsity improves quality (PSNR) and why the training and rendering times do not increase despite the additional processing steps? - Is the receiver/rendering device the same as the encoding device? If the encoded frames/videos are sent to the client side (a different device), shouldn't the rendering be evaluated on a more reasonable device instead of the A100, which is server-level and too powerful for a typical client device? - Can you discuss/study the accuracy-memory trade-off achieved by the quantization and sparsity techniques?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors acknowledge the limitation of exploiting inter-frame redundancy for very dynamic videos, which is reasonable. However, the paper evaluates video quality solely using metrics like PSNR and SSIM, which may not fully capture the viewing experience. Including a user experience study would provide a more comprehensive evaluation and strengthen the paper's conclusions. Also, including the accuracy-memory trade-off study with quantization and sparsity would be beneficial.