Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting

Summary

This paper proposes using Hash Encoding to model the Deformation Field for dynamic scenes. The authors first decompose 4D encoding into four 3D encodings to avoid the losses caused by the low-rank tensor assumption. They also introduce an attention module to decouple spatial and temporal features. Since explicit modeling can result in insufficiently smooth delta predictions for the deformation field, the authors incorporate a smooth loss to strongly regularize the outputs of Hash Encoding. Experiments on both synthetic and real-world datasets show substantial quality improvements compared to prior state-of-the-art methods. In addition, Grid4D shows a significant FPS improvement compared to Deformable-GS (without lightweight).

Strengths

I think the main strength of this paper is the authors' deep understanding of Deformable-based Gaussian splatting. To be compatible with the densification of vanilla Gaussian splatting, the deformation field needs to output delta (x or rotation). The key to the success of outputting delta lies in the sufficiently smooth output of the deformation field. This is why Deformable-GS [1] can outperform D-4DGS [2] significantly on datasets with accurate camera poses, since Hexplane is not as smooth as MLP. In addition, Hexplane not satisfying the low-rank tensor assumption in dynamic scenes is also a **potential reason**. The authors insightfully addressed the issue of MLP being overly smooth and explicit methods being insufficiently smooth. Therefore, they used Hash Encoding, which does not require the low-rank tensor assumption, to ensure high-frequency details, while employing smooth loss to ensure the foundation of the deformable-based Gaussian splatting. The design of the deformation network and the experiments with strong baselines are compelling. Ablations are also appreciated. [1] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101,2023. [2] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.

Weaknesses

1. low-rank assumption -> **low-rank tensor assumption** is more accurate. 2. I think the following papers should also be cited: - Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis by Zhan Li et al. - 4D Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes by Yuanxing Duan et al. 3. Datasets: - D-NeRF: I think `Lego` (wrong scene with inconsistent training and test set) should not appear in the table, or it should use the Deformable-GS [1] setup with the validation set as the test data. This can prevent misleading subsequent research. - HyperNeRF: I do not find the results on HyperNeRF convincing because the camera poses in the HyperNeRF dataset are inaccurate. For example, in the `3D Printer` (Fig. 5), Grid4D is visibly clearer than TiNeuVox, but the metrics do not reflect this. Therefore, I suggest that the authors adopt `NeRF-DS` [3] dataset with more accurate camera poses for real-world scenes comparison. 4. More difficult motion: the motions in the datasets used for experiments are mostly rigid or easy to explain. It would be interesting to see if the method is able to handle more difficult deformations, like non-rigid or large motion. Based on my understanding of the Deformation Field, it is challenging for it to handle large motions effectively. 5. More comparisons: I wonder if the authors have considered SC-GS [4]. It could be better if the authors could compare their method with SC-GS. I believe the approach of SC-GS could be applied to Grid4D to achieve higher rendering quality and FPS. [3] Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8285–8295, 2023. [4] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.14937, 2023.

Questions

A minor question: Lines 227-231 state that different scenes use different hyperparameters. I would like to know how significantly these different hyperparameters affect the scenes, as this greatly impacts the method's versatility.

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

Please refer to the Weakness section.

Summary

This paper proposes a Grid4D representation for dynamic scene rendering. It breaks low-rank assumptions on 4D-GS and propose a decomposed 4D hash-grid representation for encoding canonical 3D Gaussians. A attention module is used to aggregate the spatial-temporal features of 3D Gaussians. More training strategies are applied in Grid4D to ensure efficient training and rendering quality are also better than 4D-GS, and deformable-GS.

Strengths

1. The paper proposes a deformable representation for dynamic scene rendering and without relying on low-rank assumption. 2. The attention module looks reasonable for feature aggregation. 3. A smooth training strategy is employed to ensure the smoothness of Grid4D.

Weaknesses

1. Why don’t you compare with [13]? SC-GS now preserves the state-of-the-art (SOTA) rendering quality in D-NeRF. 2. If you choose 4D-GS as your baseline, please provide the complete results on the Neu3D dataset and all other metrics (average training time, average rendering speed, average storage consumption). 3. Only selecting some figures and reporting PSNR/LPIPS/SSIM for comparison is not quite enough in 3D vision. The reviewer recommends that the authors submit more comparison videos as supplementary material. 4. If the authors want to propose a method that can replace the deform-based dynamic Gaussians (4D-GS, SC-GS, deform-GS) and become a new baseline, they should provide strong, comprehensive results in the experiment section. 5. Additionally, the D-NeRF dataset is just a toy. I don't think much efforts are deserved to used in the dataset after the publication of SC-GS (PSNR already up to 40). If the authors are still interested in dynamic scene novel view rendering, I recommend focusing on monocular setups (DyCheck, NeurlPS2022) or multi-view (Neu3D, CVPR 2022). 6. The figure looks unattractive.

Questions

1. I want the authors to demonstrate that Grid4D is better than 4D-GS, Deformable-GS, and SC-GS in most evaluation metrics. Please use average results instead of the range 0.3~1.5 in Table 7, as that range is too large. If some metrics are worse, please provide more discussions. 2. Solving a low-rank problem to achieve better rendering quality in the D-NeRF/NeRF-DS/HyperNeRF dataset is indeed a contribution. However, the reviewer believes that the core challenges of novel view synthesis in this dataset have already been solved. There are many important topics in dynamic scene novel view rendering, such as large motion and monocular novel view synthesis.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

As mentioned in weakness and questions.

Summary

The paper presents a grid-based method to compute deformed gaussians to render dynamic scenes from input images. It proposes to perform a 3D decomposition of the 4D input, using multiresolution hash encoding to access spatial (static) and temporal (dynamic) feature vectors. The static features are fed to an MLP responsible of computing a directional attention score, which is composed with temporal features coming from another MLP, resulting in the deformation features. Those features pass through a multi-head decoder, which infer the deformation parameters. Those parameters are used to deform the set of canonical gaussians, which are rasterized to generate the final image.

Strengths

(1) Contextualization is very good, cited works are sufficient. The Introduction and Related Work sections may be successfully used for familiarization with the fields needed for proper understanding of the paper. (2) Every part of the method is intuitively presented and the Math holds. (3) Mathematical notation is very clean. (4) I liked the introduction of an attention mechanism in this context and the idea behind the smooth regularization.

Weaknesses

(1) I am not convinced about the temporal smoothness and coherence of the proposed method. The interpolations needed for grid-based methods are known to introduce discontinuities even in static cases (NGLOD, Instant-NGP). The paper does not present a supplementary video nor a temporal coherence metric (such as FID-VID or FVD). Moreover, all presented experiments are based on specific frames of the datasets. It is not currently possible to qualitatively or quantitatively evaluate this aspect of the proposed approach, which is crucial for a 4D method. This is the core reason behind my current rating. Since the authors are not allowed to add a supplementary video nor include a link in the rebuttal (as far as I know), I believe a way to answer this critic with data would be to show a table with temporal coherence metrics. However, I encourage the authors to present other arguments if they find creative ways to be convincing. I liked the paper and I am willing to increase my rating in case the authors are convincing in this subject.

Questions

(1) Small presentation advice: at the introduction, it would be better to talk about Figure 1 before Figure 2, because Figure 1 is very good as a primary description of how the 4D encoding is decomposed. Starting with Figure 2 is a little bit more confusing. (2) I think the Preliminaries section should include a small introduction to multiresolution hash encoding too. It would help the reader to have better context for the remaining of the method section. (3) I would like to know the values of $\lambda_c$ and $\lambda_r$.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes, the limitations are presented in the supplementary.

Summary

The paper introduces Grid4D, a novel dynamic scene rendering model that leverages hash encoding for 4D input decomposition, enabling high-quality and speedy rendering of dynamic scenes. Unlike traditional plane-based methods that suffer from excessive feature overlap due to low-rank assumptions, Grid4D uses a tri-axial 4D decomposition into spatial and temporal 3D hash encodings. This approach, combined with a novel directional attention mechanism, improves the discrimination and aggregation of deformation features, enhancing both the visual quality and rendering speed of dynamic scenes.

Strengths

Grid4D's use of 4D decomposed hash encoding addresses the limitations of plane-based methods by effectively reducing feature overlap, which enhances feature discriminability and rendering accuracy. The novel attention mechanism aligns well with the diverse deformations across different scene components, allowing for more accurate deformation prediction and rendering. It is also good to demonstrate superiority over SoTA models.

Weaknesses

1. Despite its advancements in rendering performance, Grid4D does not significantly improve the training speed compared to existing models. 2. The sophisticated architecture involving multiple hash encodings and the directional attention mechanism might complicate the implementation and tuning of the model. 3. The model's complexity and the specific tuning required for different datasets might limit its generalizability or lead to overfitting on particular types of dynamic scenes.

Questions

1. How does Grid4D handle scenes with highly non-uniform motion patterns? 2. Could the directional attention mechanism be adapted for use in other types of neural network architectures beyond scene rendering? 3. What are the specific challenges faced when applying Grid4D to real-world datasets compared to synthetic ones? 4. How does the smooth regularization impact the model's performance in terms of real-time rendering capabilities?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

1. Grid4D shows limited improvement in training speed. 2. It does not fully address challenges related to extremely dynamic or unpredictable environments. 3. The effectiveness of the model heavily relies on the novel encoding and attention mechanisms, which might not translate as effectively to different rendering tasks or simpler scene compositions.

Feedback from reviewer T422

Thanks for authors' response and the additional experimental results based on the review. I will not dwell too much on the authors' measurements of SC-GS rendering metrics. Overall, the rendering quality of SC-GS and Grid4D is similar (please refer to the homepage of MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting). However, the authors have underestimated the FPS and training time of SC-GS. For instance, the FPS measurement of SC-GS should fix the KNN, rather than querying it in every iteration, achieving nearly 300 FPS on D-NeRF at 400x400 resolution. The training time is about 40-50 minutes, far less than 75 minutes. Moreover, I believe 4DGS (FDU, real-time 4DGS) should be compared in Neu3D dataset. Considering my personal concerns about the future of deformation-based Gaussian splatting, the per-scene optimized hyperparameters by the authors, and the incremental (or lack of) improvement compared to SC-GS, I have decided to lower my score to `weak accept`.