DN-4DGS: Denoised Deformable Network with Temporal-Spatial Aggregation for Dynamic Scene Rendering

Summary

This paper proposes a novel framework for modeling dynamic scenes. To address the noisy 3D Gaussians in canonical space, the authors introduce a Noise Suppression Strategy (NSS) with a two-layer Deformation Field to resolve this issue. Additionally, they propose a Decoupled Temporal-Spatial Aggregation Module (DTS) to efficiently aggregate neighboring temporal features (for the first deformation) and k-nearest neighbors' spatial features (for the second deformation). Experiments on real-world datasets show substantial quality improvements compared to D-4DGS [1]. [1] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.

Strengths

This paper aims to improve the capability of 3DGS-based dynamic reconstruction methods in modeling real-world scenes. The proposed Temporal Aggregation and Spatial Aggregation are sensible, and the detailed ablation studies in Tab. 4 demonstrate the effectiveness of each component. In terms of rendering metrics, the experimental results in this paper surpass the previous state-of-the-art 3D-GS-based methods in real-world scenes. The design of the dual-deformation network and the experiments with strong baselines are appreciated.

Weaknesses

1. I think the following papers should also be cited: - (CVPR 2024) Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis by Zhan Li et al. - (ICLR 2024) Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting by Zeyu Yang et al. 2. Issues with the HyperNeRF dataset: I believe that the visualization comparisons on the HyperNeRF dataset should include HyperNeRF [2], 4DGS [3], and even Deformable-GS [4]. Based on my understanding of the HyperNeRF dataset, its camera poses are not accurate. This leads to a slight misalignment between the rendered images and the ground truth due to the inaccurate camera poses. Most current image quality evaluation metrics, such as PSNR and SSIM, heavily penalize slight misalignments more than blurriness. For example, in the Fig. 7 `Chicken` scene, I found that both D-4DGS [1] and DN-4DGS exhibit blurriness under the sparse Init setting. I am confident that HyperNeRF and Deformable-GS can render relatively sharp images, but slight misalignments with the ground truth caused by inaccurate poses result in these methods having much lower rendering metrics compared to some blurrier methods. (In Tab. 2, HyperNeRF, which has higher rendering quality than D-4DGS, has lower rendering metrics.) Therefore, I will not rely heavily on the quantitative metrics of the HyperNeRF dataset. The visual results are more important, and comparisons with more previous state-of-the-art methods are needed to demonstrate the effectiveness of the proposed method. 3. Selection of the baseline. I do not think D-4DGS [1] is an appropriate baseline. The essence of why the Deformation Field (outputting delta) works is due to its sufficiently smooth output. D-4DGS's Hexplane clearly disrupts this smoothness, which is why it performs poorly on the D-NeRF dataset. In real-world scenes, due to the high sensitivity of 3D Gaussian explicit representation to inaccurate camera poses, it does not have a significant advantage over NeRF-based methods (like HyperNeRF). I fully agree with the authors' attempt to improve the modeling capability of Deformation-based Gaussian splatting, but I think the authors need to explore more fundamental reasons, such as the accuracy of camera poses and the negative optimization effects of D-4DGS's strong regularization of deformation in real-world scenarios. Methods like 4DGS [3] and spacetime-GS [5], which decouple time features into each GS, can better handle the inaccuracies of camera poses in real-world scenes [2] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021. [3] Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642. [4] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101,2023. [5] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. 2023. Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis. arXiv preprint arXiv:2312.16812 (2023).

Questions

A minor question: I would like to know if the authors have tested the effectiveness of their method on the D-NeRF dataset.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Please refer to the weakness.

Summary

The author proposed a Denoised Deformable Network to enhance the rendering performance of 4D-GS. In this network, a two-stage deformation prediction method was introduced to suppress noise and achieve better performance.

Strengths

Based on the performance on three dynamic datasets, the proposed method outperforms the compared methods.

Weaknesses

1. Novelty: The author claims that the novelty of the proposed method lies in suppressing noise to achieve more accurate deformation prediction. However, from the figures and descriptions, it appears to only apply a two-stage deformation prediction without any denoising operations. 2. Lack of detailed explanation of the method. 3. Insufficient comparison with other methods and related work.

Questions

Method: 1. The description of the method is unclear. In section 4.3.1, the feature $F_{t}(i)'$ is obtained by concatenating $F(i)$, $F_{max}(i)$ and $\gamma_{i}$. However, $F(i)$ and $F_{max}(i)$ have different dimensions. It is not explained how these are concatenated or what the shape of $\gamma_{i}$ is. 2.During optimization, why is an additional embedding $\gamma_{i}$ included? What is its purpose? For instance, in HexPlane, pairs of 2D planes are initialized to represent specific spatial or spatiotemporal axes. How does $\gamma_{i}$ function in this context? 3. In the DSAM (section 4.3.2), the operation is similar to that in section 4.3.1. There is no direct denoising operation, only a two-stage deformation prediction. Related Work and Comparison: Many new works have appeared in CVPR 2024 in terms of dynamic NeRF and Gaussian diffusion. Ablation study: From the methods section, it appears that the NSS includes both the TAM and DSAM. How, then, can it be part of the ablation study?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations.

Summary

This paper presents a two-stage pipeline for reconstructing 4D scenes with 3D Gaussians. Similar to previous works, the authors defined a set of canonical 3D gaussians and use MLPs as the deformable fields to deform them into new frames. The key difference is that the authors notice that the deformable field contains a lot of noise, so they propose to suppress the noise by using a two-stage deformation process. In the first stage, they use a temporal aggregation to aggreate temporal information. In the second stage, spatial aggregation is performed with nearest neighbor. The training loss is standard gaussian noise with TV loss. Results demonstrated that the proposed method outperforms previous methods.

Strengths

- The authors notice that the problem of current 4DGS is that the canonical space contain a lot of noise - The authors propose a noise suppression strategy and temporal-spatial aggregation to remove the noise - The authors perform extensive experiments and ablation studies to verify their idea

Weaknesses

Personally I feel like the biggest issue for this paper is the presentation for the introduction and method are not that clear for me. Firstly, the authors claim that previous 4D Gaussian methods can have noise canonical space. I am not fully understand what does noise means here, there should be more explanation on this. Secondly, it seems that the noise supression strategy is just using temporal aggregation followed by spatial aggregation. But the authors claim that there are two components in Line 46: NSS and DTS, but seems that they represent the same thing. On Line 140, the authors claim that after the first deformation, the noise is attenuated. Why is that? With this being said, what is the difference of the last two rows of Table 4?

Questions

What is the training time of this method compared with other 4D Gaussians? Would two stage training and aggregation delay training speed? Considering the weaknesses I mentioned, I am happy to discuss with the authors and hope to see a better exposition of this paper, especially about the NSS and how that relates to DTS.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have discussed their limitations.

Summary

This paper proposes DN-4DGS, a deformation-based Gaussian splatting technique for dynamic scene rendering. DN-4DGS utilizes a grid-based representation to model the deformation field. It features a two-stage deformation inference pipeline that aggregates temporal and spatial information to reduce noise in the deformation field. These two stages are trained sequentially. Experimental results show that DN-4DGS outperforms other methods in rendering quality.

Strengths

The Temporal Aggregation Module and Denoised Spatial Aggregation Module leverage temporal and spatial features to regularize the deformation field. This idea is both sound and novel. Based on the reconstructed canonical Gaussians demonstrated in the paper, I find that the reconstructed deformation field may be closer to the true deformation. The authors provide clear intuitions about these designs.

Weaknesses

There are some inaccurate description in the paper. In line 158, $G_t$ has the time parameter t. Does this mean that the position of the Gaussian is deformed to the time t? In formulas (3) and (5), the function depends only on the current coordinates x, y, z and time t. However, it actually also involves temporal features and neighboring points in space. These parameters are not described in these two formulas. The authors claims that the reason for performing temporal optimization first is to reduce the impact of noise on the KNN in the second stage. However, in the ablation study, there is a lack of experiments that only use DSAM without TAM, where TAM could be replaced with an ordinary deformation network. Therefore, it is difficult to determine the correctness of this claim. The authors did not provide video results.

Questions

Compared to the 3DGStream method, what are the advantages and disadvantages of DN-4DGS? Adding a discussion of the latest methods can highlight the unique features of this method.

Rating

6

Confidence

4

Soundness

4

Presentation

2

Contribution

4

Limitations

A potential limitation could be the inability to handle scenarios with significant movement.

Feedback by reviewer umjH

Thanks so much for authors' response and the additional experimental results based on the review. According to the rebuttal results, I understand that the essence of the work is to reduce noise in the canonical space of deformation-based Gaussian splatting to improve rendering quality. However, I believe this viewpoint is not valid. In the current deformation-based Gaussian splatting, there is a crucial detail: there is no assumption that `delta=0` at `t=0`, meaning the canonical space does not necessarily represent the state at `t=0`. Therefore, the canonical space is the result of free optimization, and introducing a zero-canonical constraint could actually reduce rendering quality. I'm not sure if authors have conducted zero-canonical experiments. If authors' starting point is to reduce noise in the canonical space, I believe the improvements of DN-4DGS are more due to the increase in MLP layers (equivalent to deepening the deformation field layers). Therefore, to prove authors' point, authors should provide a comparison between the baseline method with an MLP of the same depth and the zero-canonical constraint.