Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction

Summary

This paper proposes the masked space-time hash encoding to efficiently reconstruct dynamic scenes. The insights behind the paper is that: most part of the scene is static, simply modelling such static parts can dramatically increase the probability of hash collisions; while increasing the hash table entries requires more memory. To solve these issues, this paper proposes to decouple the scene into a static part and a dynamic part, where the two parts are jointly learned through an uncertainty mask. The uncertainty mask is modeled by an individual voxel grid, on which each voxel stores an value of the uncertainty field. To bridge the gap between uncertainties and masks, a neural estimator is adopted to approximate the mutual information. Experiments are conducted on the Plenoptic Video Dataset, the Google Immersive Dataset, and the self-collected time-synchronized multi-view videos. As shown in the quantitative results, MSTH surpasses state-of-the-art dynamic NeRF methods (HexPlane, K-Planes, etc) in terms of both the reconstruction quality and training/rendering time. Ablation studies also shows the effectiveness of the proposed masked space-time hash encoding.

Strengths

This is a technically very solid paper. Compared to previous methods, the proposed method can reconstruct dynamic scenes with higher quality while highly reduced the training time. I like the idea of using uncertainty to mask the static part and dynamic part in a scene instead of simply a combination of 3D voxel grid and a 4D voxel grid. The experiments are exhaustive and highly support the effectiveness of the proposed method.

Weaknesses

(1) In Figure 3 (c), it seems some static parts have high uncertainty values (the top-left part and the middle-right part)?

Questions

- (1) Is Figure 3(c) revealing that the uncertainty does not accurately model the static part and dynamic part? - (2) In Section 4.3, the authors proposed two variants for ablation study: (1) A pure 4D hash encoding. (2) A simple decomposition which is an addition of a 3D hash table and a 4D hash table. Actually, for the second variant, I'm not quite sure whether the authors split the scene into a static part or a dynamic part using additional models (for example, using the **segment anything model (SAM)** to split a non-rigid object/human from the static background), and then separately model them. If it is not, I wonder would it better to separately model the dynamic part and static part with some pre-computed masks instead of jointly learn the uncertainty during training?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors address the major limitations of existing dynamic NeRF methods, e.g. the efficiency and rendering quality. Some other limitations, such as reconstructing monocular dynamic scenes, blurry scenes, etc, are mentioned in the paper.

Summary

The paper presents MSTH, a method that efficiently reconstructs dynamic 3D scenes from multi-view or monocular videos. The proposed solution uses a space-time hash encoding, a prediction of masks, and uncertainty values that help a method to identify 3D points that belong to moving objects. The intuition is that this masking and uncertainty will help a method identify static points which are easy to reconstruct but also handle well the moving points. Lastly, the formulation of the method allows it to render in fast manner also preserving the size of the model. The paper presents experiments on three real-datasets: Plenoptic Video Dataset, Google Immersive Dataset, and a newly introduced dataset. The presented experiments show that the MSTH improves the PSNR while keeping both the training time and inference time short as well as maintaining a constant size model.

Strengths

In sum, I find the paper to be well executed and explained. Overall, I think the problem of reconstructing dynamic scenes is a challenging but important problem to solve. In detail here are the strengths I find: S1. I find the use of masking to identify the static and dynamic points an interesting and simple idea. I think the idea has good intuitive rationale and it should be effective. S2. The inclusion of uncertainty in the formulation I think is an important example of why uncertainty is very important in machine learning. I support the idea of combining uncertainty with the mask prediction as a mean to improve results. S3. The experiments use real-data and challenging scenes to showcase the benefits of the proposed approach. Overall, I think the experiments are well executed and highlight the benefits of the MSTH. S4. The clarity and description of MSTH is good. Overall, I think the proposed method should be easy to reproduced given the level of clarity of the narrative.

Weaknesses

Overall, I find the paper to be well executed and clear. I only have one weakness to point out: W1. I think the paper is lacking a more in-depth discussion about the newly created dataset to test MSTH. Whil I understand that the algorithm is important, I also think the data part is as important as the algorithm. At the end of the day, the data is required and crucial to solve many problems in the era of deep learning. Thus, I suggest that a final version of the paper includes an extended discussion of the dataset curated by the authors. While I like the approach and explanation of MSTH, I do have some minor concerns: W2. The uncertainty described in the method in a way breaks with the intuitive understanding that an uncertainty value corresponds to a confidence value ranging from 0-1 and could be interpreted as a probability of correctness. However, the "uncertainty" introduced in Section 3.3 deviates from this common understanding. I think the paper should explicitly state a concrete meaning of uncertainty to clarify that the common understanding of uncertainty does not apply to this narrative. The main reason I bring this up is because I think this can confuse readers in the future. W3. In several parts of the narrative (e.g., lines 188 and 186) the paper uses the term "dynamics of a point". I find this term to be confusing since in many fields (e.g., robotics or control) the dynamics typically refer to a model that describes how the state of something changes over time. I think the paper should clarify what this exactly means to clear any misconception of this term. W4. Missing definition of different terms: 1) $\tilde{m}(\cdot)$ is never defined in the text; 2) what does "large $m$" means in line 168? Isn't $m$ the learned mask? Or is it a value? ---- Post Rebuttal After the rebuttal, all my concerns were clarified and I still think this is a good contribution. Thus, I maintain my rating.

Questions

Please see Weaknesses section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

I think the paper clearly states the limitations of the approach.

Summary

This paper present a method for efficient dynamic scene reconstruction. They represent the dynamic scene with a weighted combination of a 3D hash encoding (for static part) and a 4D hash encoding (for dynamic region). The weight is learnable and can be represented by a multi-resolution hash table or a 3D voxel grid. For each query 3D point, they can then get a weighted feature encoding from the hash tables. The photometric loss is then used to supervise the training. To better supervise the learnable mask/weight representation, they further propose an uncertainty guided mask learning and exploit the mutual correlation between "3D point uncertainty" and "3D point mask" to supervise the learning of the mask representation. The experimental evaluations are conducted on two public datasets and a self-collected dataset. The experimental results demonstrate the effectiveness of the proposed method.

Strengths

1: The paper presents a novel method to improve both the reconstruction quality and efficiency for dynamic scenes. The experimentral results demonstrate its effectiveness. 2: The paper is well written.

Weaknesses

I do not find severe flaws of the paper and would like to discuss with other reviewers if necessary.

Questions

The mask representation is not time dependent and the uncertainty supervision is based on the inconsistency between the rendered images & the real captured images. I have following two questions: 1: Since the 3D scene is dynamic, the mask(static/dynamic) for each 3D position should also depend on time. Why you choose to use a time-independent representation. 2: Is it like that as long as the 3D point being occupied by a dynamic object (even for partial frames), it will also be predicted as dynamic (which would have a high uncertainty from Eq. 7)? If so, will your method be able to work well if the dynamic object occupy a larger portion of the image and sweep over most of the 3D space across time. To what extent it can perform well?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

If Question 2 is right, please address the corresponding limitation in Section 5 of the main paper.

Summary

This paper tries to address the challenge of efficiently representing 3D dynamic scenes. It proposes a decoupled representation that uses separate neural implicit representations for dynamic and static 3D points. This approach can reduce hash collision and save the storage of the multi-level hash feature grids.

Strengths

1. An automatic learning procedure to detect dynamic points in the 3D scene. This is achieved by integrating Bayesian learning to treat dynamic points as noise and reduce their weights in the photo-consistency loss. 2. A separate 3D and 4D hash feature grids to save the storage. 3. The optimization of mutual information between uncertainty and mask is demonstrated to be beneficial to the rendering quality.

Weaknesses

1. The concept of representing scenes with separate static and dynamic parts has been investigated in NeRFPlayer or NeRF in the wild. Integrating Bayesian learning to predict the dynamic points is also not new. 2. Lack quantitative ablation study on the mutual information optimization, which weaken the contribution of this step. 3. Inadequate references. There are missing references related to neural implicit representation for large scale and dynamic scenes: Xiuchao Wu et al., Scalable Neural Indoor Scene Rendering, SIGGRAPH 2022 Haithem Turki et al., Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs, CVPR 2022 Liao Wagn et al., Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time, CVPR 2022 Oral

Questions

In Fig.6, what is the meaning of each row?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The proposed method is most effective for scenes with a significant amount of static parts. For scenes with a small amount of static parts, the storage savings will be less significant.

Thanks for the rebuttal

Thanks to the authors for the rebuttal. All of my concerns are addressed. Then I decided to keep my current rating for this paper.