NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos

Summary

This paper presents an algorithm for the realization of novel view synthesis in dynamic scenarios, leveraging multi-view video data. In order to tackle this task, the paper introduces two key components: a keyframe dynamic field and an interframe velocity field. These fields serve the purpose of accurately representing the motion, geometry, and color information inherent in the recorded scenario. By incorporating these fields into the algorithm, the proposed approach aims to achieve effective synthesis of new views in dynamic scenarios.

Strengths

This paper introduces two novel 3D datasets for dynamic object scenarios and dynamic indoor scenarios. Given the existing limitations in available datasets for view synthesis in dynamic scenarios, the introduction of these new datasets is expected to significantly enhance the progress and development of algorithms in this field. Additionally, while this paper presents innovative contributions, it also incorporates comprehensive experiments to support its claims. Furthermore, the paper is well-written and maintains a coherent structure, making it easily understandable and accessible to readers.

Weaknesses

The primary concept explored in this paper involves the acquisition of a key-frame representation and the establishment of connections between key frames and intra-frames through the utilization of intraframe velocity. However, it is crucial to acknowledge that the notion of learning a canonical field and a deformation field has been previously introduced and extensively discussed in several notable publications. Notably, papers such as "D-NeRF: Neural Radiance Fields for Dynamic Scenes" (with over 500 citations) and "Nerfies: Deformable Neural Radiance Fields" (also with over 500 citations) have extensively addressed and explored this concept. The concept of learning a deformation field has also been extensively studied and advanced in previous research, as evidenced by the paper "Dynamic View Synthesis from Dynamic Monocular Video" which has accumulated over 100 citations. Moreover, while this paper introduces two novel datasets, it is important to note that there exist minimal distinctions between the proposed datasets and preexisting ones such as the Nvidia dataset, Nerfie dataset, and Iphone dataset.

Questions

### Performance-related questions: 1. When compared to previous algorithms, such as D-Nerf, which rely on a canonical field, this paper employs Hexplane as a backbone. Is the improvement observed in this algorithm attributed to the utilization of Hexplane? 2. In comparison to HexPlane, which employs a single representation to learn features for the entire scenario, this paper learns multiple key-frame representations. Does the enhancement in performance stem from the usage of multiple representations instead of just one? ### Innovation-related questions: 1. Instead of directly learning the interframe velocity field, the proposed algorithm initially learns an acceleration field. Why was this approach chosen, and what evidence supports this decision? 2. Instead of learning the precise positions of key-frames, this paper employs uniform sampling for key-frame selection. Does this decision align with the inherent characteristics of dynamic scenarios, considering that motion within such scenarios may not exhibit uniform speeds? ### Dataset-related question: 1. What are the advantages of the proposed datasets in comparison to previous datasets used in similar studies?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The primary limitation of this paper revolves around the level of innovation exhibited by the proposed algorithm and the dissimilarity between the proposed datasets and their predecessors.

Summary

This paper proposes a new representation of dynamic scenes, using keyframe NeRF + Velocity Field. This representation disentangles appearance & geometry from velocity, which allows many exciting applications like future frame extrapolation, unsupervised semantic 3D scene decomposition, and dynamic motion transfer. Obtaining such representation from videos requires a carefully designed system and physics-constrained losses. To validate this model, this paper collects two new synthetic datasets and show impressive results on these datasets.

Strengths

1. Learning a velocity field from videos is an exciting direction and could be impactful for future research, especially for physics-aware 3D/4D tasks. This paper makes an impressive attempt in this direction. 2. The method is well-motivated and intuitively reasonable. The whole idea is easy to follow and sound. 3. The results are impressive, and the applications of the velocity fields are exciting, especially the 3D semantic field segmentation results could clearly show different parts of objects.

Weaknesses

1. Lacking limitation discussions. The current paper clearly has several limitations and they are expected to be discussed in detail in the paper. (1) As briefly mentioned in the broad impact, this paper doesn't have real-world data to validate whether it could work on real scenes. (2) This method seems to require videos with many views at the same timestamp as inputs. I am curious how this method works with monocular videos/sparse views. (3) How does the proposed method deal with abrupt motions and non-typology changes? For summary, the paper should discuss the potential limitations in detail. 2. Some presentations could be improved. It would be nice to give a brief explanation about losses and PINN used in this paper. And Algorithm 1 table could be more precise. The current paper seems to omit too many technical details and the meaning of some losses is confusing.

Questions

I am generally positive about this paper but I think the limitations of this paper (and the questions in that bullet) should be discussed in detail.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The current version doesn't give a convincing discussion about limation and potential broad impact. See the weakness for detailed comments.

Summary

This paper proposes a model for dynamic 3D scenes from multi-view videos. Different with previous method, this paper proposes a physical velocity field instead of a deformation field. The method uses a dynamic radiance field to model key frames, and use a velocity field to warp the key frame to intermedia frames. For optimization, the photometric loss of rendered keyframes and interframes are optimized together, along with several PINN terms. Also, the authors show some applications of their model: future frame extrapolation, semantic 3D scene decomposition, dynamic motion transfer. Two new synthetic datasets are introduced.

Strengths

1. Although velocity field has been introduced to this area (Neural Radiance Flow for 4D View Synthesis and Video Processing), this paper introduces a more complete solution, including the PINN terms and the warping mechanism. 2. The implementations of T-NeRF_PINN and HexPlane_PINN are not clear to me, but the future frame extrapolation of the proposed method is impressive. 3. Two new synthetic datasets are proposed.

Weaknesses

1. The method shows inconsistent performance on interpolation. 2. For dense frame optimization method, T-NeRF is not an appropriate method to compare with, since it is a very naïve solution which simply adds a time dimension to MLP input. To prove the effectiveness of the proposed method, I think NSFF (Neural Scene Flow Field) is a reasonable and meaningful method to compare. NSFF also has warping constraints. 3. The proposed method is basically a canonical based method which separate the whole video into multiple sections and assign a canonical model for each section. From this aspect, it is not so fair to compare with TiNeuVox in this way. How would TiNeuVox perform if also assign the same number of canonical models to it? 4. The potential application of motion transfer is not clear to me. 5. No interpolation performance for ablation study. 6. No experiment on real dataset. NHR dataset (Multi-view Neural Human Rendering) may be an appropriate dataset for this paper. 7. From my aspect, the authors spent too much space on applications. The performance of the interpolation is more important for me to identify the ability of the proposed method to model the dynamic scenes.

Questions

1. For motion transfer, how would the learnt velocity field align with new objects? Put original images of Gnome in Figure 3 (c) may help understanding.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

The authores addresed the limitations, including no experiment on real dataset.

Summary

The paper aims to model dynamic 3D scenes from multiple-videos. The method simultaneously learns for geometry, appearance, and physical velocity of the 3D scenes from video frames. The show different application like future frame extraploation, unsupervised semantic 3D scene decomposition and dynamic motion transfer using the current framework. They further proposed two dynamic 3D datasets with extensive experiments.

Strengths

The paper uses three major components to tackle the dynamic scene, the first is using the keyframe dynamic radiance field to compute radiance field at different time instances would be a good initial 3D estimator. the second is to use velocity field to interpolate the intermediate frames instead of recomputing NERF per frame which is very expensive. Finally a joint keyframe and velocity based interframe optimization produces an smooth trajectory and reconstruction of the dynamic scene. Instead of just showcasing the advantage of the method, displaying the robustness of the algorithm to multiple downstream tasks like future frame Extrapolation signifies the advantage of this method. Good ablation study and the introduced datasets are helpful for future research in dynamic scene understanding.

Weaknesses

There has been many datasets proposed to tackle the problem of dynamic scene understanding, What is the advantage or difference with respect to these datasets has not been well studies. Specifically, Line 187 states that all of the datasets currently available are not useful for this task. It would be interesting to see the effect of the proposed method on one these datasets to understand the advantages and the disadvantages of the proposed algorithm. Effect of choosing keyframe is specific to the dataset. Since the current dataset has only one moving object may be using 16 frames is sufficient. The ablation should show more experiments to justify the keyframe selection process. The effect of the number of cameras is not explored in the analysis. Since you are using dynamic scenes, the effect of fewer frames should be analyzed.

Questions

How does the method perform on other dyanmic scene datasets. even using simple multi-view datasets like panoptic studio should be explored to see the effect of the algorithm. Are the camera poses already provided for the dataset?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

limitations have been discussed

Summary

The paper presents the framework to simultaneously learn the geometry, appearance, and velocity from only video frames. The paper's contribution lies in three directions: 1) Keyframe dynamic radiance fields to learn the time-dependent volume density and appearance. 2) Interframe velocity field to learn the time-dependent 3D velocity and 3) joint keyframe and interframe optimization to train keyframe and interframe fields together with physics-informed constraints. The main contribution lies in the joint keyframe and inter-frame optimization, which introduces three loss functions that help precisely learn to disentangle object masks, types, and materials.

Strengths

S1) The framework is simple to understand yet elegant in achieving many tasks. S2) The paper is well written, with detailed derivation of major contributions. Easy to understand. S3) The work presents Dynamic Object and Dynamic Indoor Scene datasets. S4) The evaluation of the proposed approach is well presented on multiple baselines and downstream tasks (Frame extrapolation, Scene decomposition, Motion transfer). The proposed approach outperforms the existing approaches.

Weaknesses

W1) All the evaluation results are presented on a synthetic dataset where the motion information is constrained ideally. It is unclear from the experiments how well the results translate to real-world scenarios. W2) All the results presented in this work are evaluated only on the authors' dataset. It is vital for the paper to present results on publicly available datasets to establish benchmarks (at least for a few downstream tasks).

Questions

Q1) From qualitative results, the dataset used in this work has limited spatial resolution. So, how would the approach scale to real-looking scenes with larger resolution?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See the weaknesses section.

Impressive results for extrapolation, but less competitive performance for interpolation

Thanks for the hard working to address my concerns. Now, the only concern that remains to prevent me from recommending the paper to be accepted is the relative incompetitive performance for interpolation. As the paper title suggested, this is a method to representing dynamic 3D scenes. I intend to see at least competitive performance to model the dynamic 3d scenes. Also, I notice that the LPIPS of the extrapolation is not always the best in Table 1 and Table 5, which does not match the huge difference of the PSNR. What may be the reason?

Reply to Rebuttal

Thanks for the explanation. After reading all reviews and rebuttals, I think the rebuttal address my concerns and I improve my final rating to acceptance.

Reply to rebuttal

Thank you very much for your detailed responses and valuable contributions, which undoubtedly contribute to the advancement of our community's understanding. I am truly grateful for your clarification regarding the representation employed in your research. However, I do have a couple of concerns that I would like to address. Firstly, I was wondering if your method has been evaluated on a more extensive and realistic dataset, such as the ones utilized in Nerfies and the original Hexplane paper, like the Plenoptic Video dataset. As the Nvidia dataset, you proposed appears to comprise only a limited number of frames for each scenario, I believe that testing on a broader range of real-world data could provide more comprehensive insights. Secondly, I would like to kindly suggest that you consider reviewing the paper titled "Temporal-MPI: Enabling Multi-Plane Images for Dynamic Scene Modelling via Temporal Basis Learning," which also explores the concept of interpolation. It might be interesting to examine and potentially compare your approach with the ideas presented in that work. Once again, thank you for your time and efforts, and I look forward to any further insights you might provide on these matters.

Summary of Rebuttal Discussions

We thank the valuable feedback from all reviewers and conduct a series of additional experiments and ablation studies to further validate and improve our method. Below is a consolidated summary of the changes made and their corresponding results: - **New Experiments on Real-World Datasets:** We evaluate our method on the NVIDIA Dynamic Scene and DyNeRF datasets. For the NVIDIA Dynamic Scene dataset, our method achieves significantly better results in the challenging task of future frame extrapolation, as evidenced in Table 1 of the Author Rebuttal. However, on the DyNeRF dataset where motions are chaotic (e.g., a man pouring coffee/flaming a steak), our method struggles to learn meaningful physical velocities and simply predicts static frames for future extrapolation, as shown in Figure 3 of the appended PDF. - **More Baseline Evaluation:** We include an evaluation of a new baseline, Neural Scene Flow Fields (NSFF), on our Dynamic Indoor Scene dataset. The results, as shown in Table 4 of the Author Rebuttal, highlight the advantages of our method over this state-of-the-art alternative. - **New Ablation Study on Keyframes:** We conduct an ablation study on the number of keyframes on our Dynamic Indoor Scene dataset. Interestingly, we find that fewer keyframes tend to result in better performance (Table 2 of Author Rebuttal). Additionally, using just 4 keyframes instead of 16, our method outperforms the strong baseline TiNeuVox in both interpolation and extrapolation, as shown in Table 7 in the response to Reviewer hQnr. - **New Ablation Study on Training Cameras:** We perform an ablation study on the number of training cameras on our Dynamic Objects dataset. The results (Table 3 in Author Rebuttal) indicate that performance drops sharply with fewer camera views, primarily due to insufficient visual information for physical motion learning. - **More Experiments with TiNeuVox:** Additional experiments for TiNeuVox with multiple canonical spaces reveal that our method still outperforms TiNeuVox in terms of learning physical velocity fields (Table 5 of Author Rebuttal). - **New Evaluation on Monocular Videos:** We evaluate our method on the truck scene from the NVIDIA Dynamic Scene dataset in a monocular setting. The results (Figure 2 in appended PDF) demonstrate challenges due to depth ambiguity, affecting the disentanglement of foreground and background. - **New Evaluation on Random Keyframe Selection:** Our evaluation with random keyframe selection on the Dynamic Scene dataset shows a decrease in performance (Table 6 of Author Rebuttal). - **Better Training Settings for both Interpolation and Extrapolation:** On the real-world NVIDIA Dynamic Scene dataset, we opt for better training settings, achieving comparable performance to TiNeuVox in interpolation and significantly better results in extrapolation (Table 8 in the response to Reviewer hQnr). Our method demonstrates superior performance in both interpolation and extrapolation with better choices of training hyperparameters. In summary, the additional experiments and ablation studies have provided further evidence of the robustness and effectiveness of our method. We hope that our rebuttal materials can adequately address all your concerns and contribute to a more comprehensive evaluation of our work.