DynPoint: Dynamic Neural Point For View Synthesis

Summary

DynPoint is an algorithm proposed to enhance the synthesis of novel views in monocular videos using neural radiance fields. It focuses on predicting explicit correspondence between neighboring frames, enabling information aggregation for view synthesis. The demonstrated experimental results show a significant reduction in training time, comparable outcomes to previous approaches, and robustness in handling uncontrolled and lengthy videos without requiring a canonical representation of video content. To achieve it authors propose to improve the estimated scene flow from the optical flow with corrected depth values

Strengths

the proposed technique for scene flow estimation is a solid contribution. It takes into account two inconsistencies in depth and flow that are denoised with the method. The improvements can be applied to any baseline as a separate technique to improve baseline (e.g. NSFF). Moreover, this allows extracting correspondences for the arbitrary 3D scene. hierarchical scheme to render the semi-explicit structure is a novel and a trustable way to get the fast method based on existing research Experiments demonstrate the substantial of the proposed method and provides a strong contribution to the field.

Weaknesses

The proposed method relies heavily on existing models and it is unclear whether the quality depends if you replace them with others Authors don't investigate the quality of the correspondence, which can be interesting by itself. The easiest way is to extend section 4.4 with a comparison versus method like "Deformable Sprites" Ye et al. The presentation of the visual results can be improved. Figure 3 should have a close-up by emphasizing the region of interest, otherwise, results are not clear as well as others. Not compared with “DynIBaR: Neural Dynamic Image-Based Rendering”. Yes, I understand that it is a recent work. At least add numbers to the table for Nvidia dataset. Without video results, it is difficult to reason even with such metric improvements on average. No limitation or ethical section to give a reader explicit statements.

Questions

1. How do you measure the time? Does it include phase 1 and the hardware is the same as competitors? 2. “Neural point cloud” technique is similar to [1, 2, 3] representations in some sense (humans & static scene examples). Could you extend related works in the neural point cloud as well? Does hierarchy is an important contribution for the neural point cloud? If so, it can be 3. What is the main drawback of the method for iPhone dataset? Is it possible to extend the comparison on this benchmark with others, since the gap is much smaller than on others? 4. Do you consider including video metrics for the results (e.g. VMAF)? 5. How issues in Figure 5 can be addressed and what is causing them? [1] PointAvatar: Deformable Point-based Head Avatars from Videos [2] Self-Improving Multiplane-To-Layer Images for Novel View Synthesis [3] Pulsar: Efficient Sphere-based Neural Rendering

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Fast changes in the scene as well as thin objects can be a problem for the method, which is quite common for this class of models. Compared with other methods this one needs more information and additional pretraining steps. the category-specific methods can be better for non-rigid parts. The scalability of the method is limited due to two-stage training for each of the scenes.

Summary

The paper proposes a novel method for performing novel view synthesis from monocular captured videos. This is done by learning scene flow and depth parameters which enable the accurate aggregation of appearance information from nearby frames. The paper proposes a novel method of acquiring both consistent depth estimations for a video, and consistent scene flow represented by an MLP. These two sources of information are used to synthesize novel views at a specific time from an arbitrarily specified viewpoint by sampling from corresponding information in a window of nearby frames. The proposed method is demonstrated to outperform existing methods based on representation learning in synthesizing novel views from monocular videos, and this is shown for a wide variety of datasets. AFTER REBUTTAL: I have read the author rebuttal, and believe that it has addressed my questions. I appreciate the detailed discussion on the comparison to DynIBaR. I also believe the additional videos provided have definitely helped improve my opinion of the method, although I wish the method was tested with more qualitative video examples (as mentioned by another reviewer). However, I believe the testing now is sufficient.

Strengths

In my opinion, the main strengths of the paper are that: 1. The presentation of the paper is concise, clear, and thorough. - The introduction and related work I find to be comprehensive, and accurately describe what the contributions are and what the current issue is with the practice. - The methods section is described in detail, and is relatively clear to understand and follow. - The figures are all informative and demonstrate the capabilities of the method. One thing I wish could be added is a video result demonstrating consistent novel view synthesis. 2. The proposed contributions are original and impactful. - The module for estimating consistent depth for a monocular video seems novel, and while there is existing work tackling this (see questions), the method proposed seems to generate good enough results for novel view synthesis with DynPoint. - The method for estimating and representing the scene flow between adjacent frames and using this to aggregate information from reference to target frames seems novel (although based on image-based rendering ideas), and learning this from the video seems like an important contribution. 3. The evaluations are sound. - The evaluations are done for many (4) different datasets, and compare to many different dynamic scene representation methods (Nerfies, D-NeRF, NSFF, HyperNeRF among others). Because of how thorough these evaluations are, and DynPoint outperforms all of the baseline methods, it is very convincing that the proposed method is robust and is able to generate high-quality results. - The ablations are relatively thorough, and Table 4 shows how the individual parts of the representation and aggregation pipeline contribute to the final quality. I found it to be informative about the method.

Weaknesses

I do not view there to be many major weaknesses of the paper. One thing which would be interesting is a comparison to the method DynIBaR [31]. The paper mentions that this method is limited to short videos due to the capacity of MLPs or other representation mechanisms (L34), but from my understanding this method uses a similar philosophy as this work where new frames are synthesized as inspired by image-based rendering and scene flow estimation. This seems to me to be the current state-of-the-art. Additionally, it’s not clear that the videos this method is evaluated on are long enough for this to be a concern, considering NeRF-based methods can be evaluated and these are constrained by the capacity of the MLP being used to represent the scene. Additionally, I think there could be a bit more discussion on the limitations of the proposed method (see limitations).

Questions

- How does the generated consistent video depth estimation compare to other work in this field, for example: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/. Could this method be dropped in and used for this estimation step?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The paper adequately addresses the limitations of the method. One additional limitation that discussion of would significantly improve the quality of the paper, is the amount of differing from the monocular video path which can be taken and still synthesize reasonable novel views. Obviously, since this method is using appearance information in nearby frames, it is likely to not be able to hallucinate information beyond these. However, NeRF-based methods are able to put something there and potentially reconstruct better results due to the smoothness of the representation learned by the MLP. Some comparison of the existing methods and the proposed method on this axis (range of possible novel views synthesized) would be helpful, and if this is a limitation of the proposed method then it should be discussed.

Summary

This paper proposes an algorithm for novel-view synthesis in dynamic scenes. According to the abstract and introduction section, this paper takes a monocular video as an input (L.98) and cope with uncontrolled or length scenarios (L.3). Using geometric priors, such as monocular depth and optical flow from off-the-shelf methods, this paper exploits the pixel-wise correspondence to encode sceneflow within the network.

Strengths

Overall, I am a bit not that positive for this paper. Let me put more comments on the weakness section.

Weaknesses

_1. Wrong experimental setup._ This paper claims that they focus on rendering scenes from a monocular video to handle lengthy video. However, this is not true. NVIDIA dataset [63] captures the scenes using a camera rig (2x8 array camera? not sure), which is not a monocular video. I think that the correct meaning of the monocular video is addressed by the Dycheck dataset [17] which is called iPhone dataset [17] in this manuscript. Moreover, NVIDIA dataset [63] only has a few training frame which is not well aligned with the addressed points by the authors in the abstract and introduction. At least, HyperNeRF dataset [42] provides more than 200~ frames per scene, where I could say this is a bit lengthy video. Such inconsistency in addressed problems and the proposed solution is quite not good. I am not that satisfied with the quality of the writing. Moreover, where can I find the qualitative results on Dycheck dataset [17]? While table 3 provides the quantitative results, I could not find any qualitative results. _2. No video demo_ More seriously, though this paper tackles the neural rendering in dynamic scenes, I could not find any video demo even in the supplementary material. __Lack of video demo is a big problem__. _3. What is the difference in technical contribution in comparison with NSFF [30]?_ NSFF [30] also exploit the sceneflow understanding within the network, similar to this paper. What is the main difference? At least authors mainly discuss this issue and provide some quant/qual difference within their experimental section. But I could not find any difference or novelty in this part. _4. Lack of related papers to address the speed in dynamic NeRFs._ This paper is not the pioneering work that provides fast rendering/training speed in dynamic NeRF. I hope that the authors to look at the paper of TiNeuVox, Siggraph Asia 2022 (https://github.com/hustvl/TiNeuVox). If this paper wants to claim the novelty in speed, the authors should have fairly compared with TiNeuVox.

Questions

I hope that the authors properly address my concerns which are listed in the weakness section. Overall, this paper does not provide the whole information that is necessary for the reviewers to properly judge its validity.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

It's okay to me.

Summary

The paper proposes a method for dynamic scene synthesis from monocular video. The authors aim to speed up training and deal with long-duration videos. To do this, they propose to estimate scene flows of surface points supervised by the signals which are generated by the proposed consistent depth algorithm. Then the information is aggregated to form a neural point cloud and novel view images are rendered by sampling amount this point cloud.

Strengths

1. The paper proposes a novel solution to dynamic scene. Instead of estimating density and color of the scene for each time, the paper explicitly warps the pixel features of reference images to target time according to the explicitly reconstructed 3D geometry. This would reduce the time for training, as there less learnable components. 2. The experiment shows SOTA results on Nvidia dataset.

Weaknesses

1. Please consider to rewrite the Equation 3. 2. The results on Nerfie dataset and Iphone dataset are less consistent compared to Nvidia dataset. Why not test TiNeuVox on Nerfie and Iphone dataset? Or test the proposed method on HyperNeRF dataset? 3. Please highlight all best results in Table 2 & Table 3. 4. The paper claim that the method could handle long-duration video, which I agree. However, no experiment is conducted to prove this ability.

Questions

1. What is the render speed of the method, considering the relative complex neural point cloud construction process. 2. An open question: is it possible to extend this method to a generalizable method, which could generalize to new scenes without training? For example, replace scene flow MLP with an un-learnable method and use a generalizable renderer. 3. Equation 7 is confused. The warped 2K point clouds are summed or combined?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors addresed the limitation I concerned, which is the potential failure caursed by the flaws in explicit depths and flows.

Demo video to all reviewers

Dear ACs & Reviewers, To address the reviewers' queries, we've prepared a demonstration video showcasing our algorithm's performance on the HyperNeRF dataset. You can access the video through this anonymous link: https://drive.google.com/file/d/1g7AdPvE6SKzZvWnuKfv6hUNW8z0freHE/view?usp=sharing Best regards, Authors of Paper 724

Demo video 2 to all reviewers

Dear Reviewers and ACs, To enhance the validation of our model's performance, we conducted three distinct experiments. Firstly, we generated a "novel-view" video, maintaining the initial frame's motion while rendering around the camera pose. Secondly, a "stabilized-view" video was produced by keeping the camera pose of the first frame consistent throughout the entire replay. Lastly, we crafted a "bullet-time" video by maneuvering the camera around the pose of the initial frame during playback. The "novel-view" experiment was carried out on both the curls and tail scenarios of the Nerfies dataset, comparing NSFF and DynPoint. Notably, NSFF demonstrated the second-best performance based on Tab. 2 (Main text). In the case of the "stabilized-view" experiment, the scenarios 3d-printer and chicken (link in our last comment) from the Hypernerf dataset were utilized. Here, we also compared NSFF and DynPoint, with NSFF showcasing the second-best performance according to Tab. 1 (Rebuttal PDF). Lastly, the "bullet-time" experiment was executed on the iPhone dataset, particularly on the pillow and mochi-high-five scenarios. A comparison between Nerfies and DynPoint was made, with Nerfies achieving the second-best performance per Tab. 3 (Main text). Link: https://drive.google.com/file/d/131bIp_wOM8H9Nsqv56L-0VNdn6vNKfJd/view Best, Authors of Paper 724

Reply

I want to thank the authors for their effort in addressing my concerns. After reading their response and considering the feedback from other reviewers, I believe that the new experiments provide a comprehensive evaluation of their proposed approach for me, and I maintain a positive view on its acceptance.