Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation of the captured object poses challenging. Moreover, the state of the art dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains. In this paper, we present a novel method utilizing a point-based representation and Linear Blend Skinning (LBS) to jointly learn a Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward-warping approach achieves state-of-the-art visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared to existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reposable 3D reconstructions without the need of object-specific skeletal templates. Code will be made available at https://github.com/lukasuz/Articulated-Point-NeRF.
Paper
Similar papers
Peer review
Summary
The authors present a method for dynamic radiance fields of subjects whose motion can be described by skeletal animation. The method automatically extracts a skeleton using medial axis transform. Further it obtains a object feature point cloud from a pre-trained NeRF and expresses their position as a function of the skeleton via linear blend skinning. The animatable point cloud (including skinning weights and animation) is optimized using the PointNeRF renderer using losses for RGB, rigidity, animation smoothness and 2D CD against the mask. After optimization, it can be manually reposed and rendered from novel views. The method is evaluated on the Robots and Blender datasets where it compares favorably against previous work.
Strengths
- Using a point based representation to solve the given task is a good idea and makes sense, given that deformations might be very local and not restricted to a smaller set of rigid parts. - The ability for automatic part decomposition and reposing are interesting contributions that adds value with respect to other dynamic NeRFs that might produce better visual results. - The method compares favorably against the most relevant previous work WIM. - The extracted part decompositions and skinning weights look very good and intuitive - The paper is well written and very clear. - The authors clearly discuss limitations, show failure cases and provide ablations
Weaknesses
- The whole work is purely constructive (not many novel conceptual insights) and most parts of the presented method are existing concepts: The method combines NeRF (as TiNeuVox), PointNeRF (adding rotation equivariance), MAT, and ARAP. However, there is novelty in the joint system. I think the positive aspects of this work outweigh this. - The method in general is pretty close to some of the methods for animatable human radiance fields [80-93]. It differs mostly in that it does not rely on a human template but extracts one from scratch and thus, also works on other subjects. The paper would profit from a comparison with one of them on a human subject though. - In general, while relevant related work is mentioned, it is not always compared against. Some relevant examples are NeRFies [6] and NSFF [4], which both are known for producing better results than D-NeRF, which is used in the comparisons.
Questions
It does not become clear to me what exactly is done with the features $\mathbf{f}_i$ coming from TiNeuVox. Are they just used as point feature initialization and then further optimized using PointNeRF or are they kept fixed? I can imagine that the PointNeRF renderer could adapt to these features without modifying them. This boils down to the question of which parameters of the model are optimized when, which seems to be not completely clear in the paper.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Limitations and potential negative societal impact are discussed.
Summary
Authors presents a method to learn articulated model from multi-view video. And demonstrate an ability for efficient learning skeleton pose along as view synthesis model for dynamical structures. Moreover the suggested method drastically improve convergence compared to naive approaches. To extract the skeleton authors apply Medial Axis Transform for initalization and RBF weights for the joints. The results on the D-NeRF dataset are demonstrate the superiority of this model over predecessor.
Strengths
I appreciate the ideas in the method especially skeleton construction and mask loss. - The approach doesn’t take into account pre-defined joint structure - Convergence is quite fast - Can handle non-rigid motions (in theory) - This a first method learning articulation model purely from the input images The description of the method is clear and solid.
Weaknesses
The main idea is interesting, but the comparison is poor. The idea of skelton initializing is mostly based on the previous works, and the it is not clear how much the final quality degrades, it is necessary to compare it with the ideal human skeleton, for example. To apply this technique you first have to pre-train a dynamic version of NeRF, that complicates method compared to others. All experiments are done for a single backbone and extremely limited variety of data As you mentioned this method assume single object in the centre of the scene. Simplified model produces unrealistic results (figure 6) the skeleton even cannot cover actions from training. There are some works (e.g. TAVA) that incorporate idea of non-template methods (fairly they utilize keypoints) that have to be mentioned Human related baselines can be interesting here.
Questions
Why do you need time-dependent rotations? Does your training time include pre-training of the dynamic scene in Tab 2. Do you need ARAP and transformation loss, how do they influence on the training. Have you consider using something different to TiNeuVox method, there are plenty of much faster and accurate versions (e.g. HexPlane, Tensor4D)?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
Highly relying on the retrained method Speed and memory consumption for higher resolution Simplification and skeleton construction itself is based on heuristic and can be very poor on some scenes.
Summary
The method reconstructs a reposable Dynamic NeRF of an articulated object from multiview videos. This is achieved by using linear blend skinning (LBS) of an automatically extracted skeleton to represent the deformation from canonical to observation space.
Strengths
Combining LBS kinematics with NeRF appearance model effectively produces compelling reposing results. Yielding a posable skeleton enables numerous traditional animation workflows.
Weaknesses
A quick literature search yielded https://arxiv.org/abs/2208.14851 which talks about reposing with a different way of avoiding inverse mapping issues. It may warrant a mention in related work. Could that approach be combined with this paper's automatic skeleton extraction (they use a known rigged template mesh) to yield similar results? Given that the method yields a posable skeleton, the results could have been much more compelling with artist-made or even mocap-driven animations (rather than just blending between a couple of poses).
Questions
Thank you for the supplemental video. I think the Ground Truth T-Rex needs to also be animated. Line 95 mentions Fi_d as a backward-warping MLP, did you mean Fi_b? Line 98 uses p' in an expression for v_m, was that supposed to be p^c? Specific descriptions of various MLPs listed in line 101 would be helpful. In line 103, did you mean to use delta_i instead of just delta? Is the N in equation 8 (L_arap) the same N as in L_smooth? How come the simplified skeletons for the man and dinosaur are so sparse, compared to the two robots?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors address the main limitations of their approach, namely that LBS restricts the method to individual rigidly-linked bodies (as opposed to deformable objects or full scenes).
Summary
The paper presents an approach to articulated view synthesis, introducing the concept of Template-free Articulated Neural Point Clouds. The authors utilize a structure-free point-based NeRF representation which supports forward-warping of canonical objects to any poses through Linear Blend Skinning (LBS). Such a representation allows for the joint optimization of LBS pose parameters as well as dynamic NeRFs in a short training time. The method is evaluated on two datasets and compared with existing methods, such as D-NeRF, TiNeuVox, and WIM, showing superior or comparable performance on dynamic novel view synthesis and novel pose synthesis.
Strengths
(+) The paper provides thorough explanations of the model and experiments. (+) The paper proposed an effective point-based NeRF representation to support the forward-warping of the canonical space for template-free objects. (+) The method achieves better novel view synthesis and reposing results in a shorter training time on two datasets compared to existing approaches. The visualized learned LBS weights fields and poses are cleaner than WIM.
Weaknesses
(-) Inconsistent Motivation: The paper's motivation appears to be inconsistent throughout the text. Initially, the primary motivations are identified as 1) **Reposability** and 2) **Efficiency**. However, in the model section, the method is built upon a pre-trained TiNeuVox as initialization, aiming to animate TiNeuVox to new poses. Consequently, the efficiency of the proposed method heavily depends on the backbone/representation used, and the main motivation here seems to be **Reposability**. In the experimental section, the method primarily compares with WIM to demonstrate its efficiency over WIM on novel view/pose synthesis. A more convincing demonstration of the **Reposability** performance of the proposed method would involve comparing it with WIM in __its original training iterations__. To prove the **Efficiency** of the method, it would be beneficial to investigate the influence of different backbones/representations on the proposed method. (-) Performance and Efficiency: The performance and efficiency of the proposed method do not seem promising compared to TiNeuVox. Given that the method builds upon a pre-trained TiNeuVox as initialization and aims to animate TiNeuVox to new poses, one would expect the method to perform at least similarly to TiNeuVox on the novel view synthesis task. However, as shown in Table 1 of the supplementary material, the method performs worse than TiNeuVox on 5 out of 7 scenes and requires significantly longer training time. (-) Dependence on Pre-trained Model: The proposed method relies on a pre-trained model as initialization, whereas WIM does not require such a pre-trained model. For a fair comparison, it would be advisable to also compare with a modified version of WIM that is initialized from a pre-trained model. This comparison would provide a more balanced view of the strengths and weaknesses of the proposed method.
Questions
* Please refer to the weaknesses section. * How to optimize blend skinning weight vector $w_i$? And what is the motivation for defining another $\alpha$ for scaling the weights? * $\Phi_d$ should be $\Phi_b$ in line 95 to represent the backward-warping MLP.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
* The authors could discuss how variations in image quality, such as resolution, lighting conditions, and the presence of noise, might impact the performance of their method. * How well does the method perform when applied to longer sequences or more complex scenes?
Summary
This paper tackles the task of dynamic novel view synthesis from multiview videos and aims for the ability of reposing. It tackles the problem with a point-based rendering approach. More importantly, it does not need any pre-defined or class-specific template/skeletons and learns a per-video data-driven skeleton. The effectiveness is verified on two datasets, i.e., Robots and Blender.
Strengths
1. The paper is well-written and easy to follow. 2. The proposed skeleton learning is quite interesting: using a data-driven point cloud from TiNeuVox; then approximating an initial skeleton via Medial Axis Transform (MAT); finally using another data-driven component to refine the skeleton from MAT.
Weaknesses
### 1. Robustness to Initialization a. The proposed approach heavily relies on the NeRF pretraining: 1) geometry-wise, the initial skeleton needs the pre-trained NeRF's density function to sample points, which will be later used to run MAT; 2) appearance-wise, the points' features come from the pre-trained NeRF. b. A natural question is how robust the proposed approach is to the initialization. One way to demonstrate this could be: 1) train TiNeuVox until convergence; 2) initialize the proposed approach from different checkpoints of TiNeuVox's training; 3) draw a plot of the final performance wrt different initializations. c. What is related is: can authors clarify what the statement "we uniformly sample the density $\sigma$" (L124) means? Does it mean that you will first compute densities on plenty of points to get a sense of min and max densities? And how will you sample the point cloud with such min/max values? ### 2. Evaluation of Human Dataset Arguably, one of the most important articulated view synthesis scenarios is human-related rendering. I am wondering whether authors can provide quantitative and qualitative results for the proposed approach on some human-related datasets, e.g., ZJU-MoCap used in [20]. Such results could provide a more complete assessment of the proposed approach.
Questions
See above.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors provide a discussion on limitations.
Thank you for the response! Since my score is lower than others, I will clarify my position after rebuttal. I appreciate the comparison on the ZJU-Mocap and ask to add it into the main text as well as different backbones to improve the quality of the work for the readers. I am still interested in the initialization of the ideal skeleton for humans and comparing the learned LBS with TAVA. Moreover, a lot of implications are made with Blender dataset and can be unrelated to the real data (as demonstrated in the example of ZJU-mocap). I wouldn't be against the paper, since the idea is interesting and the experiment design satisfies me, although, I would recommend authors to make their conclusions more fair by adding human data into the main text with comparison over relevant baselines.
Thank you for your feedback. We will add an experiment to our manuscript, where we explore our method initialized with an ideal human skeleton similar to the one used in TAVA. Furthermore, we will incorporate the ZJU-Mocap experiments into the camera-ready manuscript. We hope that will allow you to adjust the recommendation in the official review form.
Thanks for the rebuttal
My rating remains. A comment regarding "Reposable Skeleton Animation": You already have the capability of specifying new skeleton poses and rendering a 3D model from them. (quoting from line 231: "smoothly interpolating between user-defined poses"). I was only suggesting to use an animation artist to specify more interesting poses to blend between. I don't think that's an open problem.
Thank you for your feedback. We will add a more complex manually defined animation of a walking “Spot” robot to the manuscript to better showcase abilities of our method.
Dear Authors, Thank you for your detailed rebuttal and the additional experiments provided. Your response has addressed several primary questions and concerns raised during the initial review. I do not hold a strong opposition to the acceptance of this paper given the merits of the ideas and the further clarified experimental comparisons provided in the rebuttal. I recommend incorporating the clarifications and improvements from the rebuttal into the revision. Best, Reviewer ZYjY
Thank you for your comment and acknowledgment that important concerns have been addressed in the rebuttal. We hope that this will allow you to adjust the recommendation in the official review form. As suggested, we will add the additional experiments and results into the manuscript wherever the page limit allows.
Rebuttal Reply
I appreciate authors's effort in addressing my concerns. After reading the rebuttal and other reviews, I think the newly-added experiments provide a more complete evaluation of the proposed approach and make this a solid work. Therefore, I maintain my positive attitude toward acceptance.
Thanks
Thank you for the answers to my question. I appreciate the additional experiments regarding initialization. It is interesting to see that fine tuning of features is not necessary at all, if they are initialized from TiNeuVox. I would have expected that the impact on quality is larger (in a negative sense). It seems to show though, that the feature spaces that PointNeRF and TiNeuVox come up with are quite compatible. I am not fully satisfied with the responses regarding additional comparisons. I welcome the qualitative results on ZJU-mocap but a quantitative comparison against previous human focused works would still improve the paper, even if the presented method performs worse. In the end, it is to be expected that the presented method will be slightly worse since it solves a harder task (without any human templates involved). It would be interesting though to see how large the gap is. Looking at the qualitative results it seems that the presented method has trouble to fine tune to the real data, leading to smoothing and some artifacts. I agree that it probably is due to issues in the data, e.g. incorrect camera poses. However, this makes a comparison even more important to assess the method. Regarding Nerfies and NSFF: can you confirm that the same conclusions still hold when you add them to the tables? All in all, my opinion has not changed in either direction after rebuttal and I tend to keep my original score.
Thank you for your continuing support. We agree that the initialization experiment is interesting and that it shows that the features and their decoding are robust to the choice of their representation. We plan to expand our performance comparison with additional time-variant methods. To this goal, we were not able to find author’s benchmarks of NFSS and Nerfies applied to object-centric datasets that are the aim of our method. We could attempt to test these for the camera-ready revision if requested but we do not anticipate them to deviate far from performance of D-Nerf from the same era. Moreover, even a potentially higher reconstruction accuracy would not change our conclusions. This is because NFSS and Nerfies rely on unstructured backward deformation field or hyper-parametric canonical space, both of which are incompatible with the reposing goal. Hence, while the lack of kinematic constraints may allow for a better overfit to data, a good performance of these methods does not challenge our core contribution but instead hints at potential backbone alternatives. Instead of NFSS and Nerfies, we add additional comparisons to more recent methods Hex-Plane [2] and and K-Plane [3]. Table 1 shows that the reconstruction quality of these methods in the Blender dataset is in a similar range as TiNeuVox [1] shown in our submission. Therefore, the conclusions made in our manuscript remain valid. **Table 1: Average Performance on *Blender* with More Baselines** | | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Reposeable | |----------------------|-----------------|-----------------|--------------------|-------------| | D-NeRF [0] | 30.50 | 0.95 | 0.07 | $\times$ | | TiNeuVox-B [1] | 32.67 | 0.97 | 0.04 | $\times$ | | HexPlane [2] | 31.04 | 0.97 | 0.04 | $\times$ | | K-Planes hybrid [3] | 31.61 | 0.97 | - | $\times$ | | WIM [4] | 23.81 | 0.91 | 0.10 | $\checkmark$| | Ours | 29.10 | 0.94 | 0.06 | $\checkmark$| **References** [0] Pumarola, Albert, et al. "D-nerf: Neural radiance fields for dynamic scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. [1] Fang, Jiemin, et al. "Fast dynamic radiance fields with time-aware neural voxels." SIGGRAPH Asia 2022 Conference Papers. 2022. [2] Cao, Ang, and Justin Johnson. "Hexplane: A fast representation for dynamic scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [3] Fridovich-Keil, Sara, et al. "K-planes: Explicit radiance fields in space, time, and appearance." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [4] Noguchi, Atsuhiro, et al. "Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.[5] Shao, Ruizhi, et al. "Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Decision
Accept (poster)