Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis

Summary

The authors propose to combine a reposable 4D reconstruction from multi-view video based on a skeletal LBS model with 3D Gaussian splatting. To this goal they introduce a novel strategy for estimation of the skeletal model from a superpoint clustering. The results demonstrate a superior image quality and, thanks to the representation, also fast rendering.

Strengths

S1) Implementation details provided for reproducibility. S2) The claims are validated on common datasets. S3) The result quality is visibly better than the prior work. S4) The skeleton construction is novel, technically sound and produces reposable models in variety of scenes. S5) The quality of exposition is good.

Weaknesses

W1) The skeletons are over-segmented and unnatural and likely would not be very friendly for a human animator. They may still be suitable for data fitting but do not provide nearly as much regularization as an "optimal" (ground truth) skeleton would. W2) The limitations and broader impacts are only discussed in the Appendix which I do not see as a responsible practice. It suggests that the authors do not give downsides of the method the same importance as to the upsides. W3) The authors claim to report Statistical Significance without further comments (checklist item #7), but I cannot see any such features in the paper. W4) It may be a good idea to consider a higher quality captured dataset than ZJU-Mocap. It does not seem to allow for a useful comparison between the methods. Other minor issues and suggestions: - Figure 1: Each superpoints -> superpoint - L145: related rotation matrix -> relative? - $\mathbf{W}$ is overloaded in Eq. 1 and Eq. 6 for two distinct things which is not ideal. - Eq. 13: $\mathcal{L}$ without suffix is not defined. ------------------------ **Justification of recommendation** A solid paper with its incremental but non-trivial contribution stemming mainly from the novel skeleton construction. The experimental results are convincing and the main downside is the clutter and complexity of the recovered skeleton. Despite this, I am currently comfortable recommending acceptance under the assumption that the exposition issues are addressed (especially limitations). My final decision might change based on the rebuttal.

Questions

Q1) The Eq. 13 could use more discussion. Does the formulation avoid over-segmentation of large but correctly rigid parts to multiple sub-parts? It would be useful to see the LBS visualized for the final shapes. Q2) Why are the skeletons so complex and noisy? Is that perhaps related to my other question about over-segmentation in Eq. 13? Are such skeletons practical for re-animation? How was re-animation done in the video? How many parameters / joint motions had to be defined? Q3) Why are there different resolutions reported for each method (Table 2 & 3). Does it mean each method was validated against a different reference image resolution? Or does it mean the methods were all tested against a full resolution reference but some were only trained using low-resolution data? That could potentially introduce considerable bias. Q4) The authors also do not provide any failure case examples/images. Does the method work perfectly for all tested scenes? What about the robots from the limitation Figure 6 of AP-NeRF? Does the new method handle these cases better?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations and broader impacts are only discussed in the Appendix which I do not see as a responsible practice. It suggests that the authors do not give downsides of the method the same importance as to the upsides.

Summary

This paper presents a novel approach for learning articulated objects, alongside the skeletons/kinematic trees directly from input videos, eliminating the need for pre-defined meshes or hand-crafted skeleton priors. Specifically, the paper introduces a hierarchical 3D Gaussian representation, where a set of superpoints are used as guidance for deforming the sub-points using linear blend skinning (LBS). The skeletal structure is further derived from the superpoints, providing articulated structures that enable explicit pose control. Rendering is done by Gaussian Splatting, enabling real-time performance. Overall, the paper tackles a very challenging and useful problem for 3D modeling, the manuscript is easy to follow, and the approach is interesting.

Strengths

The strengths of this paper lie in how it brilliantly leverages 3D Gaussians for capturing the underlying articulated structures in a video, without the need for 3D annotations or pre-defined structure priors. The design of superpoints naturally models prominent candidates that can serve as control points. Furthermore, as the control points are “learned” automatically, it can potentially arrive at a representation that better suits the possible motion of the articulated subject. Overall, the approach presented in the paper is pretty neat, and the experiments show pretty promising qualitative and quantitative results.

Weaknesses

The approach does have some room for improvement. Specifically, - Limited reposability. As mentioned in L372-373, the approach is limited to the motion space in the input video. It would be great if the papers could include visual results for these failure cases. It will be interesting to see how good the learned LBS weights are. - Evaluated on datasets with limited motions: the videos used in the paper mostly contain repetitive motion sequences, and/or with small motions. It will be interesting to see how the proposed method performs on videos with complex/diverse/large motions (e.g., AIST datasets). Also, it is similarly unclear how the method can perform on in-the-wild videos with uncontrolled lighting, or with only a single view. Overall, these weaknesses are very common among template-free approaches, not specifically to the proposed method itself. Nevertheless, it would be great if the paper could include more figures, visual results, and analysis regarding these cases. There are also some issues regarding the experiments, which I detailed in the Questions section below.

Questions

Some comments regarding the evaluation sections: - Tab 1, 2: does the performance gain come mainly from using a higher resolution, or does it come from “capturing better articulated structures”? While Gaussian splatting enables us to use higher resolution due to its rendering speed, it would be great if the paper could also include results with resolutions comparable to other setting (400x400 in Tab 1, and 800x800 in Tab 2). - Is there a way to properly evaluate how good the learned skeleton structure is? E.g., training a skeleton-based 3D articulated model using the skeleton from AP-NeRF v.s. WIM v.s. the proposed method. Also, one small issue: - L147: should be (R^t_b)^-1 instead of R^t-1_b

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper discussed some of their limitations, but it would be great if the paper could include more analysis/visual results for the issues mentioned above in the Weaknesses section.

Summary

The paper introduces a method combining 3D Gaussian Splatting and superpoints for dynamic object modeling, achieving real-time rendering and high visual fidelity. Empirical results show that the proposed method achieves state-of-the-art results on several benchmarks.

Strengths

1. The paper is well-written and easy to follow. The main contribution and methodology are well illustrated. 2. The use of an adaptive control strategy to manage superpoints is innovative and helps in optimizing the model, avoiding redundancy, and maintaining efficiency.

Weaknesses

1. Although this paper achieves real-time rendering compared to AP-NERF, I find it somewhat incremental and lacking in innovation since most parts of the method are existing concepts. 2. This paper emphasizes the concept of "Reposable," but the related experiments are very limited. A thorough analysis of this aspect could effectively distinguish this paper from AP-NERF. 3. This method compares fewer baselines, and the quantitative results do not show significant improvements in rendering effects and speed compared to the baselines, as shown in Table 3, Table 4, and Table 5.

Questions

1. The number of $M$ is not given in the paper, the author should do an ablation study on it. 2. The proposed method and results should be discussed with the latest relevant methods as referenced in [1]. [1] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Please refer to the Weaknesses and Questions.

Summary

The paper proposes a novel approach for reconstructing reposable dynamic 3D objects from RGB videos using Gaussian Splatting, without requiring any template as input. To achieve this, the paper suggests grouping Gaussians around superpoints, which are intended to represent rigid parts of the scene. By optimizing and analyzing these superpoints, a full skeleton model of an articulated object in the input video can be built, refined, and used for reposing purposes. **Details** The approach consists of two main stages: 1. *Dynamic Stage*. After optimizing a canonical 3D Gaussian Splatting (3DGS) representation for a few iterations, a set of superpoints is initialized in the scene. A deformable field mapping each superpoint to a time-variant 6DoF transformation is optimized. These transformations are used to derive the motion of Gaussians by interpolating transformations with neighboring superpoints through linear blend skinning (LBS). The paper also proposes a gradient-based strategy to control (prune, merge, or densify) the number of superpoints in the scene. Toward the end of the dynamic stage, a skeleton structure with joints is enforced and discovered in the scene by analyzing the distance between and configuration of superpoints. 2. *Kinematic Stage*. After discovering the skeleton model of the scene, the number of Gaussians and superpoints is fixed and optimized along with a new MLP mapping skeleton joints to time-variant rotation matrices. These matrices are used to compute the motion of each Gaussian along the kinematic chains using LBS. After full optimization, the skeleton can be used for reposing and editing the reconstructed object. The paper presents extensive experiments and demonstrates higher rendering performance and speed compared to concurrent reposable dynamic Radiance Field methods.

Strengths

1. The paper is well-written and easy to follow. 2. The task addressed by the paper is challenging but crucial for many applications in both graphics and robotics. I appreciate the proposed strategy, which successfully retrieves kinematic chains from RGB videos. 3. The quantitative evaluation presented in the paper is convincing and clearly demonstrates the superiority of the approach over concurrent methods.

Weaknesses

1. The paper may lack sufficient skeleton examples to effectively demonstrate that the proposed approach can recover meaningful structures from RGB videos. Indeed, the primary goal is to recover skeleton structures and enable reposing capabilities, but only a single skeleton example is provided (Figure 5). Including more qualitative examples would likely make the paper more convincing. 2. The paper does not provide details on the optimization time and required resources (e.g., VRAM) for the proposed approach. It appears that a large number of training iterations is needed; a comparison with previous state-of-the-art models would be valuable. 3. The limitations of the approach are interesting but are only discussed in the supplementary material, which is problematic in my opinion. These limitations are crucial for further research and should be included in the main text.

Questions

1. What are the optimization time and required resources (e.g., VRAM) for the proposed approach? How does it compare to state-of-the-art methods? 2. Do the authors have insights on how the method would perform in the context of a monocular video with a moving camera, which is a more realistic setting than multi-view videos? 3. In Figure 5, the skeleton appears to be quite accurate, but some bones are located outside the geometry. Would it be possible to enforce the skeleton to be located “inside” the geometry, perhaps by applying a penalty that encourages superpoints to be close to the centroid of their associated Gaussians?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

As I already mentioned it, the limitations are only discussed in the supplementary material, which is problematic in my opinion. I encourage the authors to try to move the limitations to the main paper in the final version.

Thank you

Thank you for the rebuttal, I have no further questions at the moment. The visualization of the LBS weights in the PDF is useful and while it rather shows limited physiological plausibility of the animation tree, it would be good to include it in the paper to visualize any such limitations.

Thanks for the detailed responses

After going through the author's responses, as well as the comments from fellow reviewers. The experimental results, along with the rebuttal responses, adequately support the claim in the paper, and I appreciate how the work neatly combines existing techniques/concepts to capture the 3D appearance and, most importantly, the motion structure. Therefore, I stand by my original rating -- Weak Accept.

Thanks!

I would like to thank the authors for the efforts they made during the rebuttal as well as the answers to my questions. I think the paper tackles a very challenging task (recovering skeletons with a template-free optimization pipeline) and provides an interesting solution relying on 3DGS. Even though the output skeletons are still over-complicated for potential applications in animation, the proposed approach is certainly a step forward for the field. For these reasons, I recommend acceptance and decide to maintain my initial rating.