Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses

We propose a novel method for learning representations of poses for 3D deformable objects, which specializes in 1) disentangling pose information from the object's identity, 2) facilitating the learning of pose variations, and 3) transferring pose information to other object identities. Based on these properties, our method enables the generation of 3D deformable objects with diversity in both identities and poses, using variations of a single object. It does not require explicit shape parameterization such as skeletons or joints, point-level or shape-level correspondence supervision, or variations of the target object for pose transfer. To achieve pose disentanglement, compactness for generative models, and transferability, we first design the pose extractor to represent the pose as a keypoint-based hybrid representation and the pose applier to learn an implicit deformation field. To better distill pose information from the object's geometry, we propose the implicit pose applier to output an intrinsic mesh property, the face Jacobian. Once the extracted pose information is transferred to the target object, the pose applier is fine-tuned in a self-supervised manner to better describe the target object's shapes with pose variations. The extracted poses are also used to train a cascaded diffusion model to enable the generation of novel poses. Our experiments with the DeformThings4D and Human datasets demonstrate state-of-the-art performance in pose transfer and the ability to generate diverse deformed shapes with various objects and poses.

Paper

Similar papers

Peer review

Reviewer RKb25/10 · confidence 4/52024-07-07

Summary

The authors propose a skeleton-free pipeline for implicit 3D pose representation and transfer. They introduce the prediction of Jacobian fields to achieve shape-preserving representation, which facilitates the application of transferred poses. To enhance accuracy, a per-identity refinement step is included, utilizing intrinsic-preserving loss. Additionally, a cascaded diffusion model is employed on the compact pose representation to generate novel poses.

Strengths

1.The use of Jacobian fields for shape representation is innovative. 2.The method of incorporating an intermediate step for intrinsic preservation is justified. 3.Training a diffusion model on extracted implicit poses further demonstrates the effectiveness of the pose representation.

Weaknesses

1.FPS is utilized to sample keypoints as a basis for pose representation. However, for complex poses, FPS-sampled keypoints may not be evenly distributed on the surface due to a lack of awareness of mesh connectivity, potentially resulting in poor pose representation. 2.Given that the pose is represented by keypoints' coordinates and corresponding latent features, the shape information is inherently entangled in the keypoints' coordinates. This could lead to the shape information being encoded in the latent features.

Questions

1.The proposed method may face limitations when animating non-T-pose meshes if it relies heavily on T-pose assumptions for pose estimation and transfer. 2.It would be beneficial to demonstrate pose transfer results for both source and target meshes from the Mixamo dataset and compare these with results from the ECCV 2022 paper "Skeleton-free Pose Transfer for Stylized 3D Characters" to assess performance and efficacy. 3.Comparing the training and inference times with baseline methods would provide insights into the computational efficiency of the proposed approach.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes, they are discussed.

Reviewer RKb22024-08-12

Response to rebuttal

Given the additional clarification and experimental results, I am more inclined to recommend acceptance of this paper. However, I am also want to see the potential drawbacks of using jacobian fields for shape representation and failure cases caused by simply using FPS as basis(for extreme poses and loose clothes). Also, the performance seems drop a lot for SPT after your retraining, is it possible to train on the same dataset to compare with their reported results?

Authorsrebuttal2024-08-13

Dear reviewer RKb2, We sincerely appreciate your positive feedback. Due to space constraints, we were unable to fully address all of your comments in the rebuttal. Please allow us to further address them below. **Potential Drawbacks of Using Jacobian Fields** We did not encounter noticeable failure cases associated with the use of Jacobian Fields. Our ablation study demonstrated that the overall quality of pose transfer improves with Jacobian Fields, both quantitatively and qualitatively. We will include additional qualitative results from the ablation study in the final revision. One potential drawback of using Jacobian Fields is the requirement for differential operators, such as the cotangent Laplacian and gradient operator. These differential operators may not be available for meshes with multiple connected components, as noted in the original NJF paper [Aigerman et al., ACM ToG 2022]. We will also clarify this limitation in the revision. **Failure Cases due to FPS** We also did not observe specific failure cases related to the use of FPS keypoint sampling. While we believe that simple FPS point sampling is sufficient for our framework, in the final revision, we will further test our method with other keypoint sampling techniques, such as uniform sampling, and report the results. Thank you for your detailed comments and suggestions. Please understand that the new experiment requires more time than is available in the remaining discussion phase. **Comparison with SPT** We would like to clarify that the results reported above are based on training both SPT and our model with shapes of a **single** identity, which aligns with our problem setup. In our work, we focused on learning pose representations from shapes of a single identity and transferring them to a new identity. In contrast, SPT uses a much larger dataset that includes a variety of identities as training data, which is why we could not directly apply their training/test splits to our method. While it is technically feasible to extend our framework to train with multiple identities, we found that it would require substantial changes to our algorithm. Also, please note that our method requires preprocessing of meshes to compute differential operators, and some Mixamo shapes fail during this preprocessing. In the revision, we promise to extend our method to train the networks with multiple identities and report the results using the closest train/test splits to those of SPT (excluding models that fail in the preprocessing). We would also like to emphasize that, while this is not a perfect apple-to-apple comparison due to the different train/test splits, the pose transfer accuracy of our method trained with a **single** identity ($2.28 \times 10^{-3}$ PMD) is comparable to that of SPT, which was trained with a much larger dataset ($2.39 \times 10^{-3}$ PMD, as shown in Table 4 of the SPT paper). This demonstrates the effectiveness of our method, even with a significantly smaller training dataset.

Reviewer xBSu9/10 · confidence 5/52024-07-11

Summary

This paper presents a novel representation learning framework for pose estimation which is disentangled from the identity of the object. This implicit representation is used for generating and transferring poses using a cascade diffusion model. A keypoint-based hybrid pose representation with a sparse mesh is used to ensure it is compact enough for the generative model. Moreover, the authors use Jacobian fields as the shape representation, allowing their implicit deformation network to take mesh face coordinates as input as opposed to vertex coordinates. The latent representations learned from this are used in a self-supervised per-identity refinement module for further improvement. This allows the proposed network to transfer motion sequences from one object to another. Moreover, these representations can be used to animate previously unseen characters. The authors provide extensive experiments with both quantitative and qualitative ablations, showing the superiority of their approach over three baseline networks, using both animals and humanoid objects from different datasets.

Strengths

- The paper is very well-written with clear descriptions of motivation, research question, methodology, mathematical formulations, and experiments. - There are extensive experimental results to establish the superiority of the proposed method. In particular, the qualitative results clearly help visualize the improvements. - It is very easy to read

Weaknesses

- No weaknesses

Questions

-The results look very promising, and I am curious how this proposed method can be transferred to images or videos (with textures and backgrounds) insead of only mesh generation. Do you have any insights about how this will work and how your method can be adapted for this scenario?

Rating

9

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitations of Jacobian fields are discussed in the conclusion

Reviewer skND6/10 · confidence 4/52024-07-13

Summary

This paper introduces a novel 3D generative model for pose-identity disentangled representation of 3D shapes. It proposes to use a set of keypoints with features to represent the pose of a 3D shape and learn a pose-extractor and pose-applier to accomplish pose transfer between instances. Experiments are conducted on widely used benchmarks with two categories: animals and humans, showing superiority of the proposed method compared to the baselines.

Strengths

- The studies task is novel and important, with broad applications in different communities. - The proposed method is sound and effective. Using a dense set of keypoints to represent the pose of 3D shape is natural and well-motivated, which also results good results in the experiments. - The evaluation is extensive and informative. Experiments are conducted on widely-used datasets, with comparisons against enough baselines. For baselines without the code, the paper provides results with own implementation. Ablation studies show the effectiveness of the proposed components. - The writing of the paper is mostly clear, illustrating their method and evaluation in a clean manner.

Weaknesses

- Although the exposition is mostly good to me, there still remains some questions that are not fully addressed. See the questions below. - No video results or animated results are provided. Therefore it's hard to evaluate the 3D/4D generated results (e.g., whether the result is 3D consistent from any view or whether the temporal consistency is good). - No qualitative results for the ablation study is provided. - It would be better if analysis on the keypoints can be provided. This might include analysis on the number of keypoints and the way to sample it.

Questions

- How would the number of keypoints affect the result? What would be the fewest number of keypoints that is still feasible for this pipeline? - Regarding the experiments: - What is the detailed setting of NJF? If I understand correctly, NJF takes the pose parameter of the target pose as input. What should be the pose parameter for the deformingthings4d dataset? - How is the texture of the generated mesh predicted? In L265, the authors mentioned to use FID to evaluate the visual fidelity, yet how is this implemented, given that the generated mesh seems to not have texture? Also, how is the camera determined to generate the images used to calculate FID? - How is the "template" mesh defined in the framework, especially for categories without a template, such as animals? For humans, I understand that you may use T-pose human mesh as the template, but what would the case for the animals?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No failure cases are discussed in the paper. Under what condition would the proposed method fail?

Area Chair xCa42024-08-10

Dis

Hi reviewers, Thank you for your hard work in reviewing the paper! Please check out the authors' responses and ask any questions you have to help clarify things by Aug 13. --AC

Reviewer xBSu2024-08-12

Thank you for your response. After reading other reviewers' comments and looking at the additional experiments the authors have provided, I believe this work shows significant progress in an important field for the research community. The extensive qualitative results clearly show the advantages of the proposed approach. The authors' responses to other reviews are also satisfactory, and I stand by my original score for this paper. I strongly recommend it for acceptance.

Reviewer skND2024-08-14

Thanks for the rebuttal! Most of my concerns have been addressed, and I will raise my score. I wish the changes and comments from all reviewers could be incorporated to the camera-ready version of this paper.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC