Learning 3D Garment Animation from Trajectories of A Piece of Cloth

Summary

The authors propose a method to transfer the deformations of the observed garments to any other garment. Previous methods either rely on a large-scale dataset for training or analytical physics model with limited expressive ability. On the contrast, the proposed method first learns the constitutive relations from the observation by a neural network (EUNet), then use it as an energy prior to regularize the training of garment deformation model. This design addresses the limitation of previous works and show better results.

Strengths

The strength of the paper is the proposed method does not need to collect huge amount of data with varied body pose and shape and garment types for training. Through theoretical analysis, they prove that they can learn a more physically accurate energy model to describe the deformation of garment. In this way, they do not need an explicit physical model, which tends to have limited expressive power. The derivation is theoretical sound.

Weaknesses

To learn this energy model by EUNet, the authors rely on the synthetic data simulated with blender. A cloth with known geometry (vertices and faces) is assigned with a specific material type. However, this setting is too ideal. In real scenarios, we are more interested in transferring the material of a real cloth to another garment. But having the geometry of a real cloth usually is not feasible. Even though we can have the mesh of the cloth through some registration process, how to get the shape of mesh when it is hanging and dangling is still a problem. In this paper, I do not see the possibility of using the proposed method in real applications. This is the critical weakness.

Questions

1. In Fig. 4, the results of MGN-S+EUNet on dress are not similar to the ground truth data. 2. What is the unit of the errors in Table 1? The errors on the leather and denim seem too big compared with the others.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

See the weakness.

Summary

This submission presents a method that could effectively learn the dynamic patterns of different garments from a single piece of cloth. The key insight is that the motion of different cloths is governed by both external forces and the constitutive relations rather than specific garment topologies. Thus, an Energy Unit Network (EUNet) is proposed to learn the topology independent constitutive laws. Then the animation is performed by energy optimizations. Experimental results shows improved results comparing to previous methods and baseline methods.

Strengths

The paper is well written and easy to read. 1) The paper is well structured. 2) Many terms are well defined and explained. The proposed method is both novel and interesting. 1) The disentangled learning scheme of using a network to learn the constitutive law, which generalizes to different garment types, is physically intuitive and natural. More importantly, this design helps alleviate the needs for large amount of training data of various cloth shapes in dynamic for learning based animation. This disentanglement between topology and energy is achieved by using mesh edge as a unit instead of the whole cloth mesh. 2) The proposed disturbance training strategy helps stabilize the training and improves the generalization of EUNet. As a constraint, it accompanies the direct supervision on the energy form by taking into account the physical meaning of equilibrium state. This helps the network to learn a more reasonable manifold of the energy distribution. Experimental results: 1) Improved results over previous methods are shown both qualitatively and quantitatively. 2) According to the ablation study, the design of both the contrastive loss and dissipation unit loss are validated as effective.

Weaknesses

Some details regarding the design of the EUNet is missing. Although some descripsions on the EUNet design is provided at the experimental section, it is relatively hard for the reader to follow and develop a more coherent understanding of the presented work. Some limitations and open questions that the work might not cover: What about anisotropic materials? How to adapt the current model design to also fit to cloths where its material is anisotropic? How well does the method handles cloths with more complex topology that goes beyond a single layer of cloth? As also pointed by the authors, the method does not handle self-collision. It would be interesting to see how it can be adapted in that axis.

Questions

See weakness.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

See weakness.

Summary

This work proposes a method to learn the constitutive model of cloth materials from observed cloth trajectory using a neural network. It adopts an MLP that operates on individual edges and predicts per-edge distortion based on the deviation of edge geometry from rest shape and trains the network using a combination of supervision on potential energy change with ground truth and optimality of incremental potential. The learned potential energy can be used as a constraint to train neural simulators for garment animation.

Strengths

- I appreciate the novelty in the idea of learning the constitutive model of cloth materials in a data-driven manner. Potentially this formulation could allow the neural networks to understand the intrinsic physical property instead of mimicking the behavior of specific examples, thus of scientific significance if implemented correctly. - The paper is well-written and mostly clear.

Weaknesses

- On the methodology side, the major question is probably the design of dissipative energy. On the one hand, why it is and is merely a function of \(X^t - X^{t-1}\) is questionable. In fact, whether it should be modeled as an absolute quantity is a question because the total amount of dissipative energy seems not that meaningful. The only observable quantity is the relative change of dissipative energy in a physical process. - On the other hand, with the presented framework, it is very hard to learn the major sources of energy dissipation: collision, and friction, since they are neither present in the training data, nor fully modeled (e.g. self-collision) in the formulation. While the dissipative energy is not the focus, the problem is that without correctly modeling dissipative forces, I doubt the possibility of learning an accurate elastic potential energy function. - On the evaluation side, the problem is that the method is only evaluated in a simplified setting, without comparing against methods or in settings that are practically useful (see below). In my opinion, there are two ways to demonstrate that the learned constitutive model is useful: either 1. demonstrate that it is more accurate than an analytical model on real data, or 2. show that it leads to more realistic animation than existing methods (including traditional numerical models). - The evaluation section only shows that the MGN trained with the learned constitutive model is better than those trained with ground truth garments or analytical constitutive model. On the one hand, it does not compare with other state-of-the-arts like HOOD, SNUG, and PBNS that are also formulated in a self-supervised manner. On the other hand, the claim that it is better than the analytical constitutive model is not convincing because the discrepancy may be caused by the limited accuracy in the neural simulator (or even by the mini-network mentioned in Sec 3.3). To truly demonstrate that it is better than an analytical one, it must be compared using a numerical integrator that is guaranteed to converge to the energy minimum.

Questions

See the weaknesses section.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The discussion seems adequate.

Summary

The paper proposes a novel method for animating garments by learning from a single piece of cloth. This approach circumvents the need for large-scale garment datasets, which are resource-intensive and time-consuming to create. The core idea is to use a disentangled scheme where constitutive behaviors are learned from observed cloth and then applied to animate various garments. The proposed Energy Unit Network (EUNet) captures constitutive relations in the form of energy, bypassing the need for traditional physics models.

Strengths

The paper introduces a novel disentangled approach that separates the learning of constitutive behaviors from the animation process. The EUNet models constitutive behaviors using energy units, allowing for direct learning from observed cloth trajectories without traditional physics models. The approach significantly reduces the data requirement, relying on a single piece of cloth for training, making it more practical and less resource-intensive. The method produces animations that are both robust and generalizable, capable of handling various garment types and materials.

Weaknesses

The energy optimization process, although effective, can be computationally intensive and may require fine-tuning to achieve optimal results. The paper would benefit from more extensive experimental validation, including comparisons with a broader range of existing methods and more diverse garment types.

Questions

How does the performance of EUNet compare with traditional physics-based models in terms of computational efficiency and accuracy? What are the limitations of using a single piece of cloth for training, and how can these limitations be mitigated in future work?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Comparisons with a broader range of existing methods would be appreciated, for example, one of the SOTA method named 'HOOD: hierarchical graphs for generalized modelling of clothing dynamics' which has been cited in the paper is worth comparing with.

Summary

This paper proposes to learn garment dynamics using a disentangled learning framework and the Energy Unit Network (EUNet). Instead of relying on extensive garment datasets, the approach learns constitutive behaviors from a single cloth piece and dynamically animates garments through energy optimization.

Strengths

The writing is clear and technical details are described clearly. The visual aids and diagrams are well-integrated, enhancing understanding.

Weaknesses

My main problem with the paper is that the problem of learning/recovering cloth dynamics from structured sample tests has been studied intensively for a long time, and the authors seem not to be aware of this whole field. This is a well-studied problem, and the authors do not position their method against significant prior works. Many existing works have also attempted learning from real-world fabric sample tests or indirect representations (video), which is a much harder problem. To mention a few: 1. "Predicting the Drape of Woven Cloth Using Interacting Particles" Breen et al., 1994 2. "Estimating Cloth Simulation Parameters from Video" Bhat et al., 2003 3. "Data-driven elastic models for cloth: Modeling and measurement" Wang et al., 2011 4. "How Will It Drape Like? Capturing Fabric Mechanics from Depth Images" Rodriguez-Pardo et al., 2023 5. "Estimating Cloth Simulation Parameters From Tag Information and Cusick Drape Test" Ju et al., 2024 The authors should thoroughly review the literature and reposition their contribution and provide experimental comparisons against existing works. Additionally, the literature review section does not include important works from the physics simulation community. Including these references and discussing how the proposed method builds upon or differs from them would strengthen the paper significantly.

Questions

1. Literature Positioning: Can you clarify your awareness and positioning of your method in relation to the existing body of work on learning/recovering cloth dynamics from structured sample tests? As discussed above, many significant studies in this area were not referenced. 2. Physics Simulation/Graphics Community References: The literature review section did not include important works from the physics simulation and graphics community. How does your approach relate to or differ from the significant contributions in this field? Including a discussion of this could provide a better contextual grounding for your work. 3. Experimental Validation: Can you provide more details on how your experiments validate the proposed method against these existing works? Specific comparisons and metrics would help clarify the effectiveness and novelty of your approach. Can you validate your approach against different datasets, including synthetic datasets generated from different simulation engines as well as real-world datasets used by current works?

Rating

4

Confidence

4

Soundness

1

Presentation

2

Contribution

1

Limitations

The authors provided a discussion on the limitations of edge-wise discretization and a lack of self-collision handling.