TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation

Summary

This paper proposes diffusion guidance (Test-time Procrustes Calibration, TPC) for human image animation. TPC incorporates an auxiliary branch into the diffusion conditions process and provides a calibrated reference image latent. Experimental results show the effectiveness of the provided method.

Strengths

1. This paper studies an interesting problem of human image animation, that is, when the given human image and pose image have shape differences, the performance of the original model is poor. 2. The structure of this paper is well-organized and easy to follow. 3. The experimental results show the effectiveness of the proposed method.

Weaknesses

1. In Figure 1a, I acknowledge that there may be a large shape difference between a given image that needs to be animated and a guided image. In this case, I think the most direct and useful way is to find a way to align the two images, and I don't think it is very difficult to align the two images. So why do we need the proposed TPC method? 2. In line 54, the method of injecting guided images into the model should be more than just cross-attention. Would TPC work if the human image animation methods do not use cross-attention to guide image injection into the model? 3. The last paragraph of the introduction introduces the proposed TPC approach. However, this part is to analyze the gains that TPC can bring, and I think this part is more suitable for methods or experiments. 4. As for the TED man in the demo of the experimental results, why does the animated video only show the man's upper body? 5. For the unseen domain shown in the provided demo, I think in human image animation, we'll give the model images that need to be animated and guided images. Also, the base model is trained on a lot of images of human bodies, and the model should have seen something like this before, so calling it an unseen domain isn't accurate. For this kind of scene, there is a big gap between the character identity in the generated video and the given image, and TPC seems to have no way to help the original model maintain the character identity.

Questions

Please see the above.

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

1. The necessity of TPC 2. The experimental results do not help the original model retain the details of the input image. 3. Introduction needs to be reorganized

Summary

In this paper, the authors propose TPC, an alignment algorithm for human image animation systems. optimal precision is currently achieved only when the physical compositions of the human shapes in the reference image and target pose frame are aligned. Misalignment leads to a significant decline in fidelity and consistency. To address this issue, the authors propose Test-time Procrustes Calibration (TPC). TPC provides a calibrated reference image for the diffusion model, enhancing its ability to understand the correspondence between human shapes in the reference and target images. This technique can be applied to any diffusion-based image animation system in a model-agnostic manner, improving effectiveness at test time without requiring additional training.

Strengths

(1) TPC bridges the gap in shapes between reference and target by providing correspondence guidance latent. With this guidance condition, diffusion-based animation systems achieve robustness to fidelity variations and maintain temporal consistency among frames. (2) The TPC is simple and works in a model-agnostic manner without additional training.

Weaknesses

(1) The implementation details of Iterative Propagation are not relatively comprehensive, and I suggest the authors make an algorithm table to depict the concrete process of the proposed Iterative Propagation. (2) There are some skeleton alignment algorithms in the previous works, such as Everybody Dance Now. The authors should conduct additional quantitative and qualitative comparison experiments between TPC and other previous skeleton alignment algorithms. (3) According to my observations of your provided demos, the backgrounds of some rotation cases encounter considerable changes during the animation process. The background details don’t align with the given reference image, such as the layout and color distribution. For instance, in the galaxy case, the positions of the stars in the background of the animated video are not the same as those in the original given reference image.

Questions

(1) In some particular situations, the protagonist in the given reference image only shows half of the whole body, while the driven target pose shows the whole body parts. Can TPC handle this sort of situation? Could you please give some quantitative and qualitative experiment results regarding this situation? (2) I am unsure how you can estimate or ensure the accuracy of the subset X. Could you please provide some visual comparison results in terms of different subsets with different numbers of key points? Additionally, could you provide the details of the common filtering process?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper starts from an interesting problem that what happens when the motion condition and the reference image are not well aligned. It analyses the robustness of an existing human animation network given different levels of misalignment, and tries to find the underlying cause through attention maps. Then this paper propose TPC, a test-time calibration module to mitigate this issue. This module provides a calibrated latent feature as an additional condition to the diffuser. The calibrated latent is optimized via Procrustes Warping, and Iterative Propagation is proposed to improve temporal consistency among calibrated features.

Strengths

This paper is among the most innovative work I've ever seen in human animation area. It is a curiosity-driven work starting with an interesting problem. The solution is neat and elegant. The presentation is clear and easy to follow. The analysis in Fig 1 and Fig 2 is very interesting and insightful. Technically, using keypoints as the evidence to align shapes is quite direct and reasonable. Using Procrustes analysis to optimize a transformation is sound.

Weaknesses

I did not find obvious weakness, except that the demo results still contain some artifacts. But it is mainly due to the baseline MA or Disco. I recommend the authors to try some new methods as the baseline that have higher quality. The method is insightful, but only for human animation, a small area. It would greatly enhance the value of this paper if it can be applied to some more general tasks.

Questions

What if directly calculating the transformation between the reference image and the target pose via some heuristical evidences? For example, considering the torso part, i.e., keypoints set {6,7,13,12}, since torso keeps rigid to some extend, compared to arms and legs. Then using the vector (mid(6,7) , mid(13, 12)) to compute the rotation angles, the perimeter for scales, the center coordinate for shifting. I know this design is not optimal. But how is the result with such simple idea? I noticed Table 2, the "Linear" and "Affine" setting, but seems not exactly the same one. Can this method be potentially applied to a broader tasks that require spatial alignment between some input variables?

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The Broader Impacts and Ethic Statements have been adequatedly addressed.

Summary

This work proposes a method that combines exsiting human image animation method and Procrustes Warping, which improves the robustness of image animation approach. In addition to the explicit image warping, this work also proposes an interative propagation method to improve the temporal consistency. Experiments show that the proposed method is effective in enhancing the animation performance.

Strengths

1. The proposed method is well motivated, the observation of the misalignment between reference image and target pose is straightforward and insightful. 2. The experiments in this work are solid, both the comparisons and ablations are extensive and detailed. Compared with the baseline methods, the improvements are noticeable. 3. The writing is clear and easy to follow.

Weaknesses

1. As far as I know, the Procrustes alignment is popular in the evaluation of 2D/3D human pose estimation. So, this work introduces this simple technique into the human image animation but technically this contribution is not novel enough. 2. There is no detailed description about the training process of this method, it could be better to introduce more about the training process in the main text. 3. From the Table 2 we can see that the best M is 20, but the evaluation metrics shown in the table are SSIM and FVD, it could better to show more results to help reader understand the ablation comprehensively. 4. Are there any trade-off boserved between the temporal smoothness and motion precision? For example, if M=1, the temporal consistency is the best while the pose precision is the worst?

Questions

How do you apply TPC in DisCo and MagicAnimate? I am a little bit confused since the Figure 5 only depicts the structure of Anymate Anyone.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Discussed.