Learning Motion Refinement for Unsupervised Face Animation

Summary

This paper anaylizes the limitations of existing face animation methods in capturing the finer facial motions, and hence design a non-prior-based motion refinement approach to achieve finer face motion transfer in local areas. A correlation volume between the source and driving images is constructed as non-prior motion evidence, and a refinement module is introduced to generate the fine facial motions iteratively. Extensive experiments show the effectiveness of this paper.

Strengths

1. The idea of involving a non-prior based motion refinement module is effective to capture the fine motions in local areas. 2. The manuscript is well written and clearly states the main idea and contribution in face animation. 3. This paper performs extensive experiments, and the results of the same-identity video reconstruction task outperform many previous methods.

Weaknesses

1. This paper concentrates on the motion issue, but the video results of this paper are not impressive enough. No obvious improvements can be seen given the video results on the self reconstruction. 2. This paper does not show the quantitative results in terms of some usual metrics, i.e. CSIM and FID, on the cross-identity reenactment task. I wonder the performance of this work in preserving identity. 3. The introduction should be improved, as it only concludes limited contributions.

Questions

1. Could the authors give more explations about the meanings of flow in the figures? I can not see the changes after updation. There are obvious improvements on the occlusion map, does the improvement in motions mainly come from the enhanced occlusion map. 2. Since Face vid2vid [1] shows a promissing performance on the cross-identity reenactment task, it is suggested to give a comparison with Face vid2vid. It is good to provide the video comparisons. [1] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR, 2021. 1. It is better to provide computation cost evaluation at inference, i.e. Flops, memory, and FPS, compared to other SOTA methods.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This paper is effective to refine the motions to obtain better results, but is still limited to the quality of the learned keypoints, as the limitaions in the paper stated, which I think is more urgent to solve for unsupervised face animation. And this paper did not give comprehensive evaluations on the performance in identity preserving, which is important for face animation in practical applications. I'm not sure whether it is qualified for the conference. If the authors could give reasonable response for the problems stated in **Weakness** and **Questions**, I would like to consider improving my rating.

Summary

This paper proposes a method for face animation via learning to refine a coarse motion field. The refinement is performed in a recurrent manner using the previous iteration's motion, occlusion map and structure correlation volume. Both the qualitative and quantitative results show improvement over prior art.

Strengths

1) Qualitative results show that the proposed method is clearly model finer facial movement much better than prior art, especially around the eye. FNeVR is very close but the proposed method has fewer uncanny artifacts. 2) Quantitatively as well, the proposed method outperforms prior art. 3) The paper is well written.

Weaknesses

1) Across all the results, I notice a strong identity shift during animation. While this is common (and seemingly worse) in prior art as well, it is important to quantify. One way to measure this is to report the FaceIDLoss or L2-loss as the head is rotated 15, 30, 60 degrees from the original head-pose. I believe this is an important evaluation to include for the sake of completeness.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

1) An important limitation of this method is that it is sensitive to the scale of the face of the driving video. For example, around the 1:47 of the supplementary video, the face structure of the animated face deviates significantly from the source and is actually closer to that of the target. This is unavoidable due to the use of 2D key points in the method, in fact it is expected. The authors must include this as a limitation of their work.

Summary

This paper contributes to the field of unsupervised face animation by introducing a novel motion refinement method to overcome the limitations of existing prior-based motion models, especially in estimating detailed facial movements. The paper's approach introduces a new method which uses a structure correlation volume built from keypoint features. This provides motion information that does not rely on prior data. This information is used to iteratively refine the coarse motion flow estimated by a previous motion model. The authors have conducted numerous experiments on challenging benchmarks to test the effectiveness of their approach. The results indicate that this new method enhances the capability of prior-based motion representation by learning how to refine motion. This suggests that their approach can effectively increase the accuracy of unsupervised face animation tasks.

Strengths

In this research, the authors present a new unsupervised face animation approach that concurrently learns both coarse (global) and finer (local) facial motions. Their method integrates a local affine motion model to learn the global, coarse facial motion and a novel motion refinement module to compensate for the local affine motion model's ability to model more detailed facial motions in local areas. The motion refinement process is based on the dense correlation between the source and driving images. To achieve this, a structure correlation volume is first constructed using the keypoint features of the source and driving images. The authors then train a model to iteratively generate minor facial motions from low to high resolution. The learned motion refinements are combined with the coarse motion to generate the new image. After performing extensive experiments on widely used benchmarks, the method was found to deliver the best results among existing state-of-the-art methods. The authors have also committed to making the source code for their method publicly available in the future.

Weaknesses

1. The goal of this article is to learn fine facial motion, but using the VoxCeleb dataset may not be sufficient for this purpose. In my view, finer facial motion refers to the ability to reproduce wrinkles, eyeballs, and micro-expressions at high resolution, which may require higher resolution datasets. 2. As noted by the author, the reliability of keypoints estimation heavily influences the quality of finer facial motion. It is not clear whether the improvement of existing models comes from more accurate keypoints estimation or the proposed module. 3. This paper is very similar to RAFT from ECCV2020 in terms of insight and specifically for the module. A clearer comparison between the two works and their essential differences would be helpful. 4. It is important to note that fine facial motion may not equal to better image quality. Therefore, a more thorough evaluation of the estimated motion would be beneficial. Additionally, conducting this task on higher resolution facial images would provide more convincing results. 5. The author sets the iteration number to 8, which is an important hyperparameter. It would be useful to know why this setting was chosen and whether it significantly affects the model's runtime.

Questions

1. This paper is very similar to RAFT from ECCV2020 in terms of insight and specifically for the module, which raises novelty concerns. 2. Related works [1] are not discussed and compared in this paper. [1] Tao, Jiale, Biao Wang, Tiezheng Ge, Yuning Jiang, Wen Li, and Lixin Duan. "Motion Transformer for Unsupervised Image Animation." In European Conference on Computer Vision, pp. 702-719. Cham: Springer Nature Switzerland, 2022.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Both limitations and social impact are discussed in this paper.

Summary

The paper presents an unsupervised method for learning to create continuous video animations of faces given a single input image and a driving video of the same or a different subject. The method relies on the optical flow between the driving and source images to warp the features of the source image, which are then passed to a generator to create the final output frames. The main contribution of the proposed method lies in the two-step formulation of the flow estimation, in which starting from a coarse prediction using a known prior-based method (Affine Transforms, Thin Plate Splines) it iteratively produces updates to reach a more fine-grained flow (and occlusion) estimation with more local details. The flow updates are learnt in an unsupervised way without using prior models, but by building a correlation volume between the source and driving images. The paper includes experiments, generated images and supplementary videos that validate the authors' claims for more detailed animations in comparison to similar methods.

Strengths

- The paper is well written overall and it is easy to follow along with the presented concepts and results. - The paper includes experimental results that validate the authors' claims, as well as ablations that offer a better understanding about model design choices. - The method produces higher-quality generated images/videos than the compared methods for unsupervised learning of face animation. - The idea of using motivation from RAFT to built a similar correlation structure for estimating motion flow for face animation is interesting.

Weaknesses

- Some design choices are not empirically or experimentally justified in the paper. For example, why are specifically 6 iterations used to update the initial coarse estimations? Why does it make sense to sample from the same correlation structure for all iterations, while each iteration operates at a different image scale? - Even though the method performs well compared to previous methods, there is still some evident identity shift in cross-subject experiments, meaning that the shape of the face slightly changes from the original to the one of the driving frame. This is evident in Figure 5 (for example 2nd row) as well as in the supplemental video. This effect makes sense because the estimated flow between different subjects possibly includes more deformation information, rather than only deformation because of expressions or rigid motion. Disentangling identity from other motions is a common weakness of cross-identity animation methods, which exists in this method, too.

Questions

- In 3.4 it is mentioned that a perceptual and an equivariance loss are employed for training. Is the model trained with both same and different subjects for source and driving frames? - How are C^{i} combined to get the final values from the correlation pyramid? - What is the effect of the correlation pyramid to the memory requirements of the method? How does the memory requirements compare with the compared methods?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have included a section discussing the method's limitations and possible negative societal impact. A mentioned limitation (the correlation matrix relying on learned keypoints) might also be the reason for changing the facial shape in cross-subject animations.

Summary

1. This work proposed a new unsupervised face animation approach to learn simultaneously the coarse and finer motions. 2. The results outperformed the state-of-the-art methods on two representative datasets.

Strengths

1. The target task is important. 2. The method is intuitive and reasonable. 3. The results are good. 4. The paper is well-written.

Weaknesses

1. Compared to prior model. The proposed non-prior models seems like a non-linear version of prior model based on affine transformation. When the number of keypoints increasing and the grid of affine transformation being much finer, will it approach the proposed non-prior model? Another related question is that why the proposed model works better than a local thin-plate-spline motion model (such as [38])? 2. Visualization. 1) In the visualization of comparison in video, the results of the proposed model looks close to FNeVR. Authors could also provide user study in visual quality for comparison. 2) Also, the results on CelebV-HQ is really important, since it is a much challenging dataset. Visualization results on this dataset is necessary, in comparison, ablation study and video. 3) Why didn't report the results of FNeVR on CelebV-HQ on Table 1? I am inclined to accept this paper, if the concerns can be solved.

Questions

None.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, limitations are discussed in the main paper.

Summary

The paper presents a new approach to generate human face videos based on a source image and a driving video, which simultaneously learning both coarse and finer facial motions. The proposed approach utilizes a structure correlation volume constructed from keypoint features to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow that is estimated by a prior motion model. The main contributions of the paper are: 1. A non-prior-based motion refinement approach to compensate for the inadequacy of existing prior-based motion models. 2. Utilize the keypoint features to build a structure correlation volume that represents the structure correspondence between the source and driving images across all spatial locations. 3. Extensive experiments on challenging benchmarks that demonstrate the effectiveness of the proposed approach in enhancing the capability of prior-based motion representation through learning the motion refinement.

Strengths

1. The paper presents a novel approach to generate human face videos that simultaneously learns both coarse and finer facial motions. The proposed approach utilizes a structure correlation volume constructed from keypoint features to provide non-prior-based motion information, which is used to iteratively refine the coarse motion flow estimated by a prior motion model. The approach addresses a significant problem in the field of unsupervised face animation. 2. The paper is of good quality, and the method proposed is clear. The authors provide a detailed description of the approach, including the structure correlation volume and the motion refinement module. The experimental results demonstrate the effectiveness of the proposed approach in enhancing the capability of prior-based motion representation through learning the motion refinement. 3. The paper is well-written and easy to follow. The authors provide a clear and concise description of the proposed approach, including the key components and the experimental setup. The paper is well-organized, and the authors provide a clear summary of the contributions and limitations of the proposed approach. 4. The proposed approach has significant implications for the field of unsupervised face animation which addresses an important problem in this field: the inadequacy of existing prior-based motion models to capture detailed facial motions. The proposed approach is effective in enhancing the capability of prior-based motion representation through learning the motion refinement. The approach has potential applications in creating imaginative image animations for entertainment purposes, but it also has the potential to be used in creating deepfakes, which could have negative impacts. The authors acknowledge this limitation and provide recommendations for future work.

Weaknesses

1. The pictures in Figure 1 are too small, especially the optical flow map is not clear enough, no obvious changes can be seen before and after refinement. Also, is the schematic diagram of affine transformation in Figure 1 drawn based on a real example? Why does the affine transformation change so much after refinement? 2. Straightforward combination of existing techniques. The innovation of this paper is not enough, the main innovation point lies in the motion refinement module. However, the method of correlation volume and iteratively refine optical flow used in it is very similar to the correlation matrix and iteratively update in some flow estimation[1,2,3] and correspondence estimation[4,5] methods, and this set of process of first constructing a 4D correlation volume, and then iteratively updating and optimizing optical flow has also been used in some neural style transfer(NST) methods[6,7], but the author did not clarify the differences between the module they used in this paper and related modules in these NST methods. 3. There is an error in line 180 of the article: the author claims that the look up operation on the correlation volume is shown on the left side of Figure 2, but there is no corresponding content in Figure 2. Should Figure 2 be changed to Figure 3? 4. In the comparison video provided in the supplemental material, the video of each method is too small, and it is difficult to see the tiny facial deformation details. It is recommended to arrange the videos of each method in the form of a nine-square grid, and enlarge the size of each video window. 5. The experiments of verifying the effectiveness of the proposed method are insufficient. The framework used by your method is the framework of unsupervised image animation [8,9,10,11], which should be able to generate animation videos on any object category. In addition to human faces, this framework can also be applied to human bodies and animals. Previous unsupervised image animation methods [8,9,10,11] have also been tested on related datasets of different subjects, including TaiChiHD[9], TED-talks[10], and MGif[8]. Your method has only been tested on face-related datasets, but from your overall framework, I don't see any modules that restrict the method to only work on human face. Therefore, I think that the non-prior-based motion refinement module should be applied to datasets of different objects to further test its effectiveness. 6. Lack of quantitative experiments on the Cross-identity task, such as user study that have been used in previous methods [8,9,10,11]. References [1] Dosovitskiy, Alexey, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. 2015. “FlowNet: Learning Optical Flow with Convolutional Networks.” In 2015 IEEE International Conference on Computer Vision (ICCV). doi:10.1109/iccv.2015.316. [2] Teed, Zachary, and Jia Deng. 2020. “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow.” In Computer Vision – ECCV 2020, Lecture Notes in Computer Science, 402–19. doi:10.1007/978-3-030-58536-5_24. [3] Xu, Haofei, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. 2022. “GMFlow: Learning Optical Flow via Global Matching.” In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr52688.2022.00795. [4] Kim, Seungryong, Stephen Lin, SangRyul Jeon, Dongbo Min, and Kwanghoon Sohn. 2018. “Recurrent Transformer Networks for Semantic Correspondence.” arXiv: Computer Vision and Pattern Recognition,arXiv: Computer Vision and Pattern Recognition, October. [5] Zhang, Pan, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. 2020. “Cross-Domain Correspondence Learning for Exemplar-Based Image Translation.” In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr42600.2020.00519. [6] Liu, Xiaochang, Xuanyi Li, Ming-Ming Cheng, and Peter Hall. 2020. “Geometric Style Transfer.” Cornell University - arXiv,Cornell University - arXiv, July. [7] Liu, Xiao-Chang, Yong-Liang Yang, and Peter Hall. 2021. “Learning to Warp for Style Transfer.” In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr46437.2021.00370. [8] Siarohin, Aliaksandr, Stephane Lathuiliere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. “Animating Arbitrary Objects via Deep Motion Transfer.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr.2019.00248. [9] Siarohin, Aliaksandr, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. “First Order Motion Model for Image Animation.” Neural Information Processing Systems,Neural Information Processing Systems, January. [10] Siarohin, Aliaksandr, Oliver J. Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. 2021. “Motion Representations for Articulated Animation.” In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr46437.2021.01344. [11] Zhao, Jian, and Hui Zhang. n.d. “Thin-Plate Spline Motion Model for Image Animation.”

Questions

It’s suggested that the author should clarify the differences between the module they used in this paper and the related modules in some NST methods [6,7]. Also, experiments on more datasets of different object categories are needed.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes, they have adequately addressed the limitations.

Anonymous link of the provided video.

https://user-images.githubusercontent.com/38600167/259588792-0b7d0816-e316-4304-8d50-dd1a54ddc48d.mp4

Rebuttal Update

I would like to thank the authors for the rebuttal. After reading it and the other reviews, I am inclined to keep my rating. I would strongly encourage the authors to include the table from Q3 of the general response in the main paper.