VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Summary

The paper present VASA, a new method for talking head video generation. The method is build as a diffusion model using Transformer. To increase the performance of the model and allow more control on the generated video the authors decided to use the representation from [1] and learn to disentangle its components. With this the model can control the gaze, expression and camera position. The method achieve impressive qualitative and outperforms the methods they compare with. [1] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022

Strengths

The qualitative results are very impressive and probably better than the state-of-the-art. The idea of disentangling the existing representation of [1] is nice. The model is very fast at inference on costumer grade GPU. The paper is well written and easy to understand. [1] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022

Weaknesses

My main issue with the paper is that despite the work being presented as reproducible in the checklist I feel that many too many details are missing to actually reproduce the method or the experiments : - The new proposed CAPP score lack details for reproducibility e.g training procedure. Sharing this model would also be of interest to the community (and do not raise ethical concern which could justify keeping the code private). As of now it is difficult to know if the metric is actually sound. - It is unclear if the entire voxceleb dataset is used and how many clips remain after preprocessing ? - Lack of details of the new OneMin-32 dataset : size, type of videos, resolution, origins of the videos... - The paper says that the model is trained on "4 NVIDIA RTX A6000 GPUs" and that the model "train on massive talking face videos from a large number of identities". How long does the training take and how much data is actually used. Because if all of VoxCeleb + the new unreleased dataset are used the training could be very long. More details are required here. - It is not clear how the condition are used ion the network. Are they simply concatenated to the motion latent or used in cross Attention inside the Transformer - It is not entirely clear if the architecture of [2] used out of the box to obtain the facial latent or if it was modified for the disentanglement. Assuming that the dataset used is very large (>10e6 samples) is the comparison against the other methods that use 50k-100k samples for training fair. An ablation with a training on a dataset of that scale would have been interesting. Without it the impressive qualitative results of the method could simply be due to the huge amount of data. The novelty is limited the paper mostly reuse existing modules and innovation is mainly in the disentanglement. The comparison against sota is limited, the most recent method, SadTalker, is from 2022. The FVD score on VoxCeleb should have been shown anyway, other method from the literature present it. The method apparently used [1] for gaze direction estimation. However [72] appears to be a method to classify gaze between different modes (fixed, quick motion...). Was the method modified to obtain gaze direction g that is used in the paper ? The ablation only present results on the gaze and audio condition. It would have been interesting to also see the effect of the expression condition. [1] Raimondas Zemblys, Diederick C Niehorster, and Kenneth Holmqvist. gazenet: End-to-end eye-movement event detection with deep neural networks. Behavior research methods, 51:840–864, 2019. [2] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022.

Questions

How do the authors deal with the resolution difference between methods when computing FVD ? With CAPP score are the head motions related to speech semantics measured ? (e.g head shake when saying no.) to finish

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

The authors address the limitation in the appendix.

Summary

The paper presents a method for generating highly realistic talking head avatars that combines the diversity of facial expressions with the real-time generation speed. It provides a practical and commercially valuable approach to the field of talking head generation.

Strengths

1. The overall structure of the paper is very clear and coherent, with a well-defined problem statement. 2. By decoupling the information in dynamic faces, better control over the expressiveness of the generated faces can be achieved, meeting the needs of the users. 3. The visual presentation of the video is excellent and leaves a lasting impression.

Weaknesses

1. The paper does not explain why the proposed method can achieve real-time generation; the use of a diffusion transformer structure might actually lead to a decrease in speed. 2. There are some unclear configurations in the implementation section of the method, such as the scale of the video used for training.

Questions

1. Will this project be open-sourced? As it could actively promote progress in the field of talking head generation. If it is not open-sourced, it is suggested to provide more implementation details. 2. Why is there no comparison with the recent EMO[1] method, for which there are already corresponding implementations in the open-source community? [1] Tian L, Wang Q, Zhang B, et al. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions[J]. arXiv preprint arXiv:2402.17485, 2024.

Rating

8

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

1. Although VASA-1 has made significant progress in generating realistic facial dynamics and head movements, the paper mentions that it currently only handles the portrait area up to the torso and does not extend to the full body. The coordination of full-body movements is necessary to achieve more natural and realistic virtual characters. 2. The paper mentions that, despite using a 3D latent representation, a more explicit 3D facial model is not employed, which may lead to some artifacts caused by neural rendering, such as texture adhesion issues.

Summary

This paper introduces a two-stage talking head method that can generate impressive talking faces. It includes 1) A diffusion-based model to generate implicit facial dynamics and head movements from audio and additional conditions. 2) A modified 3D-aided face reenactment decoder for generating faces from latent space. This method delivers high video quality with fast inference speed

Strengths

1. Although many works utilize diffusion models to map audio to intermediate facial features, VASA-V1 demonstrates excellent engineering and generation capabilities, achieving appealing results. 2. This method surpasses real-time speed at a resolution of 512x512, with fast startup cost and ID switching speed, leaving an impressive effect. 3. The method outperforms existing comparative methods in terms of visual effects and numerical results for video realism and audio-visual synchronization.

Weaknesses

1. The 3D-aided face reenactment framework stage should be crucial for the overall method. However, some descriptions are too brief and vague, making them hard to follow. 2. The paper's explanation of the fusion method for condition signals in the Diffusion Transformer is confusing and needs more specific details. 3. The comparison methods in the paper lack implementation details. Considering the different scales of training data for various methods, are the comparison results in the table fair?

Questions

1. How does the number of layers in the 8-layer transformer encoder affect the results in the paper? 2. Will the proposed CAPP in the paper be open-sourced? 3. Does the 3D-aided face reenactment part of the method use distillation to speed up? How can the inference speed of MegaPortraits be accelerated? 4. What are the parameter counts for each stage of the model?

Rating

7

Confidence

4

Soundness

4

Presentation

1

Contribution

3

Limitations

The supplementary materials of the paper include relevant discussions.

Summary

This paper aims to effectively and efficiently generate high-fidelity audio-driven talking head videos. To improve performance and efficiency, the authors have designed a Diffusion Transformer model within the latent space of motion signals, encompassing facial dynamics and head movements. Additionally, they propose a data-driven metric named Contrastive Audio and Pose Pretraining.

Strengths

- The paper applies the diffusion model to the task of generating audio-driven talking head videos, innovatively defining the diffusion model within the latent features of motion rather than those of the image, which is quite interesting. - The paper is well-written and easy to follow, with detailed experiments that convincingly demonstrate the effectiveness of the proposed method.

Weaknesses

The primary concern is the paper's contribution, as the realism and liveliness of the generated videos could be attributed to the performance of MegaPortraits. MegaPortraits' encoders effectively learn latent motion and appearance representations, supported by robust 3D warping generators and an image generator that ensures high-quality outputs. VASA-1, in a way, learns to generate latent motion representations akin to those in MegaPortraits through audio inputs. Despite this dependency, the method performs well overall. Therefore, my overall assessment leans towards accepting it, albeit with some reservations.

Questions

- At inference time, how does the model generate condition signals like the main eye gaze direction and head-to-camera distance, given that the driving signal is only audio? - This does not decrease the novelty of this work. However, a quantitative comparison between VASA-1 and EMO would be quite interesting. Given that EMO's code is unavailable, leveraging the image and audio from their officially provided videos for comparison is encouraged.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes.

Final Rating

Thank you to the authors for their feedback and efforts. After reviewing the rebuttal, I note that the authors have addressed some of my concerns, which leads me to maintain my initial rating. However, I recommend that the final version of the paper include more detailed explanations, particularly regarding the contributions and the reasons behind the statement, "We did find the 3D-aided representation to be promising." These details will enhance the clarity and impact of the work.