GAN-Based Fine-Grained Feature Modeling For Zero-Shot Voice Cloning

With the continuous development of deep learning and speech signal processing, speech synthesis technology has greatly improved in naturalness and comprehensibility, and many application technologies such as artificial intelligence voice assistant and personalized navigation have been widely used in real life, and the demand for personalized speech synthesis is increasing.Personalized speech synthesis requires models that can achieve speech timbre migration, also known as speech reproduction, with only a small number of target speaker speech samples.However, since human speech is highly expressive and contains rich information, including speaker identity information, prosody, rhythm, emotion and other factors, the limited speech data will lead to poor similarity and rhythmic performance of the model-generated speech, and the model needs to be fine-tuned to improve the quality of the synthesized speech.Therefore, personalized speech synthesis with few samples is a very challenging task.To achieve the goal of speech cloning, this paper proposes a personalized speech synthesis method based on FastSpeech2.By using fine-grained feature modeling module containing prosody extractor and prosody predictor, and a training strategy based on Generative adversarial network (GAN) and meta-learning, it is realized that personalized speech with high similarity and naturalness can be generated with a very short reference audio.The subjective and objective experiments also demonstrate that the model proposed in this paper can achieve high quality speech replication without fine-tuning the model under a few or even a single reference audio of the target speaker.