Combining speakers of multiple languages to improve quality of neural voices

In this work, we explore multiple architectures and training procedures for\ndeveloping a multi-speaker and multi-lingual neural TTS system with the goals\nof a) improving the quality when the available data in the target language is\nlimited and b) enabling cross-lingual synthesis. We report results from a large\nexperiment using 30 speakers in 8 different languages across 15 different\nlocales. The system is trained on the same amount of data per speaker. Compared\nto a single-speaker model, when the suggested system is fine tuned to a\nspeaker, it produces significantly better quality in most of the cases while it\nonly uses less than $40\\%$ of the speaker's data used to build the\nsingle-speaker model. In cross-lingual synthesis, on average, the generated\nquality is within $80\\%$ of native single-speaker models, in terms of Mean\nOpinion Score.\n

cs.CL cs.SD eess.AS Speech Recognition and Synthesis Neural Networks and Applications Speech and Audio Processing

Paper

Similar papers

Combining speakers of multiple languages to improve quality of neural voicesJavier Latorre, Charlotte Bailleul, Tuuli Morrill et al.2021 In Library

Efficient neural speech synthesis for low-resource languages through multilingual modelingMarcel de Korte, Jaebok Kim, Esther Klabbers2020 · arXiv (Cornell University) · 23 citations In Library

Training Multi-Speaker Neural Text-to-Speech Systems using Speaker-Imbalanced Speech CorporaHieu-Thi Luong, Xin Wang, J. Yamagishi et al.2019 · Interspeech · 24 citations In Library

Cross-Lingual, Multi-Speaker Text-To-Speech Synthesis Using Neural Speaker EmbeddingMengnan Chen, Minchuan Chen, Shuang Liang et al.2019 · Interspeech · 61 citations In Library

Efficient Neural Speech Synthesis for Low-Resource Languages Through Multilingual ModelingMarcel de Korte, Jaebok Kim, Esther Klabbers2020 In Library

Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice CloningYu Zhang, Ron J. Weiss, H. Zen et al.2019 · Interspeech · 205 citations In Library

Building a mixed-lingual neural TTS system with only monolingual dataLiumeng Xue, Wei Song, Guanghui Xu et al.2019 · Interspeech · 30 citations In Library

Data Augmentation based Cross-Lingual Multi-Speaker TTS using DL with Sentiment AnalysisB. Lalitha, V Madhurima, Nandakrishna Ch et al.2023 · ACM Transactions on Asian and Low-Resource Language Information Processing In Library

Multilingual Text-To-Speech Training Using Cross Language Voice Conversion And Self-Supervised Learning Of Speech RepresentationsJilong Wu, Adam Polyak, Yaniv Taigman et al.2022 · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · 16 citations In Library

Cross-lingual Multispeaker Text-to-Speech under Limited-Data ScenarioZexin Cai, Yaogen Yang, Ming Li2020 · arXiv (Cornell University) · 9 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal