Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

Summary

This work presents a text-conditional speech synthesis model with an optimal transport path and conditional flow matching. The model is trained with a large-scale cross-lingual dataset for zero-shot style transfer and content editing.

Strengths

The work adopts a conditional normalizing flow with an optimal transport path for speech synthesis. The proposed model can synthesize and edit the speech with different styles.

Weaknesses

Although this work successfully executes the zero-shot TTS and speech editing, I have doubts on the originality of this work and I think that the authors should have conducted more comparisons with the recently proposed TTS models (not YourTTS, not Vall-E), and speech editing or speech correcting paper. I have some comments on this work. 1. The concept to train the model by infilling speech given audio and text has been already presented in many works. First, SpecAugment [1] presented a time masking for context learning and Conformer [2] successfully adopted it for the ASR task. Self-supervised speech representation also models such as wave2vec 2.0 successfully learned the context with masking. Moreover, there are many similar models which can edit the speech conditional text information in speech editing [3] [4], correcting domains [5] [6], and also in the TTS domain [7] [8]. The authors only adopt the conditional normalizing flow with an optimal transport path for text-to-speech. [1] Park, Daniel S., et al. "Specaugment: A simple data augmentation method for automatic speech recognition." Interspeech, 2019. [2] Gulati, Anmol, et al. "Conformer: Convolution-augmented transformer for speech recognition." Interspeech, 2020. [3] Tae, Jaesung, Hyeongju Kim, and Taesu Kim. "EdiTTS: Score-based editing for controllable text-to-speech." Interspeech, 2022. [4] Wang, Tao, et al. "Campnet: Context-aware mask prediction for end-to-end text-based speech editing." IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2022): 2241-2254. [5] Tan, Daxin, et al. "CorrectSpeech: A Fully Automated System for Speech Correction and Accent Reduction." ISCSLP, 2022. [6] Fong, Jason, et al. "Speech Audio Corrector: using speech from non-target speakers for one-off correction of mispronunciations in grapheme-input text-to-speech." Proc. Interspeech 2022 (2022): 1213-1217. [7] Ao, Junyi, et al. "Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing." ACL, 2022. [8] Wang, Tao, et al. "Non-Autoregressive End-to-End TTS with Coarse-to-Fine Decoding." INTERSPEECH. 2020. 2. Specifically, the diffusion-based model (EdiTTS [3] and Guided-TTS [9]) can edit the speech even without training. The author should have compared the conditional normalizing flow with a diffusion-based model. However, the authors only referred that OT path leads to faster training, faster generation, and better performance compared to diffusion paths. I wonder if the conditional normalizing flow with OT path is better than diffusion-based models. [9] Kim, Heeseung, Sungwon Kim, and Sungroh Yoon. "Guided-tts: A diffusion model for text-to-speech via classifier guidance." ICML, 2022. 3. The audio quality is not good on the demo pages. 4. For a fair comparison, all models should be trained with the same dataset. 5. YourTTS is not a good text-to-speech model for zero-shot text-to-speech models. The audio quality of YourTTS is bad. I recommend training the VITS with the reference encoder you used and the same configuration. In addition, transferring the voice style from a reference audio, not speaker ID, is utilized in many works. For a fair comparison, the same transferring method is used for each model.

Questions

1. The authors utilized a 100 Hz frame rate to extract the high-resolution Mel-spectrogram. For a fair comparison, each model is also trained with the same time resolution for Mel-spectrogram. For example, VITS and YourTTS should be trained with the linear spectrogram extracted at a 100 Hz frame rate for high-quality. However, the details are not described for baseline models. 2. Are there any scenarios for automatic noise removal or editing? In this work, the location of audio for noise removal or editing should be segmented by the user. [5] presented the fully automated speech correction scenario with detection, correction, and generation. It would be nice incorporating ASR with this work for an automatic speech editing system. 3. I think replacing the HiFi-GAN with BigVGAN could improve the audio quality. 4. For the AR model, the dropout in the pre-net of the decoder may improve the diversity of speech. Also, VITS and YourTTS can increase the sample diversity by controlling the temperature T. The details you used should be described.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

1 poor

Limitations

They stated the limitations and potential negative societal impact on Section Conclusion.

Summary

The paper proposes voicebox, a text-guided generative model for speech at scale. By abstracting many speech tasks into speech infilling tasks, voicebox is able to conduct zero-shot text to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation in monolingual or crosslingual scenario. Experiments show that equipped with the advanced technology such as CNF and classifier-free guidance, voicebox show better performance than baseline methods on many tasks,

Strengths

1. The paper abstracts many different speech tasks into speech infilling task. 2. The paper scales up the model to 50K hours of speech and multilingual scenario, where the model is able to perform in-context learning for speech style. 3. The proposed method is well-studied on many tasks including zero-shot TTS, speech enhancement, speech editing and style generation. The experiments on ASR also show the potential value of proposed method.

Weaknesses

About the experiments in Section 5.2, it is unfair to compare VB with speech enhancement model since VB can have access to the text of noised speech, which significantly help the model to generate speech with lower WER. Also, the text of noised speech is usually not accessible in the conventional formulation of speech enhancement.

Questions

1. As Table 5 shows, VB is able to generate speech with different styles. Can this feature be used in Table 6? To be more specific, if VB synthesize more than one sample per text from the Librispeech training set, can the corresponding WER be further reduced? 2. If there is any insight for using Continuous Normalizing Flows to modeling speech infilling rather than other generative models such as diffusion models?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have discussed the limitations in conclusion.

Summary

This paper proposed Voicebox, a non-autoregressive flow-matching model designed to infill speech by leveraging given audio context and text. Notably, Voicebox capitalizes on a substantial amount of data, consisting of 50,000 hours of speech, which contributes to its impressive performance across various speech generation tasks. The model demonstrates notable capabilities in generating coherent and high-quality speech outputs.

Strengths

1. The paper exhibits a comprehensive and well-executed evaluation, yielding impressive experimental results. 2. The authors have effectively presented their work with clear and accessible writing, ensuring ease of understanding for readers. 3. The proposed framework showcases a high level of flexibility, enabling its application to various speech generation tasks and settings.

Weaknesses

1. The demos provided in the supplementary materials are not as satisfying as described in the paper. For example, issues can be found in the perceived speaker similarity in the zero-tts task and the quality of the articulation position of the mask. It would be beneficial if the authors engage in further discussions regarding these phenomena. 2. Given that speech encompasses various components (such as prosody, content, timbre, and noise), it would be better to have a more comprehensive discussion on how Voicebox specifically handles these different aspects.

Questions

It is unclear from the paper what motivated the authors to choose the flow-matching model as the audio model.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors are encouraged to add some discussions about the potential negative social impact.

Summary

This paper proposes VoiceBox, a speech infilling model based on a flow-matching generative model. VoiceBox is trained to fill in masked speech based on unmasked speech and given text, and it can perform various tasks depending on how the mask is applied to the speech during inference. VoiceBox allows for speech editing by masking the speech corresponding to the text portion that needs to be edited, and it can also perform zero-shot TTS by infilling the speech for the desired text. By masking the entire speech during generation, the model can generate voices from various speakers in a speaker-unconditional manner for a given sentence. This paper demonstrates significantly improved performance compared to existing models in various tasks and even shows that a generative model for speech can aid in performance improvement in speech recognition tasks through diverse speech generation.

Strengths

* This paper demonstrates the versatility of VoiceBox across various tasks while achieving impressive performance. * This paper demonstrates that utilizing synthesized speech generated by VoiceBox can improve the generalizability of ASR models, thereby showing its ability to effectively model the distribution of general speech. * In the zero-shot TTS task, VoiceBox particularly outperforms the previous state-of-the-art model, VALL-E, by a significant margin. * The use of a flow-matching generative model enables fast sampling. * The extensive experimental results provide strong support for the advantages of the model.

Weaknesses

* For zero-shot TTS and speech editing, both the duration model and audio model require prompting with the transcript of the reference speech as well as the duration per phoneme. This necessitates the use of MFA (Montreal Forced Aligner) during inference.

Questions

* Despite VALL-E already demonstrating impressive zero-shot TTS performance, VoiceBox appears to have excessively higher SIM-o and SIM-r scores. It is difficult to determine that VoiceBox's zero-shot TTS samples are significantly better than VALL-E when listening to only the VoiceBox samples in the demo. For a one-to-one comparison with VALL-E, would it be possible to provide VoiceBox samples generated with the same prompt as VALL-E's demo samples? * In order to calculate SIM-r as mentioned in the paper, were both the reference speech prompt and all generated samples encoded and decoded using Encodec? If calculated differently, please provide an explanation. Additionally, when measuring SIM-o or SIM-r with the reference speech, was the similarity measured with the speech prompt only or with the entire reference speech? *

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

This paper provides additional experiments on a classifier designed to detect potential misuse in order to mitigate such risks.

Summary

The authors present Voicebox, the text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation.

Strengths

It is interesting that Voicbox performs in-context learning via the masking strategy. Besides, the authors train the model in extensive data and present the SOTA results in several downstream tasks.

Weaknesses

1. One of the weaknesses should be novelty. It sounds like the combination of masked-based A3T and flow-based speech synthesis model. Besides, duration prediction models and classifier-free guidance are not new. 2. Evaluation. Firstly, why present a portion of MOS in Table 2? VITS-LJ and VITS-VCTK are compared as the baseline in the LibriTTS test-other set, but why not train in the LibriTTS dataset instead? The differences in data amount could encounter unfair comparison. 3. Unclear presentation. Voicebox still needs to provide a clear illustration. Do you need tags for different task inferences? A3T finds that it is challenging to generate speech given full-mask samples, and what do you find is the most suitable proportion of masked and unmasked regions?

Questions

1. It could be more natural to use diffusion models since the masking strategy is similar to adding noise, so what is your consideration in using flow-matching models? 2. Citations [42] and [43] are the same

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

/

Thanks for your response

Thank you for your helpful response. I acknowledge that this work first proposes the generalized model for multiple speech tasks. Moreover, I think your model will have a good impact on speech research. That is why the authors should conduct more experiments fairly for future researches who might follow your research. I don't want anyone in the future to claim "VoiceBox experimented like this and we're just following VoiceBox". I still have a concern about the comparisons for each task. I hope that you do not cut corners by just arguing that our framework is novel. >**Zero-shot TTS experiment** I still disagree with your experiments because YourTTS is not a good zero-shot TTS model. The audio quality is not good and the models are trained with different datasets. It is not fair to compare it with your model. The authors also utilize an additional Duration model. >**Diverse speech generation experiment** you just compared the model with VITS-LJ and VITS-VCTK. I could not agree that your model is better than others with the results of this experiment. You should train the model with the same dataset for a fair comparison. >**To generalize speech infilling, any powerful non-autoregressive generative models, including diffusion models, should work. We chose flow-matching (FM) with optimal transport (OT) path because [1] showed that FM w/ OT > FM w/ diffusion > score-matching (SM) w/ diffusion (the typical diffusion model) on training speed and inference compute-quality trade-off. See the comparison in Table 1 and Fig 4-7 in [1].** If you're saying this, adopting the flow-matching (FM) with optimal transport (OT) path is not your contribution. the authors should have compared all scenarios (FM w/ OT, FM w/ diffusion, score-matching (SM) w/ diffusion) to verify that the FM w/ OT is also a better method for speech generative tasks. > **Duration Modeling** I also have an additional doubt on the comparison of different models. In Appendix, the WER results of flow-matching and regression model are almost same in Table B3. This results show that the trained model with your dataset has a just lower WER so you should train the VITS or other models with the same dataset you used. I think a duration modeling with a large-scale dataset improves the pronunciation. For a fair comparison, VITS with a duration modeling should be compared. Specifically, VITS just utilizes a MAS for a efficient training without external duration modeling. In addition, there are many works which utilizes VITS with external duration modeling to improve the performance. [1] Cite as: Ju, Y., Kim, I., Yang, H., Kim, J.-H., Kim, B., Maiti, S., Watanabe, S. (2022) TriniTTS: Pitch-controllable End-to-end TTS without External Aligner. Proc. Interspeech 2022, 16-20, doi: 10.21437/Interspeech.2022-925 [2] Cite as: Lim, D., Jung, S., Kim, E. (2022) JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech. Proc. Interspeech 2022, 21-25, doi: 10.21437/Interspeech.2022-10294 [3] Zhang, Yongmao, et al. "Visinger: Variational inference with adversarial learning for end-to-end singing voice synthesis." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022. [4] Y. Shirahata, R. Yamamoto, E. Song, R. Terashima, J. -M. Kim and K. Tachibana, "Period VITS: Variational Inference with Explicit Pitch Modeling for End-To-End Emotional Speech Synthesis," ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5, doi: 10.1109/ICASSP49357.2023.10096480. Basically, I like the concept of this paper. However, current manuscript does not conducted a fair comparison. I encourage the authors to add additional ablation studies to the paper. But, they did not conduct any experiments I suggested so I could not take any action in this stage.

Two additional experiments to address Reviewer MQeA's new comments

We thank all the reviewers again for spending time carefully reading our responses. We have added two more experiments to address Reviewer MQeA’s follow-up comments, and would like to highlight the additional experiments in the common thread. Please see the details below. --- **1. We present a new experiment here to address the concern on the comparison with YourTTS in terms of data.** We trained Voicebox on LibriTTS with the reduced setup described in appendix Section B3, which is strictly a subset of what YourTTS is trained on (LibriTTS, VCTK, PT, FR SCL). Results suggest that Voicebox are still significantly better | Model | ZS-TTS WER | ZS-TTS SIM-o | | -------- | ------- | ------- | | Voicebox trained on LibriTTS | 2.1% | 0.579 | | YourTTS | 7.7% | 0.337 | --- **2. We present new experiments comparing FM w/ OT, FM w/ diffusion, and SM w/ diffusion. We confirm empirically that FM w/ OT indeed has significantly better training and inference efficiency.** We adopt the same ablation setup as Appendix Section B.3 (with loss computed on all frames and a smaller learning rate, 1e-4, to ensure convergence for all three methods). We vary the number of training and inference steps and show experimental results below: Exp 1: Training for 50K / 100K / 150K updates; Inference with 64 NFEs. **We see FM w/ OT achieves the best performance with 100K training steps, and even outperforms SM w/ diff using only 50K updates.** | Model | ZS-TTS WER (upd=50K/100K/150K) | ZS-TTS SIM-o (upd=50K/100K/150K) | | -------- | ------- | ------- | | FM w/ OT (proposed) | 2.5% / 2.2% / 2.1% | 0.424/0.487/0.508 | | FM w/ diffusion | 76.0% / 3.1% / 2.6% | 0.066/0.344/0.478 | | SM w/ diffusion | 73.3% / 17.4% / 5.1% | 0.062/0.176/0.349 | Exp 2: Training for 150K updates; inference with 8/16/32/64 NFEs. **We see that FM w/ OT can produce good results with just 8 NFEs, while FM w/ diff requires at least 16 NFEs and SM w/ diff requires over 64 NFEs.** | Model | ZS-TTS WER (NFE=8/16/32/64) | ZS-TTS SIM-o (NFE=8/16/32/64) | | -------- | ------- | ------- | | FM w/ OT (proposed) | 2.4% / 2.2% / 2.2% / 2.1% | 0.410/0.481/0.503/0.508 | | FM w/ diffusion | 11.5% / 3.0% / 2.7% / 2.6% | 0.171/0.359/0.447/0.478 | | SM w/ diffusion | 94.5% / 42.3% / 11.5% / 5.1% | 0.054/0.076/0.218/0.349 | Results on other setups show the same trend (all inference NFE and train steps combinations on ZS-TTS and diverse speech generation). We will add to the appendix in the final revision