CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

Summary

The authors introduce the CoVoMix framework for human-like monologue and dialogue generation. They point out the shortcomings of the previous multispeaker dialogue systems that they are less explored, and there are lack of high-quality, spontaneous conventional datasets. The proposed CoVoMix achieves high naturalness and zero-shot speaker similarity in both monologue and dialogue generations with its proficiency in the fluency of dialogue turn-taking and spontaneous behavior generation.

Strengths

1. The paper overall is easy to follow. 2. It is the first attempt to propose zero-shot, human-like, multi-talker conversational mixed speech generation with a simultaneous multi-stream semantic token prediction and a multi-talker flow-matching-based acoustic model. 3. The paper introduces dialogue evaluation metrics: Turn-taking Statistics, Para-linguistic Behaviors, and Speech Consistency. 4. The example demo video extensively shows the generated conversation with naturalness and intelligibility.

Weaknesses

1. While the paper shows the objective and subjective evaluation results for monologue and dialogue generation, it does not explicitly show the performance comparisons between the previous literature. The authors may compare the proposed model with previous dialogue papers. 2. The authors address that they employ the Fisher dataset which is curated for robust speech recognition. In order to generalize the proposed model, I would suggest the author to utilize one or two more datasets to verify the effectiveness of the proposed model.

Questions

Please refer to the Weakness section.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations in that they observed instances of words being omitted or duplicated occasionally in synthesized speech.

Summary

This study proposes a personalized speech synthesis model capable of generating monologues or dialogues. The study achieves this goal through the development of a text-to-semantic token model and the conversion of semantic tokens into mel-spectrograms. By utilizing the Fisher dataset, which contains natural speech characteristics, the proposed model is able to produce naturalistic utterances that include paralingustic components such as laughter and coughing.

Strengths

1. Dialogue generation, which involves converting given conversations into speech, has not been sufficiently explored. This study implements a dialogue generation model that not only addresses this gap but also incorporates personalization features. 2. Various metrics that facilitate evaluation have been devised. 3. The included demo provides a convenient way to distinguish the strengths of their model, and the various figures in the paper help facilitate understanding of the text.

Weaknesses

1. The task in this study is similar to existing dialogue-to-speech tasks, with the addition of personalization features. However, the method of adding personalization does not appear to be novel or specifically tailored for dialogue. Instead, it seems to be a combination of existing approaches. - The text-to-semantic token approach has already been widely adopted in many previous studies, particularly with recent advancements in personalized speech synthesis. The use of an autoregressive method, as employed in this study, has been extensively researched. The authors' contribution appears to extend this to dialogue data. However, the approach of using an autoregressive text-to-semantic prediction model for dialogue has already been utilized in Soundstorm. - The acoustic model handles the personalization aspect and shows a structure almost identical to Voicebox. Similarly, apart from an increase in channels compared to the original Voicebox, there seem to be no additional distinguishing features. 2. The paper mentions that Soundstorm generates in a sequential manner, but it should be noted that Soundstorm also uses an autoregressive model for text-to-semantic conversion and a non-autoregressive model based on MaskGIT for the acoustic model, similar to the approach in your model. Therefore, the statement in the paper that "generated in a sequential manner and thus sounds less realistic" could equally apply to CoVoMix. Concerns are raised that issues such as spontaneous behaviors might stem from the differences in the data used for training rather than the proposed method itself. Given that Soundstorm is one of the few spoken dialogue generation models that support personalization, it seems necessary to compare models trained on the same data.

Questions

1. I am curious if the HuBERT tokens used in this study contain any speaker information at all. Specifically, when performing personalization in the acoustic model, could any speaker information potentially present in the HuBERT tokens degrade the personalization performance? 2. To model laughter, it seems that the semantic tokens must include information about laughter. I am curious whether the self-supervised model that incorporates such laughter information in the semantic tokens is limited to the Fisher dataset, or if the speech tokenizer is designed to work robustly with other datasets as well. 3. Unofficial Implementations of Soundstorm have been made publicly available on the web. For example, https://github.com/ZhangXInFD/soundstorm-speechtokenizer offers an acoustic model capable of personalized speech synthesis using semantic tokens, similar to your approach. I'd like to ask if you have considered comparing your model with Soundstorm using publicly available implementations.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

They addressed these aspects in the conclusion, limitation, future work and broader impacts section.

Summary

This paper proposed CoVoMix, a zero-short TTS model for multi-speaker conversations. CoVoMix consists of a multi-stream text-to-units model, a flow-matching acoustic model for mixed-speaker spectrogram generation and HiFiGAN vocoder for waveform generation. The major contribution of CoVoMix is that it is one of the first attempts to generate natural multi-talker conversations in zero-shot manner and according to experimental results and demos, CoVoMix is able to synthesize natural pause, overlap and laughter in the conversation.

Strengths

1. CoVoMix is able to zero-shot generate natural two-speaker dialogue from text without additional input on spontaneous behaviors like laugh. Objective and subjective evaluations further substantiate this. 2. This line of work is a crucial step towards natural human-machine dialogue. There are prior TTS works like CHATS, which also support natural turn taking, and CoVoMix further extends it to support zero-shot, so that user can designate the speaker's voice.

Weaknesses

1. Unclear writing in the method section. Line 149-152, author says "we divide the semantic embedding into two distinct segments in the final linear layer of the decoder", does it mean dividing each embedding into two or dividing the embedding sequence into two halves? 2. Insufficient baseline comparison in the experiments. CoVoMix is only compared to CoVoSingle and a baseline sharing similar architecture. Though many of the prior works are not public, there are open-sourced TTS models like SoundStorm, and public but not open-sourced models like GPT-4o. More baseline comparison will provided readers a better sense of CoVoMix's performance. 3. The authors use speaker diarization for turn-taking statistics, but not for speaker similarity in dialogue, so we basically don't know whether it can recover target speaker's voice for dialogue generation. 4. Speech consistency is only evaluated based on speaker similarity instead of flow of speech, whether laughter is appropriate and etc.

Questions

1. Clarify line 149-152. 2. The author mentions that "the potential errors in speaker diarization could impact the fairness of the comparison", is there any examples or numbers that can be shared so we better understand the situation? also, why is it hard for human evaluation? 3. In speech consistency, the CoVoMix shows better consistency of different utterances than CoVoSingle. But CoVoSingle synthesizes each sentence given same target speaker prompt right? How can this be? Or is this speaker similarity measuring characteristics beyond speaker?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

1. CoVoMix is only applied to two-talker scenario instead of more. 2. CoVoMix lacks the option for personalizing turn-taking. The turn-taking phenomenon itself varies from person to person and even the same person in different moods.