Summary
The paper presents Style TTS 2, a new TTS model with three main innovations 1) diffusion of a latent variable to capture style (everything in the speech signal that’s not enumerated by a phone sequence t and speaker variable c), 2) the use of a pretrained speech language model (SLM) as a discriminator and 3) a novel formulation of a differentiable alignment function. The authors also enable a new joint training curriculum. These three innovations result in quality that surpasses existing available TTS approaches including VITS, VALL-E, Natural Speech and YouTTS.
Strengths
The paper is well written, and, for the most part, easy to follow. Each of the novel components are clearly described, well motivated and contextualized with other comparable approaches. Also, the quality of the samples is quite good.
Weaknesses
Given that zero-shot speaker adaptation is extremely ripe for misuse as Societal Impact and Potential Harmful Consequences
per https://neurips.cc/public/EthicsGuidelines, I would think the risk and mitigation section of this presentation should be more than 2 sentences.
Section 3.2.1 using less resources is highlighted as a contribution of Style TTS 2 – does this include the “phoneme-level BERT pretrained on extensive corpora of Wikipedia articles”? Similar question concerning Section 3.2.3 and the use of WavLM pre-trained on 94k hours of data? It seems like this assessment is limited to the resources required for the model itself, but does not include components like the discriminator and text embedding.
Some areas to tighten the presentation and contribution in the upfront material of the paper.
Introduction: the first paragraph establishes the areas of improvement for TTS as diverse, expressive speech, robustness to OOD text, and requirements of massive datasets for performing zero-shot TTS. In the last paragraph of the introduction the contributions are oriented about quality, and novel use of SLMs. However, these are inconsistent with the areas for improvement enumerated by the authors. It might be clearer to be specific about which open questions for in the first paragraph are going to be addressed in some form by the proposed technique.
The limitations of StyleTTS are again not the same as those established in the introduction. Namely they are 1) a two-stage training process, 2) limited expressiveness and 3) reliance on reference speech hindering realtime application. StyleTTS 2 addresses these three limitations, unifying the training process, and reducing reliance of reference speech while improving OOD performance and expressiveness. While this is compact and consistent as an improvement ot StyleTTS, it does not address the broad limitations of TTS established by the authors.
Section 3.2 and 3.2.1 one of the advantages of Style TTS 2 is described as an E2E training process that jointly optimizes all components. This is presented in contrast to STyle TTS which required a two-stage training process (Section 3.1). However, later in Section 3.2.1 (line 142 and Figure 1a) Style TTS2 uses a two-stage training pre-training and joint training process. If single stage training, or E2E training is meant to be an advantage and contribution of Style TTS 2, the distinction between its two-stage training and the two-stage training used by the original Style TTS needs to be more clearly defined.
Questions
What does it mean for synthetic samples to “surpass” human recordings? Not in the context of how this happens as a score, but what does it say about the evaluation itself if it’s possible for an artificial sample to be more “natural” than a human derived sample which is “natural” with metaphysical certitude. I appreciate the argument about unnatural segmentation of the audio in LJSpeech, but if this is the source of the “unnaturalness” in the test data, it’s maybe not worth making too much of a claim about its value as a quantitative evaluation.
On line 151, the augmented style vector is defined as s = [s_p,s_d], but s_d is not previously defined. Should this be s_a?
Section 3.2.3 - the generator loss being independent of ground truth is true of any GAN training, no? Is there something specific to the use of an SLM discriminator that enables this better than prior work training TTS with GAN objectives?
Section 5.1 “setting a new standard for this dataset”. What is the referent of “this dataset”? Is it the evaluation set of 80 text samples from LIbriTTS and 40 Librivox utterances spoken by the speaker of LJSpeech described in Section 4?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Very briefly in two sentences at the end of the conclusion. (My rating below is contingent on the outcome of an ethics review to assess if the limitations and discussion of ethics concerns are sufficient to comply with https://neurips.cc/public/EthicsGuidelines)