DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation

Response to reviewer u4XD

Thank you for the comments. We would like to address each of them individually in a comprehensive manner. **Q1:** Our model generates time and energy predictions by incorporating information from the entire text. One of the advantages of this approach is its ability to facilitate more stable conditioning frameworks. If the prediction were local, the entire speech would likely be less smooth or stable. Regarding the length regulator, we consistently utilize $\alpha=1$ (normal speed) and extend the signal based on the predictions of the duration predictor to ensure alignment with the actual samples. We will provide further refinement and will change the model architecture figure accordingly. **Q2:** One of the benefits of the diffusive method and working directly on the waveform is the inherent stochasticity of the signal. Therefore, if we partition the signal into independent segments and then concatenate them, there's a possibility that one segment may be spoken in a certain prosody, while another segment may be spoken differently. In such cases, even if the concatenation is of excellent quality, the delivery of words may not be continuous in terms of prosody. Hence, the significance of autoregressive processes and the transfer of information over time. For models that lack stochasticity, the suggested method of concatenation may be suitable, but it comes with the drawback of determinism. While very long signals like 512 words are indeed uncommon in everyday language usage (except in cases of reading an article or a book), it is noticeable that for an E2E model like Wavegrad2, the transition occurred even for texts of reasonable length (32 words). One possible strategy to decrease GPU consumption is to **simplify or substitute the denoiser architecture.** As an illustration, we executed a model with a lower number of residual channels (C=128). This resulted in a reduction in GPU consumption, although not to a sufficient extent. (A figure for the analysis is available on our Git repository) We haven't had the opportunity to conduct a comprehensive comparison regarding quality results, but initial observations suggest promising and good-sounding outcomes. Reducing the frame length could be considered as another method for decreasing GPU consumption, but we will address this in the response to question 3. We recognize the significance of optimizing the synthesis process, and it presents a direction for further research. Another potential research direction involves applying the autoregressive method to other time-dependent domains and assessing its effectiveness. For instance, in video synthesis, it's likely that GPU consumption over time may increase more significantly. Alternatively, the synthesis of DNA chains serves as an example for very long signal lengths. Exploring the application in these contexts presents interesting avenues for further research. **Q3:** Indeed, for the unconditional case, choosing a long frame is more comprehensible since generating a continuous signal requires obtaining all the necessary information directly from the waveform. In the conditional model, choosing the frame length involved several considerations: - Balancing between memory consumption and synthesis time: If the frame lengths were very small, the GPU consumption at any given time would indeed be lower. However, simultaneously, more serial rounds of diffusion would be required to generate a given text. This would result in a longer synthesis time. - While the representation $y^l$ from the past is valuable, from our perspective, it cannot substitute conditioning on past waveform values. This limitation arises from the stochasticity inherent in diffusion models and the waveform - given a certain representation $y^l$, it can be heard in many prosodies. Hence, to achieve natural-sounding continuity, referencing the past waveform is essential. - In certain cases, it might be acceptable to rely on a relatively recent history. However, for long phonemes or silent phonemes (which can extend up to 200 ms or more), ensuring continuity requires information from a more distant history of the waveform signal. - During the model's development, we acknowledged the significance of processing complete phonemes. Consequently, when conditioning synthesis on phoneme segments (leading to shorter conditioning), the model may struggle to discern the relative portion of the resulting phoneme. Our observations indicate that working with full phonemes produces more robust and "healthier" results.

Response to reviewer 9k2u (part 1/2)

Thank you for the comments. You brought up interesting points, and we would like to respond to them comprehensively. **TL;DR:** The models **FastDIFF, DiffTTS, Grad-TTS, PriorGrad, and ProDiff** utilize diffusion-based approaches, functioning either as acoustic models generating a mel-spectrogram (thus requiring a vocoder for waveform synthesis) or as diffusion-based vocoders. A common feature among most of these models is the aim to accelerate the diffusion process, often at the expense of audio quality, the stochasticity of synthesis, or the simplicity of the model. **Q1:** In hindsight, we acknowledge that our comparison to existing models may not have been sufficiently comprehensive. Considering your feedback, we added a comparison and explanation here and in the paper for the models you mentioned and others. Our goal was to develop an E2E model that, in addition to being constructed as a single unit, conducts the synthesis process directly on the waveform. We utilized qualitative metrics for comparison, such as MOS and MUSHRA, but also delved into examining the stochasticity in the model (as a "one-to-many" model) and the controllability of stylistic features. In our responses to question 2, we will also address the synthesis time comparison. **DiffTTS** is among the pioneering models that integrate the Text-to-Speech (TTS) task with diffusion. It utilizes the DDIM sampling equation to enable faster inference speed for Mel-spectrogram generation. **To the best of our knowledge, there is no publicly available implementation of DiffTTS, but based on official samples in the Git repository,** it seems to produce good-sounding results. However, the use of a vocoder occasionally introduces metallic or noisy sounds. **FastDIFF** incorporates noise scheduling algorithms and predicts faster sampling noise scheduling than the one used in training. The model is mainly employed as a vocoder component, but we recognize the potential for **FastDIFF-TTS** as an end-to-end model. **Given its official comparison in the paper and the Git repository, we decided to perform a comparison with ProDiff, a model written by joint authors and published later.** **PriorGrad** serves both as a vocoder and as an acoustic model. It formulates the sampling and training procedures to utilize an adaptive prior derived from data statistics. As an acoustic model, PriorGrad achieves slightly inferior results compared to GradTTS with T=10 diffusion steps. **Consequently, we chose to compare our model's results to the Grad-TTS with T=1000 diffusion steps.** **Grad-TTS** explicitly manages the trade-off between sound quality and inference speed. A notable modification involves initiating the diffusion process from $N(\mu, \sigma)$ rather than white noise. **ProDiff** adopts the generator-based method and also incorporates knowledge distillation and the DDIM method to enable synthesis with only 2 diffusion steps. To compare the quality of **Grad-TTS** and **ProDiff** , we conducted a MUSHRA test following the methodology described in the paper. **The models evaluated were Grad-TTS with T=100 diffusion steps and with ProDiff T=4.** | Model | MUSHRA | | ----------- | ----------- | | GT | 70.0 ± 2.1 | | DiffAR 200 | 66.6 ± 2.4 | | DiffAR 1000 | **67.5 ± 2.3** | | ProDiff | 64.4 ± 2.4 | | Model | MUSHRA | | ----------- | ----------- | | GT | 73.7 ± 2.4 | | DiffAR 200 | **69.4 ± 2.5** | | DiffAR 1000 | 67.7 ± 2.6 | | Grad-TTS | 68.5 ± 2.5 | (Each table is based on a different set of listeners, hence GT is not the same) Despite the advantage of fast synthesis, changing and shortening the diffusion process in both models led to a **decrease in their stochasticity and creativity as one-to-many models.** Regarding ProDiff, different syntheses for the same text were nearly identical in terms of pitch and energy values. In the case of GradTTS, initiating the diffusion process not with white noise and significantly shortening the diffusion process diminishes the stochasticity of the model. In the example provided on the website, we can see that **using T=1000 steps produces almost identical draws. Therefore, even with numerous diffusion steps, the guidance is highly explicit, leaving minimal room for stochasticity.** Furthermore, it is not possible to directly control characteristics throughout the synthesis; instead, changes can only be made indirectly by adjusting phoneme duration. **Grad-TTS mentions the option of an E2E model, but the results are not of high quality for meaningful comparison.**

Response to reviewer 9k2u (part 2/2)

**Q2:** In retrospect, we acknowledge that more explanation could have been given Regarding the synthesis time. In response to your question, we conducted the **RTF values** for the models we compare so far. | Syntax | RTF | | ----------- | ----------- | | Fastspeech2 | 2.79 | | ProDiff | 0.25 | | DiffGAN | 0.06 | | GradTTS | 3.24 | | VITS | 0.09 | | Wavegrad2 | 5.14 | | DiffAR | 31.06 | As mentioned earlier, there is a significant tradeoff, particularly in diffusion models, between synthesis time, synthesis quality, and the level of creativity or stochasticity. The primary focus of our paper was to introduce a novel approach which utilizes the information in the waveform (phase and amplitude). We aimed to develop a model capable of generating natural speech phenomena and facilitates stochasticity as a one-to-many model. Following your comment, we acknowledge a potential research direction, alongside exploring architectural changes, involves maintaining synthesis quality and stochasticity while reducing synthesis time.

Response to reviewer iPTb

Thank you for your detailed and constructive feedback. You provided insightful comments regarding the model architecture, and we would like to respond to them. Our main goal was to develop an E2E model that would be able to work directly on the waveform and produce high-quality speech. When dealing with relatively long texts (10 sec or more) , training or synthesizing the waveform jointly, especially in diffusion models, becomes almost impossible mainly due to computational and memory constraints. In order to do so, we divided the signal into frames (250 msec). We assumed conditional independence (Markovian) between frames in Eq (3) and we will state that in the paper. This is a common assumption in many speech recognition and speech generation models, and empirically produces very high quality speech. Notably, diffusion models like Wavegrad2 and GradTTS also address this challenge by conducting the learning process on segmented portions of the signal, rather than training the model on the entire text in a holistic manner. In other words, their learning process is also local with a relatively limited receptive field, tackling the same computational difficulty. An alternative solution (not ours) would have been training an acoustic model (such as Fastspeech2, DiffTTS, GradTTS, etc.) given a text, it generates linguistic characteristics with a lower dimension of the waveform. The distinctiveness of our model, compared to acoustic models, lies in its choice to skip intermediary linguistic features such as a spectrogram and to forgo the use of a vocoder. The notion is that working directly on the waveform enables the generation of more natural speech. It is important to note that both the duration predictor and the energy predictor are simple networks (comprising few fully connected layers). They cannot be considered decoders on their own. (For instance, in the case of Fastspeech2 as an acoustic model, similar predictors are also used, but they do not independently extract linguistic features from the text). Consequently, DiffAR should not be regarded as a vocoder or as a concatenation of an acoustic model and a vocoder. Therefore, we opted for a qualitative comparison with **Fastspeech2** as an acoustic model and with **Wavegrad2** as an end-to-end model. As for **Tacotron2**- the model functions as an acoustic model. It converts text into a Mel-spectrogram and uses Wavenet as a vocoder for waveform generation. Operating deterministically, it consistently produces a specific signal, for a given text. In qualitative metrics, Fastspeech2, VITS, and GradTTS have shown higher Mean Opinion Score (MOS) values than Tacotron2. Given that our model achieved better MUSHRA / MOS results than these models, we infer that in terms of audio quality, we outperform Tacotron2 as well.

Response to reviewer a3Eo (part 1/4)

Thank you for the detailed comments. You raised interesting points, and we would like to respond to them comprehensively. In hindsight, we acknowledge that our comparison to existing models may not have been sufficiently comprehensive. Considering your feedback, we added a comparison and explanation here and in the paper for the models you mentioned and others. Furthermore, we answer here and in the paper the other concerns you raised. **TL;DR:** The models DiffTTS, GradTTS, and DiffGAN-TTS generate a spectrogram, and subsequently, a vocoder is employed to produce the waveform, while our model is a single end-to-end model that directly generates a waveform. Below (and soon in the paper), we compare our model to DiffGAN, GradTTS, and VITS. The models DiffTTS, VALL-E, Voicebox, and EATS have no implementation, while YourTTS architecture is very similar to VITS and focuses on slightly different tasks than ours, namely Zero-shot-multi-speaker. Details below. **Major:** As mentioned, our objective was to introduce an E2E model that, beyond being constructed as a single unit, conducts the synthesis process **directly on the waveform ​**​(without replying on an intermediate representations such MFCC, Mel-spectrograms), hence can captures prosody (such as vocal fry). In the comparison, we referred to qualitative metrics such as MOS and MUSHRA, but **it was also important for us to examine the stochasticity/creativity** (as a "one-to-many" model) as well as the controllability over stylistic features. The models DiffTTS, GradTTS, and DiffGAN-TTS all fall into the category of diffusion-based acoustic models. These models, given text input, generate a spectrogram, and subsequently, a vocoder is employed to produce the waveform. A common characteristic among most of these models is the desire to accelerate the diffusion process, often impacting audio quality, the stochasticity of synthesis, or the model's complexity. **DiffTTS** is among the pioneering models that integrate the Text-to-Speech (TTS) task with diffusion. It utilizes the DDIM sampling equation to enable faster inference speed for Mel-spectrogram generation. **To the best of our knowledge, there is no publicly available implementation of DiffTTS, but based on official samples in the Git repository, it seems to produce good-sounding results.** However, the use of a vocoder occasionally introduces metallic or noisy sounds. **Grad-TTS** explicitly manages the trade-off between sound quality and inference speed. A notable modification involves initiating the diffusion process from $N(\mu, \sigma)$ rather than white noise. **DiffGAN-TTS** achieves a significant reduction in synthesis time by employing adversarial training of a GAN, which can occasionally introduce instability in the training process. To compare the quality of **GradTTS** and **DiffGAN-TTS,** we conducted a MUSHRA test following the methodology described in the paper. **The models evaluated were GradTTS with T=1000 diffusion steps and DiffGAN-TTS with T=4.** | Model | MUSHRA | | ----------- | ----------- | | GT | 73.7 ± 2.4 | | DiffAR 200 | **69.4 ± 2.5** | | DiffAR 1000 | 67.7 ± 2.6 | | GRAD-TTS | 68.5 ± 2.5 | | Model | MUSHRA | | ----------- | ----------- | | GT | 71.2 ± 2 | | DiffAR 200 | **69.5 ± 2.1** | | DiffAR 1000 | 68.4 ± 2.2 | | DiffGAN-TTS | 68.0 ± 2.2 | (Each table is based on a different set of listeners, hence GT is not the same) **We also investigated the stochasticity of Grad-TTS and DiffGAN-TTS** by generating five syntheses of the same text. The charts illustrating these comparisons are available on the website and in the article. **In both cases, the energy and pitch values were found to be the same or nearly identical for different samples.** This indicates that, despite being diffusion models, the stochasticity in these models is reduced or non-existent. While the small manifold of the spectrogram compared to that of the waveform might contribute to the reduced stochasticity, it may not be the sole factor influencing the phenomenon. In the case of GradTTS, initiating the diffusion process not with white noise and significantly shortening the diffusion process diminishes the stochasticity of the model. In the example provided on the website, we can see that using **T=1000 steps produces almost identical draws. Therefore, even with numerous diffusion steps, the guidance is highly explicit, leaving minimal room for stochasticity.** DiffGAN-TTS drastically reduces synthesis time by minimizing the number of diffusion steps, **resulting in an outcome that closely resembles a deterministic model.** For the models DiffTTS, GradTTS and DiffGAN-TTS, explicit control over signal properties. While in some cases, phoneme duration can be indirectly adjusted, it remains the extent of controllability.

Response to reviewer a3Eo (part 2/4))

We turn now to the models - **EATS, VITS, YourTTS, VALL-E and Voicebox**, we would like to conduct a similar comparison. EATS, although a pioneer in E2E models, exhibits performance below the baselines. Moreover, it has a relatively complex architecture and is challenging to reproduce. Due to the absence of a public implementation, we decided to focus on the other models. **VITS** and **YourTTS** share a similar E2E architecture utilizing VAE, Normalizing Flow, MAS algorithm, and adversarial training for the TTS task. While VITS addresses the TTS scenario, YourTTS focuses on the Zero-One-shot task with multiple speakers and languages. Both models employ a stochastic time predictor to synthesize speech with diverse rhythms. However, for the same text the samples exhibit energy and pitch with very similar but shifted frames, as evidenced by examples attached to the paper and the Git repository. We should note that in the training of VITS and YourTTS, the latent variables z are based on linear spectrograms obtained by stft, so the synthesis doesn't directly work on the waveform. The reconstruction loss also involves the spectrogram, and the adversarial loss on the output may reduce stability during learning. **In terms of qualitative metrics,** we compared VITS as it is more closely related to the examined task. We will elaborate on the extension to multiple speakers in our responses to the questions. | Syntax | MUSHRA | | ----------- | ----------- | | GT | 74.9 ± 2.2 | | DiffAR 200 | 69.1 ± 2.2 | | DiffAR 1000 | **71.5 ± 2.2** | | VITS | 69.0 ± 2.3 | **VALL-E** and **Voicebox** represent state-of-the-art models trained on large-scale datasets. Voicebox can be utilized as a zero-shot TTS model by infilling missing parts in the Mel-spectrogram and as an acoustic TTS model. VALL-E is an E2E model that uses a Codec code, working indirectly on the waveform, to generate speech and handle the zero-one-shot task. While these models excel on large-scale datasets, replicating their success on LJspeech and VCTK proved challenging. Voicebox lacks an available implementation, and unofficial samples suggest that VALL-E faces difficulties in generalizing well when trained only on Lj-speech. **We want to emphasize that our primary focus was introducing a novel approach and exploring the benefits of working directly on the waveform.** We acknowledge that with more resources, it could be feasible to model additional phenomena like background noise, potentially leading to improved results.

Response to reviewer a3Eo (part 3/4)

**Minor** - We have attached **RTF values** compared to the other models. As mentioned, employing DDPM+AR and working directly on the waveform comes with a cost in terms of synthesis time. Recognizing this tradeoff, we acknowledge the potential for follow-up work to explore methods for reducing processing time. One approach could involve modifying the denoiser architecture to expedite the process. Alternatively, considering more conventional methods, such as DDIM, presents its own set of advantages and disadvantages. | Syntax | RTF | | ----------- | ----------- | | Fastspeech2 | 2.79 | | ProDiff | 0.25 | | DiffGAN | 0.06 | | GradTTS | 3.24 | | VITS | 0.09 | | Wavegrad2 | 5.14 | | DiffAR | 31.06 | - As previously highlighted, our article primarily addresses single-speaker synthesis. Nevertheless, we recognize the potential for extending the model to accommodate multiple speakers and varied prosodies. One approach involves leveraging the model's versatility by integrating the speaker's embedding into the synthesis process. **We have explored this option using Titanet embedding and datasets like VCTK/LibriTTS** with detailed insights provided on GIT regarding results and architecture structure. Additionally, a combination of classifier guidance or free classifier guidance presents an alternative method for introducing features such as emotion.

Response to reviewer a3Eo (part 4/4)

**Questions** **Q1:** As mentioned earlier, we acknowledge the possibility of having conducted a more comprehensive comparison with existing TTS models and regret not doing so sooner. We hope that the attached explanation provides a broader perspective on how our work is positioned within the current TTS landscape. It's important to emphasize that the choice of wavegrad2 was not arbitrary; it is an end-to-end model utilizing diffusion, which was published around the same time as VITS. However, due to its parallel signal generation, it has inherent limitations, resulting in significantly lower audio quality compared to our model. Recognizing the importance of an extensive comparison with more advanced models than fastspeech2 and wavegrad2, we have included additional results. **Q2:** The training of the model, following the standard procedure for diffusion models, is stable (A diagram of DiffAR training is provided on git). Currently, we have opted not to backpropagate the predictors at this stage, but this remains a possibility. **Q3:** Unfortunately, pitch contour cannot be predicted only from the text as it is related to the way the speaker produces the speech, and changes each and every production. While our intention was to showcase the model's capacity to be conditioned with additional information and control pitch, we were careful not to rely too much on the pitch predictor's performance if it didn't meet our standards. As a result, we introduced DiffAR+P as an option without directly comparing it to other models. **Q4:** In the unconditional task meaningful words are rarely produced. Sometimes they are formed by chance or sound like made-up words, (a phenomenon which is also discussed in WaveNet and WaveTacotron). Even with a high-dimensional manifold (1000 ms), a diffusion model trained on a diverse set of words struggles to reproduce them without guidance. The likelihood of generating meaningful words increases when the dataset has a limited vocabulary, with short words that remain untruncated during training and are repeated sufficiently. An example of this can be seen in DiffWave with the SC09 dataset.