Rebuttal by Authors (Cont'd)
This comment follows from **Rebuttal by Authors**.
> A simple CG method in En-Es can bring an improvement about 4.5 BLEU (Table 2, Line 7), any reasons behind it?
>
We are also intrigued by the significant improvement from CG on the En-Es dataset. We suspect this may be due to two factors: (1) the relatively small size of the En-Es dataset, which contains fewer than 80k data points for training, and (2) the longer sequence lengths of Spanish speech, averaging 256 tokens in the training set and 308 in the test set, which worsen the acoustic multimodality issue. CG operates by constraining the model to learn exclusively from the distribution of the target (Spanish) speech units, leading to a more consistent distribution for translation during inference. **This is particularly beneficial for longer target sequences.**
To further support our claim, we recently extended our exploration of CG to English-German translation using the WMT14 dataset. Following the same settings as the CMLM model [5] and incorporating our proposed CG for the MT task, we observed that while the CMLM+CG model does not show significant improvement using the full WMT14 dataset, it does enhance performance by approximately 1 BLEU point when the training data is filtered to only include sequences longer than 60 tokens and the test data to only include sequences longer than 30 tokens, as demonstrated below:
| Model | T=5 | T=10 | T=15 |
|-------------------|------|------|-------|
| CMLM | 19 | 20.2 | 20.5 |
| CMLM+CG | 19.8 | 20.8 | 21.04 |
For speech-to-speech translation that has much longer unit length, the effect of CG will be more noticeable, and therefore achieving large improvement over the baseline.
> ….when there is no perturbation, BL-Dn degrades dramatically. What is the reason for this? Is there a borderline value of 𝑇 that separates these two regions?
>
We believe it is dataset-dependent to judge whether there is a borderline value of T that has drastic improvement. We performed more ablation studies using T=10 and T=30 as start time for noise injection and the result is shown below:
| Start Time | Acc-Rec | BL-Dn |
| --- | --- | --- |
| T=10 | 93.8 | 15.98 |
| T=30 | 91.6 | 17.29 |
As indicated in the result table, with a very small T=10, the reconstruction accuracy is approximately 0.94, leading to a downstream ASR-BLEU score of 15.98. This represents a significant improvement compared to the baseline CMLM result of 12.58. The accuracy of 0.94 suggests that around 6% of tokens differ after reconstruction, which is likely due to the regularization effect of the VAE model, as T is too small for the diffusion model to cause reconstruction errors. This implies that the Spanish speech dataset may contain significant acoustic variations and noisy information, which the VAE model can already partially remove. With a larger T, the diffusion model further cleans up the representation, resulting in even higher downstream results (as observed from T=10 to T=100, where there is an increase in ASR-BLEU score).
-----
References:
[1] Lee et al., (2022). Textless speech-to-speech translation on real data.
[2] Huang et al., (2023). TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation
[3]Chen et al., (2024). Robust Classification via a Single Diffusion Model
[4] Duquenn et al., (2021). Multimodal and Multilingual Embeddings for Large-Scale Speech Mining
[5] Ghazvininejad et al., (2019). Mask-Predict: Parallel Decoding of Conditional Masked Language Models
[6] Richter et al., (2023). Speech Enhancement and Dereverberation with Diffusion-based Generative Models