TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

Summary

This paper presents an end-to-end framework for speech-to-speech translation that preserves speaker and voice characteristics while leveraging unsupervised training. The authors also refine the speech tokenizer by distilling semantic information and enhance the sampling mechanism to support textless NAR acoustic modeling.

Strengths

The authors have made significant improvements throughout the S2ST pipeline, resulting in a solid contribution. They have also conducted experiments with various datasets and compared their results to those of capable models such as Seamless for S2ST and Encodec for Codec, achieving reasonable improvements. Although some of the experimental results are mixed, given the limited computational resources (only 32 GPUs) and datasets (only 5k hours), the improvements are still impressive and demonstrate the effectiveness of the proposed methods.

Weaknesses

1. The paper is well-written and informative, but it covers a wide range of topics, which can make it overwhelming to read. The introduction and subsequent sections could be restructured to better emphasize the different contributions. For example, the introduction covers four bullet points, where the first two are modeling designs for S2ST and the last two are Codec related. Then in the method section (sec 3), everything is presented together. It would be helpful to dissect section 3 into two main sections and add pointers from the introduction sections to improve clarity. Additionally, consider adding visualizations for the distillation strategy in the main content to further illustrate the proposed methods. 2. My major concern with the proposed framework is the inference speed. It would be helpful to include an analysis of the inference speed of the proposed architecture, as the use of an autoregressive decoder for predicting Codec tokens may significantly slow down the process, even with deduplicated units.

Questions

1. Following up on my second point in weaknesses, could you provide some comparisons with the UnitY system or others in terms of inference speed? Additionally, there are several direct S2ST translation works [1,2,3] that utilize non-autoregressive modeling on the CVSS dataset, and it might be worth comparing with such methods. 2. I am a little confused about Figure 2's target clip encoding. What is the motivation for using that representation for the [sep] token? From your description (lines 116-120), I do not find any mention of the use of the target speech clip. Can you explain why the pooled representation of that clip is used and what happens during inference when such clips do not exist? [1] Lee et al., (2022). Direct speech-to-speech translation with discrete units [2] Huang et al., (2023). Transpeech: Speechto-speech translation with bilateral perturbation [3] Tan et al., (2024). DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

No concerns on the limitations.

Summary

This paper proposes a end-to-end speech translation framework that adds several improvements to the existing textless encoder decoder speech translation architectures.

Strengths

1. There are several interesting ideas in this paper, such as the isochrony embedding and layer beam search. 2. The proposed method out-performs one of the previous SOTA on En-Fr translation tasks.

Weaknesses

1. The writing needs to be improved. There are some typos here and there (e.g. line 280 page 7). And the writing clarity can be improved. 2. There is a lack of ablation study to investigate how much does each design choices affect the model performance. 3. The evaluation has only been performed on translation tasks between En and Fr. It's not clear if the model will perform well between other language pairs, especially between English and non-European languages. 4. For isochronic translation, the model can either choose to translate as usual and then adjust the speech rate to fit into the timing boxes, or the model can be more smart in translating in the optimal way to reflect the isochrony constraint. I couldn't find any discussion regarding this aspect in this paper.

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Adequately discussed.

Summary

This paper proposes TransVIP, a speech to speech translation model with voice and isochrony preservation, i.e. pauses and segment durations are preserved between the source and the target, for example for automatic dubbing applications. The proposed model architecture is modular, with multiple encoders for semantic, acoustics and isochrony information, intermediate text output, a non auto regressive acoustic model to generate a sequence of codes which are then decoded to produce a waveform. Contributions include the modular architecture trained end to end, the model capabilities, in particular isochrony preservation. Experiments show that the proposed approach is either competitive with or outperforms a strong baseline (SeamlessExpressive) on translation quality, speaker and prosody similarity while substantially improving isochrony preservation.

Strengths

* This is important research work with important applications such as automatic dubbing, especially since most of the research on speech to speech translation does not emphasize isochrony preservation. * The proposed architecture is novel * The empirical results are positive and compare to a strong baseline

Weaknesses

* The empirical evaluation could be improved: validate the method in one more language pair and optionally compare to cascaded solutions. * In terms of presentation, the paper could be more self contained and include ablations in the main part of the paper vs the appendix. The paper could also benefit from proofreading (there are quite a few typos).

Questions

“acoustic information(A)”: the title only talks about voice preservation but the evaluation also measures prosody preservation. Could the authors clarify which components of the model are specifically designed to preserve prosody?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors acknowledge that only the French-English pair is involved but we still consider this a weakness for a translation related paper.

Summary

The paper introduces TransVIP, a novel speech-to-speech translation system designed to maintain both the speaker's voice characteristics and isochrony during the translation process. TransVIP simplifies the complex task of speech-to-speech translation (S2ST) by breaking it down into two sequential subtasks while retaining an end-to-end framework. It conditions the generation of the target speech not just on semantic information, but also on isochrony and acoustic details extracted from the source speech. The paper demonstrates the effectiveness of TransVIP through experiments on French-English translation, showing superior performance compared to state-of-the-art models.

Strengths

1. The motivation is interesting. The recent studies have paid attention to voice preservation during speech-to-speech translation (S2ST), and this paper further proposes to preserve the isochrony information for ideal speech translation effects. This may provide useful insights for future work. 2. The paper decouples the S2ST model into multiple modules and offers several significant innovations. 3. The proposed method achieves new state-of-the-art performance.

Weaknesses

1. The title may not be fully representative, as only a part of the innovation focuses on voice and isochrony information preservation. As shown in Section 3, only the first subsection is closely related to the title. While the latter subsections present good innovations, the overall relevance among them could be strengthened. 2. The experimental comparison is limited. This paper focuses on voice and isochrony preservation, but does not provide any comparison with related work, like VALL-E X and PolyVoice. As Seamless does not consider voice information, the real advantage of voice preservation is unclear. 3. This paper proposes multiple innovations with many techniques. However, there are few ablation studies to analyze the individual components. There is also a lack of in-depth analysis on the design choices for voice and isochrony preservation.

Questions

The paper is difficult to read, as some technical details are missing, making it challenging to fully understand the design. For example, it is unclear how the acoustic encoder learned from scratch can be expected to extract the desired acoustic information.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Not applicable

Official Comment by Reviewer JYog

Thanks authors for clarifying a few points mentioned in my review and provide additional data points on the inference speed. I will maintain my score and evaluation.

thanks + questions

Dear Authors, Thank you for the additional experiments! I rechecked my review scores which were already quite high so I'm not planning to modify them. For completeness, it may be good to include both ST + StyleTTS and ST + VALLE-X for both directions. In an updated version, it would also be interesting to discuss the strengths and weaknesses of both systems since the proposed approach is not outperforming the baseline in all categories. Can you clarify why the trend for BLEU is the reverse of the trend for ASR-BLEU? Best, --Reviewer kQBQ

Response

Thank you for the detailed clarification and sorry for the late response. While the explanation has addressed most of my concerns, I still feel that the direct comparison to strong voice cloning systems is not sufficient. The voice information is not the primary focus of the Seamless system. Additionally, I believe the title of the paper is an important consideration. The authors should choose a title that is representative of the work presented, to accurately convey the scope and focus of their research. Overall, I will raise my score to 5, as the response has addressed the majority of my initial concerns.