DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation

Summary

This paper presents a non-autoregressive direct speech-to-speech translation model based on Directed Acyclic Transformer. Experiments are conducted on CVSS-C and CVSS-T datasets, demonstrating translation performance comparable to the SOTA (Translatotron 2) but more than 16x speed up. It also demonstrates capacity of voice preservation during translation.

Strengths

1) The proposed approach is sound 2) Although the proposed approach is largely a combination of existing works (Directed Acyclic Transformer for NMT and Translatotron 2 for S2ST), its effective adoption in S2ST is novel. 3) The experimental results on translation quality, speed, and voice preservation are all promising. 4) Audio samples accompanying the paper sound consistent with the quantitive evaluation conducted in the paper.

Weaknesses

1) The main results (Table 1) is conducted on English-French translation, which is an easy case as the order of the two languages are very close. It would be helpful to include more results on languages with significant difference on ordering, such as English-German. Compared to autoregressive model, one potential disadvantage of non-autogressive model is it may not handle reordering well.

Questions

1) Voice Preservation evaluation -- from the audio samples, it can been heard clearly as the unit-based approaches don't preserve voice, but DASpeech and Translatotron 1 & 2 can (as stated by the paper). However, looking at the numbers in Table 4, the similarities from Translatotron 1 & 2 are very close to unit-based approach (with DASpeech signicantly higher) -- any explaination?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes.

Summary

DASpeech is a non-autoregressive direct speech-to-speech translation (S2ST) model. The model adopts a two-pass architecture, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. The linguistic decoder is based on the DA-Transformer model, which models translations with a directed acyclic graph (DAG). The acoustic decoder is based on the FastSpeech 2 model. Experiments on the CVSS benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline model. Compared to the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.

Strengths

1. A new direct Speech to Speech translation model that use Conformer Encoder, Non-autoregressive Linguistic Decoder, Directed Acyclic Graph and Non-autoregressive Acoustic Decoder. 2. Efficient architecture for Speech to Speech translation - up to 18.53 speedup compared to the autoregressive baseline model. 3. Get comparable results to state of the art models in terms of translation quality (BLEU) A new direct Speech to Speech translation model: This model is a neural network that can translate speech from one language to another directly, without the need for an intermediate step of transcribing the speech to text. This makes it more efficient and faster than traditional speech translation systems, which require two separate models: one for speech recognition and one for machine translation. The model uses a Conformer encoder, which is a type of neural network that has been shown to be effective for speech recognition tasks. The encoder converts the input speech signal into a sequence of hidden representations. These representations are then decoded by a non-autoregressive linguistic decoder, which generates the translation sentence one word at a time. The model also uses a directed acyclic graph (DAG) and a non-autoregressive acoustic decoder. The DAG is a data structure that allows the model to predict multiple words at a time, which can further improve the efficiency of the translation process. The non-autoregressive acoustic decoder generates the output speech signal one frame at a time, which helps to preserve the naturalness of the audio. Efficient architecture for Speech to Speech translation: The model's architecture is designed to be efficient, both in terms of memory and speed. The Conformer encoder is a relatively lightweight model, which helps to reduce the memory requirements of the system. The non-autoregressive linguistic decoder and acoustic decoder are also designed to be efficient, which helps to speed up the translation process. The model has been shown to be up to 18.53 times faster than the autoregressive baseline model. This makes it a promising candidate for real-time speech translation applications. Comparable results to state of the art models in terms of translation quality (BLEU): The model has also been shown to achieve comparable results to state-of-the-art models in terms of translation quality (BLEU).

Weaknesses

.

Questions

1. The proposed method only evaluate for Fr-En languages. Can you evaluate the proposed method with more languages? 2. What is this model offer other then speed-up? The results is comparable to T2. Can the translation results improve? 3. Can you add ablation analysis of each component?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

.

Summary

The paper describes a non-autoregressive direct Speech-to-Speech Translation (S2ST) model called DASpeech that achieves both high-quality translations and fast decoding speeds. DASpeech uses a two-pass architecture where a linguistic decoder generates the target text, and an acoustic decoder generates the target speech based on the hidden states of the linguistic decoder. The linguistic decoder used is the decoder of DA-Transformer, and the acoustic decoder used is FastSpeech 2. DASpeech achieves much better performance than non-autoregressive baseline and 18.53x speedup compared to autoregressive baseline models and preserves the speaker's voice of the source speech during translation.

Strengths

The paper proposes a non-autoregressive direct S2ST model that achieves both high-quality translations and fast decoding speeds. The experiments show that DASpeech performs better than the baseline NAT model and is comparable to the state-of-the-art AT model in terms of translation quality.

Weaknesses

However, there are some issues in the paper that need to be addressed. Firstly, the novelty of the paper is not significant. The paper is based on the DA-Transformer model heavily borrowed from the reference [11] (i.e., NAT in text-to-text translation), and there is no breakthrough in terms of technology. Additionally, it is not clear that the proposed model can solve some unique problems in speech-to-speech translation scenario. Secondly, some of the contributions mentioned in the paper are not well-supported by the experiments. For example, the paper claims that DASpeech can achieve comparable or better performance than SOTA, but the experimental results show that it performs worse. Thirdly, the efficiency comparison is not in-depth enough. Although efficiency is one of the major contributions of the paper, there is a lack of detailed analysis and tables to describe the efficiency. Moreover, since the paper is a NAT model, it is not fair to compare its speed with the AT model directly. It would be more reasonable to compare the efficiency with the NAT baseline.

Questions

n/a

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

See weakness

Summary

This paper proposes a method for fast and high quality speech to speech translation. The paper adopts the two-pass decoding framework. The first pass decoder adapts prior work on non autoregressive decoding for machine translation. The second pass decoder reuses fastspeech 2 except that the input are the hidden states obtained by the first pass decoder. The first pass and second pass components are pretrained on the speech to text translation task and the synthesis task respectively. Results show competitive performance compared to the state of the art, with substantial inference speedups. The model is also shown to be able to preserve voice when trained on a corpus with the same voice on the source and target sides.

Strengths

* the approach is effective: competitive wrt SoTA with much faster inference speed * the model can preserve the voice, which is an important feature towards authentic communication * the comparison to previous methods is very thorough as many previous methods are reimplemented/compared to

Weaknesses

There is no specific weakness other than most of the novelty comes from the prior work (DA approach, 2-pass decoding framework, fastspeech 2) and this work very effectively combines this prior work to solve a new task.

Questions

* 227: "We implement the following baseline systems." Would you be able to explain this in more details. Does reimplement mean reusing existing results, using existing implementation and retraining models on different datasets, etc. * 235: "We remove the R-Drop training [28] for simplification." It would be good to justify this choice, especially if it leads to suboptimal results for the corresponding baseline (and also given that there is an existing implementation for that work). * 245: "an autoregressive S2TT model." Would it be possible to add details on that S2TT model? Is is a transformer? A particular existing model (if so, add citation, etc.)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The limitations and broader impacts section is very effective in addressing limitations of the work and potential risks.

Summary

The paper introduces DASpeech, a directed acyclic transformer model for fast and high-quality speech-to-speech translation (S2ST). DASpeech employs a two-pass architecture, using a linguistic decoder based on DA-Transformer and an acoustic decoder based on FastSpeech 2. The proposed model captures the complex multimodal distribution of the target speech and achieves comparable or better performance than Translatotron 2, with significant speedup compared to autoregressive models. DASpeech also demonstrates the ability to preserve the source speaker's voice during translation.

Strengths

By incorporating the DA-Transformer and FastSpeech 2, DASpeech leverages the strengths of these models to capture both linguistic and acoustic diversity, resulting in high-quality translations.

Weaknesses

1. The novelty is not enough, the method of this article seems to just learn from the DAT method of NMT and use it in ST 2. While the paper addresses the challenges of high decoding latency in S2ST models, it would be beneficial to provide more detailed analysis or experimentation on the trade-off between translation quality and decoding speed in DASpeech.

Questions

Could the authors elaborate on any specific differences between DASpeech and the DA Translation particularly?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

thank you for the answers

Thank you for taking the time to reply to my review. Regarding 1. and 2. I would suggest for extra fairness in comparison to prior work to: * if the language pair setup is the same, to not only report the numbers obtained by running an existing implementation on the setup but also to report existing numbers from the literature. * similarly, regarding R-Drop, I would suggest reporting the numbers that use it as well as reporting numbers obtained by using it in the reimplementation/rerunning of experiments. Best