Summary
With the goal to improve speech translation task, this work, leverages a multi-task training approach optimizing weighted sum of ASR, ST, MT and cross-modality learning (CML) objectives. Using mBART encoder and decoder Transformer blocks, the CML objective is purposed to better align speech and text modalities. CML involves a masked token prediction (MTP), speech to text mapping (STM), and encoder representation matching (ERM) objectives.
Overall, the proposed arc (ComSL) includes a speech and textual transformer blocks, and a two layered bride adapter module. Before the multi-task step, pre-trained models (mBART and Whisper) are fine-tuned using training data for 50 languages, mainly involving CoVoST2 data set. Results are provided for low, medium and high-resource languages show ComSL's performance improvements over recently proposed multi-modal (text and speech) models such as USM and Whisper.
Strengths
- This work, is well motivated suggesting an cross-modality learning objectives, to address the modality gap in the next frontier for a working composite speech and textual model.
- The results for ST are encouraging, performing similarly or slightly better than recently proposed multi-task based speech-text models like Whisper and USM.
Weaknesses
- Although, ComSL is well motivated to focus on bridging the modality gap, the pre-training and fine-tuning using in-domain data doesn't seem to deliver significant improvements. With close results, and lack of significance testing, its rather difficult to interpret if the proposed approach is better performing.
- Given the focus on multi-task learning, the provided evaluation (only for ST task) is unexpected. Adding at least ASR task in to the comparison could have provided more substance to the report.
- Discussion and analysis part could have focused on ST cases (or examples) where CML improves over other models without using the cross-modality objectives. This would show the importance of introducing the objective.
Questions
- What was the motivation for not evaluating other tasks, as this model is trained in a multi-task setting, at least the ASR evaluation makes sense.
- Given the number of objectives involved, particularly for the CML part, how does model training and inference complexity compares?
- Table 1, all models ... are fine-tuned with CoVoST2 data, does this include USM model? if not, what is the justification for a fair comparison?
- Table 1, for Non-E2EST, does it make more sense to use Whisper + and a standard MT model trained separately?
- Given most numbers are quite close, in table 1-3, adding statistical significance could make the results more meaningful, and to the support SOTA assertion.
- L103 fix unfinished sentence
- S3.3 how does the split is formulated for z^s and z^x before passing it to mBART decoder? Obviously the split is necessary to optimize the MTP objective using z^x, have you considered directly using e^x without the concat step?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Limitations are included, reflecting the main content of the work. I suggest authors to consider a separate limitation part and to consider societal impact if available.