Scaling Sign Language Translation

Summary

This paper proposes to improve the open-domain sign language translation by scaling pretraining data, model size, and number of translation directions. The proposed approach involves pretraining with a mixture of noisy multilingual YouTube SLT data, parallel corpora, and SLT data augmented with MT models. Experiments based on (m/By)T5 models show substantial improvements over the baselines on several benchmarks, surpassing the previous state-of-the-art (SOTA) by wide margins.

Strengths

The paper is well-written and provides many insights. The problem is that improving the sign language translation performance in open-domain settings is important. Scaling model size, number of languages, and data size lead to impressive performance based on (m/By)T5 models.

Weaknesses

1. In Figure 7, the legend is blocked. 2. There is no open-source code or model, and reproducing this work requires substantial resource expenditure.

Questions

In the zero-shot ASL-to-X translation, does the model encounter similar issues seen in zero-shot text translation, such as translations in wrong languages?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The author does not release the code or model.

Summary

This paper presents an approach to Sign Language Translation (SLT) that aims to scale the field by addressing limitations in data, model size, and the number of translation directions. The authors' key contributions are: * Data Scaling: The authors leverage diverse and large-scale datasets, including noisy multilingual YouTube SLT data and augmented SLT data generated from video captions, enabling more robust pretraining. * Model Scaling: They pretrain SLT models on various sizes, initially fine-tuning them with pretrained (m/By)T5 models, demonstrating the impact of larger models on performance. * Cross-lingual and Cross-modal Transfer: The authors show that cross-lingual and cross-modal transfer from pretraining on multilingual data improvesSLT performance, even enabling zero-shot translation, translating sign language to a spoken language without explicit training in that direction. * Open-domain SLT Evaluation: The models are finetuned on five downstream open-domain SLT benchmarks covering five different sign languages, leading to substantial quality improvements over existing state-of-the-art methods.

Strengths

* The paper and approach is well motivated. * The experiments are generally well thought out. * The writing quality and analyses are generally good and informative. e.g., Sec 4.2 brought up questions that I had been wondering and did a nice job answering. * The results relative to SOTA are compelling.

Weaknesses

There are multiple references that claim to be on arXiv but don't seem to exist including the FLEURS-ASL paper. I found at least three instances: * Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multi- task evaluation. arXiv, 2024. * Garrett Tanzer. Fingerspelling within sign language translation. arXiv, 2024. * Garrett Tanzer and Biao Zhang. Youtube-sl-25: A large-scale, open-domain multilingual sign language parallel corpus. arXiv, 2024. FLEURS-ASL#0 only has 353 sentences but is used for a large percentage of the experiments. I have some question about whether or not these results are likely to generalize, especially given that these sentences were generated by a single person. Given that the FLEURS-ASL paper isn't actually on arXiv (as of July 13) I don't have a way to understand what types of sentences these include. it's important to understand the biases and breadth of what is covered. Lack of Comparative Analysis: While the paper surpasses the previous state-of-the-art, a more in-depth comparative analysis of other recent SLT methods, including their strengths and weaknesses, would strengthen the paper's argumentation. Yes, Table 4 does show SOTA numbers, but it would be useful to contrast the approaches. Minor: Phoenix is referred to in Table 4 and 8 descriptions but only acronyms are used in the tables themselves -- it's unclear which acronym is which.

Questions

See questions above. Especially, can you explain the missing references and talk through characterization of the FLEURS-ASL dataset? Can you also confirm that you are not fine-tuning on FLEURS-ASL#0? Regarding Table 5 and related analysis, is there a reason why BLEURT could have a much higher correlation compared to BLEU or ChrF? How accurate are the mediapipe landmarks on messy YouTube data? Have you looked at correlations between landmark accuracy and translation quality?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Seems reasonable.

Summary

This paper attempts to advance the development of sign language translation (SLT) technology by using large-scale pre-training data, expanding model size, and adding translation directions. Through extensive experiments, the authors have drawn many useful conclusions. Experiments show that this work can achieve the best results in multiple open-domain SLT benchmarks covering multiple sign languages, although the translation quality needs to be further improved to meet practical needs.

Strengths

- The exploration of sign language translation in large-scale open domain is very valuable. Previously, much sign language translation work was limited to some smaller fields. - The authors conducted a large number of experiments and obtained some useful conclusions, which can guide the future work of sign language translation. - Combining all the gain-enhancing methods together, the authors achieved state-of-the-art results on multiple downstream tasks of sign language translation.

Weaknesses

- Unfortunately, the sign language data and code used in this article will not be open source. I also can't seem to find the corresponding paper about "Youtube-sl-25: A large-scale, open-domain multilingual sign language parallel corpus" for the YT-Full dataset, which makes it difficult for later generations to reproduce this work. - As far as I know, using the original RGB sequence as input often achieves better results than simply using the pose sequence as sign language input because it contains more information. Can the author explain why the RGB sequence is not used as input? - The author does not seem to state the specific computing resources and time spent on the experiment in section 3.

Questions

see Weaknesses

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

yes

Summary

The paper focuses on scaling sign language translation (SLT) by leveraging large-scale pretraining data, increasing model size, and expanding the number of translation directions. The study demonstrates the effectiveness of data/model scaling and cross-lingual cross-modal transfer in improving SLT performance across multiple benchmarks. The authors showcase substantial quality improvements in SLT through scaling, surpassing previous state-of-the-art results by wide margins.

Strengths

1 The paper pushes the frontier of SLT through large-scale pretraining and the exploration of various data sources. This approach demonstrates a novel application of scaling techniques in the SLT domain. 2 The research employs a rigorous methodology, including the use of different pretraining tasks and modalities. The thoroughness of the experimental design and validation is evident. 3 The presentation is clear and well-structured, allowing for an easy understanding of the research approach and findings. The logical flow of the paper aids in comprehending complex concepts. 4 The work has significant potential to advance open-domain SLT for multiple sign languages through scalable methods. The substantial quality improvements over previous state-of-the-art results highlight the impact of the research.

Weaknesses

1 Lack of discussion on the impact of model architectures and training strategies. The study lacks an in-depth exploration of how different model architectures or training strategies could affect SLT performance, potentially limiting the understanding of scalability and generalization. 2 Limited exploration of the robustness to variations in data quality and model complexity. The research does not extensively discuss the robustness of the proposed methods to changes in data quality and model complexity, which could impact the robustness of the study results.

Questions

1 Have the authors considered the impact of different model architectures or training strategies on SLT performance, and how these factors could influence scalability and generalization?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

This paper adequately discusses the limitations in Section 6.