Gloss-free sign language translation (SLT) aims to develop well-performing SLT systems with no requirement for the costly gloss annotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify a representation density problem that could be a bottleneck in restricting the performance of gloss-free SLT. Specifically, the representation density problem describes that the visual representations of semantically distinct sign gestures tend to be closely packed together in feature space, which makes gloss-free methods struggle with distinguishing different sign gestures and suffer from a sharp performance drop. To address the representation density problem, we introduce a simple but effective contrastive learning strategy, namely SignCL, which encourages gloss-free models to learn more discriminative feature representation in a self-supervised manner. Our experiments demonstrate that the proposed SignCL can significantly reduce the representation density and improve performance across various translation frameworks. Specifically, SignCL achieves a significant improvement in BLEU score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset by 39% and 46%, respectively, without any increase of model parameters. Compared to Sign2GPT, a state-of-the-art method based on large-scale pre-trained vision and language models, SignCL achieves better performance with only 35% of its parameters. Implementation and Checkpoints are available at https://github.com/JinhuiYE/SignCL.
Paper
Similar papers
Peer review
Summary
This work uses contrastive learning of sign language gestures to improve the discrimination power of learned representations of gestures. This is motivated by showing, via TSNE projections, that gloss-free representations (that do not benefit from an intermediate representation), are less well represented that gloss-based representations. After incorporating contrastive learning, either into the pre-training, fine-tuning, or both, performance improvements can be achieved for both sign-language recognition and for sign-language translation over the presented baseline.
Strengths
The paper is tackling an important problem as accurate sign-language recognition and translation will enable accessibility for the hard of hearing and those that need to interact with the hard of hearing. The paper is well motivated from the perspective of demonstrating that the particular measure used to determine representation quality (SDR) is significantly lower for gloss-free representations, which is the focus of interest here. However, the approach itself will have limited impact in the broader NeurIPS community. The paper is using known methods applied to the representation of sign language. The work is based on open data and the authors will release their code and model with the paper. This will ensure reproducibility.
Weaknesses
The word “significance” has specific meaning, and you should not claim that results are significant without properly showing that they are so. Results for only a single run are provided. This is acknowledged in the check sheet with the justification that increasing the number of runs is computationally expensive. However, the resources required are ~12 hours on 8x NVIDIA A100s, which does not seem excessive by current standards. One solution might be to just use the best values for the hyper parameters, and show these are representative across runs. Some of the findings appear obvious -- of course representations of dissimilar gestures that project closely in embedding space will be difficult for a model to tease apart for downstream tasks. There is a lot of emphasis and conclusions drawn from the visualizations, which are TSNE projections of the embeddings. TSNE visualizations should be considered with a degree of caution, and there are better solutions if you are trying to consider the global structure of the data.
Questions
Figure 2 — is the reported SDR just for the highlighted signs in the figure or is it computed across all sign gestures? Is the SDR issue artificially inflated because of the dataset used? For example, using a corpus of signing from weather forecasting with a limited vocabulary and many signs around a small number of topics? A missing limitation of the work is that a fixed margin is used to define the threshold for frame selection for the contrastive learning. Whilst 20 frames might be sufficient, as well as being impacted by closely repeating signs, could frame-selection also be impacted by signing rate? My understanding is that two annotators select a frame representative for each gesture, and the main text said that details are in the appendix. However, I did not see how you select the final frame from the two (potentially different) annotations? Do you average the time stamps to pick the one on the middle? Some other observations: - Remove subjective qualifiers from your writing. - Your equations should be punctuated and flow with the text. - For Figure 2, I would highlight gloss-based vs. gloss-free approaches as at first glance approaches like SMKD look significantly better, and then later realized that this is expected since it is not a gloss-free approach. - For Figure 3, the binning needs to be explained in the caption. Also is the axis label for the vertical axis correct? Is this figure not showing sign recognition accuracy and sign translation accuracy? - It seems somewhat redundant including both Figure 1 and Figure 5. - I think it would be worth reinforcing in Section 2.1 when you talk about mixing the training and testing tests that the purpose of this is only to derive labels, not for the model itself. - Has the contrastive learning been used with the other baselines? It seems that since the performance there is less good, there might be more room for improvement?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Yes the limitations are addressed adequately.
Summary
This paper identifies a critical issue of representation density in gloss-free sign language translation. A series of models were employed to verify the existence of this problem. To address this, the author proposed a straightforward and effective solution, SignCL loss. This objective improves the discriminative representations of input videos, achieving promising performance in PhoenixT and CSL-Daily datasets.
Strengths
* The paper is straightforward and comprehensible, and the proposed method is somewhat effective. * The adequate visualizations validate the effectiveness of the proposed objective. * The extensive experiments on both public benchmarks show plausible performance.
Weaknesses
* The technical novelty and contribution are somewhat limited. The entire paper mainly introduces a contrastive loss (SignCL), which has been widely utilized in other various domains. * The proposed objective is only applied in the single model, which seems to lack persuasiveness. Experiments on more SLT models are essential to validate its effectiveness. * On line 118, why is there a need to optimize the test set? Why must we manually determine the best frame from $l_{v}$ and $r_{v}$ instead of utilizing CTC's gradient? For VLP, how are $l_{v}$ and $r_{v}$ generated? * In section 2.2, why are VLP, I3D, and SMKD methods grouped together? These methods should differ in terms of their training objectives. SDR of SMKD will naturally be low, because CTC loss is a discriminative loss. Achieving low SDR is essential for successful completion of the CSLR task.
Questions
* The writing in the article is overly colloquial, and the presentation of Formula 4 may be confusing. Additionally, there is an extraneous character ")" in the second line of the title in Figure 4. * Line 236 claims that GFSLT "incorporates CLIP and MBART for model pretraining and finetuning." However, to my knowledge, GFSLT does not utilize CLIP's pre-trained weights. * The performance of CSL-Daily in Table 2 is notably inconsistent. In Table 1, when Sign2GPT achieves R@L of 48.90, its B@4 is 22.52. However, in SignCL's CSL-Daily, despite achieving a similar R@L of 48.92, its B@4 drops to 16.16. Please verify the accuracy of these performance metrics. * The t-SNE visualization method is unclear. How should the SLT task be correctly labeled with gloss? * The datasets PHOENIX-2014T and CSL-Daily are relatively small in size. Can the effectiveness of the method be validated on larger datasets like How2Sign and Open-ASL?
Rating
4
Confidence
5
Soundness
2
Presentation
2
Contribution
1
Limitations
Yes, the limitations has been discussed in the main paper.
Firstly, I would like to thank the authors for their detailed responses and the additional experiments. After reading the rebuttal, I still have the following concerns regarding the paper: 1. The authors claim that the main contribution of the paper is identifying the representation density problem for gloss-free SLT for the first time. However, I believe the fundamental reason why gloss-free SLT lags behind gloss-based SLT in performance is that the Vision Encoder lacks effective supervision signals, leading to insufficient representation capacity of the visual features. The representation density problem is merely one manifestation of the weak representational capacity of the visual encoder. In general vision tasks, numerous studies have been conducted on improving the representational capacity of visual encoding, such as Moco, Dino, MAE, etc. Therefore, striving to enhance the discriminative power of visual representations is a direction that has been continuously explored by the academic community. Thus, I believe that positioning this discovery as a core contribution lacks novelty. 2. Regarding the representation density problem in the visual encoder, enhancing the representational capacity of the visual encoder is key to addressing this issue. The contrastive loss proposed in this paper does indeed improve the representational capacity of the visual model to some extent. However, from the quantitative experimental results, the improvement effects are inconsistent across different datasets. Compared to PhoenixT and CSL-Daily, H2S is a very challenging dataset due to its larger training set and vocabulary. It can be observed that the improvement brought by the proposed method on this dataset is very limited (B@4 0.79->1.58). In contrast, GLoFE [1] demonstrated that directly using the CNN model trained in [2] could achieve B@4 2.24. Therefore, these results do not provide strong evidence that the contrastive loss proposed in this paper can effectively solve the representation density problem. In other words, using a pre-trained visual model with stronger representational capacity might well solve this problem. [1] Lin, Kezhou, et al. "Gloss-free end-to-end sign language translation." In ACL 2023. [2] Oscar Koller, et al. "Weakly supervised learning with multi-stream cnn-lstm-hmms to discover sequential parallelism in sign language videos." In TPAMI 2019.
Addressing Misalignment in Perspectives on the Position of This Paper
Thank you for your thoughtful review and for highlighting key points. I believe **we are aligned in recognizing the gloss-free SLT struggles from the weak representative capacity** of the Vision Encoder due to insufficient supervision signals. The difference may lie in how we approach the problem, particularly in terms of perspective and granularity. In response to your concerns, we would like to clarify our position: >* we find that describing this problem as "insufficient representation capacity" is too broad and general. In our work, **we formalize this problem specifically as the lack of discriminative power in the visual representations of sign gestures**, which we term the "representation density problem." >* While general vision tasks have made strides in improving visual representations, we argue that the ability to distinguish between sign gestures with different meanings is crucial in the context of sign language, and prior to our work, this has not been highlighted. >* We demonstrate that the aspect of representation density significantly impacts sign language translation performance, by comprehensive investigation in Section 2. >* SignCL is relatively straightforward (in a good way) in addressing the lack of discriminative power for different sign gestures. It represents a good start, and of course, there is much more that can be explored in the future. Regarding the performance in H2S, we would like to argue two points: >* Our experiments were limited due to constraints in computational resources and time. We only trained for 40 epochs, whereas GLoFE [1] was trained for 400 epochs. >* The CNN model used in GLoFE [1] was pre-trained with GLOSS annotations to extract visual features [2]. The SignCL approach does not use any gloss annotations, whether for pretraining or as weak supervision. >We believe that **directly comparing our results with GLoFE is unfair**. We will continue training the model, but we estimate that similar training would take around 25 days on an A800*8 GPU setup due to GFSLT processing raw video inputs directly. So far, our results demonstrate that incorporating SignCL does improve GFSLT performance (B@4 0.79 → 1.58) on How2Sign. Thank you again for your thoughtful review and detailed responses. We welcome further discussion.
Kindly Follow-up on the Discussion
Dear Reviewer, We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We have provided a new response to address the concerns you further raised. Thank you for your thoughtful review and for highlighting key points. We kindly inquire if there is any further information or clarification you might need from our side. We apologize if this follow-up causes any inconvenience. Hope you have a great day!
Thanks for your detailed responses and further discussion. After carefully reviewing the rebuttal and the main paper, I still consider that presenting the discovery of the 'representation density problem' as a core contribution lacks innovation. In the field of sign language video understanding, improving the representational capacity of visual models is a common goal among researchers. Furthermore, the proposed contrastive loss function has not consistently improved performance across public datasets (the performance improvements on PhoenixT and H2S are significantly lower than on CSL_Daily), making it difficult to assess the generalization ability of SignCL. I suggest the authors may need to conduct more in-depth research on this phenomenon to enhance the persuasiveness of the proposed method. Therefore, I maintain my initial score.
Thank you for your thoughtful feedback. Based on the new concerns you’ve raised, we would like to share some additional arguments. We would like to mention Sign2GPT[a], which involves fine-tuning large-scale pretrained vision and language models (Dino-V2 and XGLM) for SLT. * According to the ablation study results presented in Table 2 of their paper, directly fine-tuning these models pretrained on general domains (termed as Sign2GPT) yields results that are only competitive with GFSLT-VLP (B@4 19.42 vs. 21.44 on PHOENIX-2014T). * To improve performance, they developed a pseudo-gloss pretraining strategy (termed as Sign2GPT(w/PGP)) on Dino-V2. In contrast, our approach, using a simple contrastive loss, achieves better improvements over GFSLT-VLP when compared with Sign2GPT(w/PGP) on both the PHOENIX-2014T and CSL-Daily datasets. --- Regarding the performance of the proposed SignCL, we believe it should be compared to more fully gloss-free methods, such as Sign2GPT [a], rather than GLoFE [1] which uses a GLOSS Dataset pretrain viusal backbone$^*$. We have provided consistent benchmarks with a competitive baseline with [a] and [b]. The improvements we observe on the two most widely used datasets in the sign language field (PHOENIX-2014T and CSL-Daily) are generally consistent with [a] and [b]. To sum up, though numerous studies in general vision tasks have explored improving the representational capacity of visual encoders, this does not diminish the novelty of our contribution to SLT. **Our SignCL outperforms those achieved using general vision pre-trained models, such as Dino-V2 in Sign2GPT[a].** While we indeed did not have sufficient resources to run How2Sign for 400 epochs (which would take around 25 days on an A800*8 GPU setup), we believe that the current results are sufficient to validate our approach. **It's fairer to compare with Sign2GPT[a] and GFSLT[b] due to the consistent setting onthe two most widely used benchmarks (PHOENIX-2014T and CSL-Daily)**. --- [a] "Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation" in ICLR'24 [b] "Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining" in ICCV'23 \* As referenced in Section 3.1 of GLoFE [1]: "The backbone is pre-trained on the WLASL dataset (Li et al., 2020a) through the isolated sign language recognition task."
Thank you for your thoughtful responses. * For Sign2GPT [1], although this method utilizes large-scale pre-trained vision and language models (Dino-V2 and XGLM), both modules are frozen during training (except for the few learnable parameters with the Lora fine-tuning). Under such setting, Sign2GPT could achieve comparable performance with other methods by training all parameters of their proposed model. This result further validates my assumption, that **enhancing the representative capacity of vision encoders is the key to improving the gloss-free SLT or gloss-based SLT.** Meanwhile, I also acknowledge that the proposed SignCL can improve the representational capacity of visual models to some extent. However, I am confused by the varying degrees of performance improvement that SignCL achieves across different benchmarks. [1] "Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation" in ICLR'24
Thank you for your insightful feedback and for acknowledging the effectiveness of SignCL in enhancing the representational capacity of visual models. We do not see the assumption you believe as a point of contention regarding the contributions of our work. In fact, we have specifically formalized this poor visual encoder as the lack of discriminative power in the visual representations of sign gestures. **Our work systematically experiments to validate this issue (Section 2), and our results clearly demonstrate that improving the discriminability of visual representations indeed leads to better translation performance (Section 3).** Regarding the inconsistency in the improvement of SignCL on PHOENIX-2014T and CSL-Daily, this issue aligns with what we addressed in Q3. **The characteristics of Chinese and German languages are quite different**, making direct comparisons across datasets challenging. Sign2GPT vs. GFSLT-VLP also exhibits similar varying degrees of performance improvement. We understand there may be concerns about our performance improvement on CSL-Daily. We have committed to release our code, models, and logs to facilitate the reproduction of our results.
Summary
This paper addresses the challenge of gloss-free sign language translation (SLT) by identifying and tackling the "representation density problem". The authors observe that visual representations of semantically distinct sign gestures tend to be closely packed in feature space, making it difficult for gloss-free methods to distinguish between different gestures. To address this, they propose a contrastive learning strategy called SignCL, which encourages models to learn more discriminative feature representations in a self-supervised manner. The paper demonstrates significant improvements in BLEU scores across various translation frameworks on the CSL-Daily dataset.
Strengths
1 The paper identifies a novel problem (representation density) in gloss-free SLT and proposes an innovative solution (SignCL) to address it. The proposed SignCL method shows substantial improvements in SLT performance without requiring additional model parameters, potentially advancing the field of gloss-free SLT. 2 The authors conduct thorough experiments to demonstrate the existence of the representation density problem and the effectiveness of their proposed solution. They evaluate their method across multiple datasets and SLT frameworks. 3 The paper is well-structured and clearly written. The problem statement, methodology, and results are presented in a logical and easy-to-follow manner.
Weaknesses
1 The sampling strategy for SignCL makes assumptions that may not always hold true in real-world scenarios. Specifically, it assumes that adjacent frames always belong to the same sign gesture and that frames far apart always represent different semantics. This approach overlooks the possibility of rapid transitions between gestures or the repetition of gestures over time, which could lead to incorrect positive or negative samples. 2 The SignCL method relies on several manually defined parameters and thresholds, such as the margin for determining positive and negative samples. The paper lacks a rigorous justification for these choices or an analysis of how sensitive the method is to these parameters. A more principled approach to parameter selection or a comprehensive sensitivity analysis would strengthen the scientific basis of the proposed method.
Questions
1 How does the performance of SignCL vary with different choices of the margin parameter in the sampling strategy? Is there a systematic way to determine the optimal margin for a given dataset?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors discuss their limitations in the Appendix。
Summary
This paper focuses on gloss-free sign language translation (SLT) and is largely motivated by the large cost to annotating glosses. The authors discussed a so-called "representation density problem" in gloss-free SLT where semantically dissimilar signs appear in a similar part of the latent space. The first technical contribution relates to a metric they introduce to quantify the representation problem, which uses Fisher’s Discriminant Ratio (FDR) to assess the difference between the inter- and intra-gloss features. To improve the separability, as their second contribution, they propose SignCL, a contrastive learning strategy that encourages the model to learn more discriminative feature representations in a self-supervised manner. The results show significant improvements in BLEU scores on the CSL-Daily dataset compared to state-of-the-art methods.
Strengths
Intuitively the motivation for the paper makes sense, the methods are relatively straightforward (in a good way), and their solution seems to work well. The authors provide a good amount of detail (especially in the appendix) in case someone wanted to reproduce.
Weaknesses
A primary motivating example in the paper is that “RECIPROCATE” and “REVENGE” live in similar parts of the latent space because the hand motions are similar (even though the facial expressions are different). The proposed contrastive approach takes samples from different parts of the same sentence, where each sign should contrast with dissimilar signs in that sentence. Perhaps this is just an issue with the motivation, but it doesn't seem like the approach would help with the motivating example. Similarly, it doesn't seem like the contrastive approach should be any more helpful with gloss-free SLT vs gloss-based SLT. I am wary about some of the t-SNE visualizations. The Sign Density Ratios don't change that much between approaches, and given that t-SNE is known to be deceptive at times, I wonder if these visualizations are representative or overstated. Sec 4.3 on qualitative analysis is interesting but is too anecdotal. It would be useful to have a much deeper set of analyses here covering a much larger set of examples. Overall, I think this paper is certainly not bad, but I'm on the fence about whether the novelty or depth is sufficient for this venue.
Questions
There are two types of issues that are highlighted: (1) challenges where signs have the same hand motion but different facial expression and (2) challenges where the same hand shape is used but with different motions (e.g., piano vs typing). Can you talk specifically about how the proposed approaches help with face and motion errors? I noticed some odd results in A.3.1. The WER results for train/test/dev are hugely different: 8.68/8.03/25.28. How is it the case that 'dev' has a 25% WER but train and test have a hugely improved 8%?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Their responses seem reasonable.
Kindly Follow-up on the Discussion
Dear Reviewer, We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Our response has been available for several days now. With the discussion deadline approaching, we kindly inquire if there is any further information or clarification you might need from our side. In our response: > We have provided a clearer explanation of the position of this paper, where this paper focuses on the overall discriminability of feature extraction in sign language translation. We apologize for any confusion caused in the original paper, particularly regarding the cases in Figures 1 and 5. >Additionally, we have included further margin distribution statistics and sensitivity analysis. The experiments demonstrate that the SignCL method is not sensitive to the margin parameter. We would greatly appreciate your prompt response and thank you once again for your valuable comments and insightful suggestions. We apologize if this follow-up causes any inconvenience. Hope you have a great day!
Summary
The main content of this paper is about improving the performance of Gloss-free sign language translation. The author discovered the representation density problem in sign language translation, that is, in the feature space, the semantic visual representations of different sign language gestures tend to be closely clustered together, making it difficult for the gloss-free method to distinguish different gestures, thus significantly affecting the translation performance. To solve this problem, the paper proposes a simple and effective contrastive learning strategy SignCL, which encourages term-free models to learn more discriminative feature representations in a self-supervised manner. Experimental results show that SignCL is able to significantly reduce representation density and improve performance in different translation frameworks.
Strengths
- The authors discovered the representation density problem for the first time in the field of sign language and conducted a detailed analysis to show that this problem does affect the performance of sign language translation. These findings will help advances in the field of sign language processing. - Based on this finding, the authors proposed a contrastive learning method to improve the representation density problem. Experiments show that this method works well in the gloss-free setting. - The author promises to open source the code and model
Weaknesses
- The core of the contrastive learning method in this paper is the selection of corresponding positive and negative samples. The analysis and supplement of strategy selection can make this work more perfect. For example, what is the impact of choosing different distance parameters in formula 4? In addition, the selection of negative examples does not seem to be limited to a single video. For example, randomly selecting other frames with the same signer may be a better choice. - The contrastive learning method proposed by the author can be considered as a method to enhance the representation of sign language videos. This method should also have a certain effect on the gloss-base method, although the benefit may not be as large as that in the gloss-free setting. I would be happy to see the author add some relevant data, which can expand the scope of application of this paper's method.
Questions
see Weaknesses
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
yes
Kindly Follow-up on the Discussion
Dear Reviewer, We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Our response has been available for several days now. With the discussion deadline approaching, we kindly inquire if there is any further information or clarification you might need from our side. In our response: > We have included additional margin distribution statistics and sensitivity analysis. The experiments demonstrate that the SignCL method is not sensitive to the margin parameter. >Additionally, thank you for your insightful suggestion, we have applied SignCL to the Gloss-based method, which has also shown improved performance. We would greatly appreciate your prompt response and thank you once again for your valuable comments and insightful suggestions. We apologize if this follow-up causes any inconvenience. Hope you have a great day!
Thank you for the response
Thank you for the responses to the questions and concerns raised. I also appreciate you taking the effort to provide the statistics over multiple runs. I will increase my score by a point. I would make two observations about your response. 1. You used the word "merely" and then put quotation marks around words that did not appear in my review. If you are using quotation marks, then you are attributing those words to someone else. The only place those words are used is in your response here and the general response above. 2. I did not say that you rely on visualizations "alone", but rather that there is a lot of emphasis on TSNE projections. TSNE projections are known to be problematic. There are alternative projections, such as UMAP, which better preserve the global structure of the data.
Thank you for your thoughtful review and for your careful suggestions. We have learned a lot, and we will be more rigorous in using quotation marks and the words we use. We also appreciate your clarification. We will include UMAP visualizations in future versions.
We would like to express our sincere thanks to Reviewer 9u2m for his/her patient and thoughtful reminder. We apologize for the lack of rigor in our use of quotation marks and qualifiers in our previous response. To clarify simply: The phrase "merely 'using known contrastive learning methods applied to the representation of sign language'" was intended to reference the sentence "The paper is using known methods applied to the representation of sign language," from Reviewer 9u2m's original comment, as well as elements of "The entire paper mainly introduces a contrastive loss (SignCL), which has been widely utilized in other various domains," from Reviewer rKGZ's original comment. We regret the oversight and will be more careful in our wording moving forward.
Gratitude to All Reviewers and a Request for Full Discussion on the Contribution of This Paper
We sincerely thank all the reviewers and ACs for their time and effort in reviewing our work. We understand there may be some differences in opinion regarding whether identifying the representation density problem as a bottleneck in restricting the performance of gloss-free SLT and providing a straightforward and effective solution to this issue constitutes a solid contribution. Given the time constraints, we hope to further discuss this matter during the closed discussion phase with the other reviewers and ACs. Once again, we greatly appreciate everyone’s time and thoughtful feedback.
Decision
Accept (poster)