Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

Summary

This paper proposed a Self-TAught Recognizer (STAR) that leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains. The proposed method includes a novel indicator that empirically integrates step wise information during decoding to assess the token-level quality of pseudo labels without ground truth and utterance level pseudo labels filtering . The STAR seems to have sufficient novelty for publication. Sufficient experiments are presented to show the effectiveness of the proposed method. The work should be reproducible since the code will be released.

Strengths

The strengths of the papers are: 1. The proposed method is novel. 2. Sufficient experiments and good analysis. 3. Good presentation and the paper are easy to follow.

Weaknesses

Major concerns about the paper. 1. It was shown in Figure 3 that the UDA converged with about 1 hour speech data for the target domain and the gain is very small with more data. Given that the performance of the STAR is still a lot worse (some are close) than the true-label model (The upper bound of STAR), I am concerning about the usage of STAR in the real scenario. One or several hours data transcription is sometimes affordable for some organizations, so that they might get better numbers than STAR. 2. In other words, UDA has an upper bound while true-label model not. Did authors have the experiments to show how many hours of true-label data finetuning is equivalent to the performance of STAR? This might help the readers understand better on selecting UDA or human-transcription in different scenarios.

Questions

Some minor concerns or suggestions: 1. Maybe I missed the sentence in the paragraph, but I don't find the description of which layer or head of the attention weights are used for the indicator A. It is worth a mention in the paragraph especially in Equation 5. 2. Continuing the indicator A, did the authors compare the confusion matrix for other layers? In other words, does the attentive score confusion matrix behave similarly in Figure 2 for specific layers or for all layers (head)?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation of the method can be restricted to certain use cases. However, the proposed method is still helpful to the community.

Summary

The paper proposes STAR, a novel algorithm for unsupervised domain adaptation (UDA) of speech foundation models (e.g., one that has a decoder like Whisper). The UDA setting is the semi-supervised setting where unlabeled data from the target domain is available, and the speech foundation model is available. The unlabeled data is first pseudo-labeled and then used to train the ASR model. The paper specifically proposes the STAR Indicator, which combines the advantages of the auto-regressive confidence score and the attentive score (the attentive score is self-defined by the authors and derived from attention scores) to finally produce a score that can improve the fine-tuning process on pseudo-labels (named as "informed fine-tuning" by the authors and a method prevalent in literature where you guide the training process in a re-weighting manner).

Strengths

The strengths of the paper are as follows: - The paper is well-written and easy to follow. The illustrations are also well-made. - The algorithm proposed for uncertainty estimation is novel, to the best of my knowledge. - The evaluation setup chosen is sufficient, and no further additions can be made to the best of my knowledge. - I like the pre- and post-result analysis of the paper. Provides intuition into the proposed algorithm and results. - Section 5.2 and 5.3 is interesting. These kind of analysis is important for speech papers and should be promoted (which is generally not found in other papers). - The Appendix is well done and provides nice extra details

Weaknesses

- As also claimed by the authors, the overall methodology of using uncertainty estimation for improving UDA is not novel. Only the estimation score proposed is novel. However, these works are properly cited. This makes the paper sound, but not very exciting to me. However, I also acknowledge that this is not a primary reason to reject. - I am not convinced why only Whisper (and some other models in Table 4) was employed for evaluation (a model that does not have a lot of training details revealed). I have generally found methods evaluated solely on Whisper to not work in a real-world setting. For example, what would have happened if we had adapted a purely LibriSpeech-trained model to the domains mentioned in the paper with the STAR algorithm? Though I understand that the paper title says "Speech Foundation Models," for me, this makes the applicability of STAR a bit less to me. I would leave this decision to other authors - Some key baselines are missing: What happens to STAR without Utterance-level Filtering? Why was the LM-rescoring baseline not used (LM-rescoring is the most traditional method for UDA where the language model is further trained on target domain texts, may be you can fine-tune the LM on pseudo transcripts? (open to discussion if this is not valid)?

Questions

The algorithm is simple and sound. The evaluation setup is sufficient except the Whisper part. I might have further questions during the rebuttal period after I see other perspectives. But overall, the paper is well done (again, I do not find it too exciting but very sound) and I am inclined towards accepting. I hope the authors can respond to my points in Weaknesses.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

- I don't see the Limitations of this paper discussed in the main paper. Can the authors please add it to the main paper? Limitations is a very important section of a NeurIPS paper. Additionally, can the authors please elaborate the limitations section with more details like: Latency and resource requirement over plain UDA fine-tuning, high noise scenarios where UDA may fail, etc.

Summary

This paper investigates the use of audio-only data to enhance ASR performance for domain adaption in Speech Foundation Models. The approach is straightforward: recognition results are used to compute a confidence score for each token (e.g., BPE in this paper), which then weights the loss function, as shown in Equation (4). The key innovation is the computation of token-level confidence scores by adopting attention scores instead of the softmax output for confidence estimation. Unlike traditional methods that operate at the utterance level, this approach focuses on token-level confidence. The concept is simple: use the attention weight as the confidence score rather than the predicted next-word probability. The proposed STAR method combines information from both token-level probabilities and the attention matrix. ASR performance appears promising, especially when compared to the strong baseline model Whisper large v3.

Strengths

1. I find the proposed confidence estimation using attention scores intriguing. 2. The authors conducted extensive experiments on various ASR corpora to demonstrate the effectiveness of the proposed STAR algorithms, along with comprehensive ablation studies. 3. Significant performance improvements are reported over the strong baseline, which is Whisper large v3.

Weaknesses

1. Several details are missing. How is the attention matrix calculated given multiple layers and heads? Are the attention weights from cross-attention or self-attention used? 2. Do you need to tune the threshold for each corpus, or simply use a fixed threshold for all datasets? What is the specific value for the threshold lambda? 3. The definition of conflict is A_l^2/C_l, which seems intuitive. Is there any particular reason for using this formulation? 4. Many studies have attempted to use entropy from the output layer instead of probability for confidence estimation in end-to-end ASR systems. 5. I am interested in whether the informed fine-tuning shown in Equation (4) is proposed in this paper. The confidence score is used to filter utterances and weigh the importance of each word. Do you have any ablation studies demonstrating the effectiveness of this approach? 6. It would be better to combine Equations (6) and (7) into an equation array to clearly state the formulation. 7. In Equation (5), the attention score is computed using the attention weight for the current word to the previous words and from future words to the current word. Is there any particular reason for this choice? Is it acceptable for the A_l to be larger than 1? Overall, this paper seems to rely heavily on empirical or intuitive settings, making it difficult to extend to a theoretically sound approach. Additionally, many key details are missing. It would be appreciated if the authors could provide more details and present the information more clearly. I am open to reconsidering my rating based on the authors' rebuttal.

Questions

I have listed my questions in the above section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper proposes STAR, a novel ASR domain adaptation technique that requires no labeled data and only a few unlabeled samples. STAR utilizes the confidence score and self-attention score obtained during decoder inference to calculate the reliability score (STAR indicator) for each token. The score of each token is then used as a multiplier to adjust the fine-tuning loss, which is based on generated pseudo-labels. In addition, STAR employs utterance-level filtering to remove noisy predictions. Extensive experiments across various ASR models, datasets, and fine-tuning techniques demonstrate that STAR achieves significant accuracy improvements compared to the baseline self-training approach.

Strengths

* The logical flow of this paper is very interesting, and the authors provide empirical evidence for each step. The motivations behind the research and method are inspiring. * The authors have conducted comprehensive experiments using multiple datasets, models, and noisy conditions. I appreciate the authors’ effort on this. * STAR does not seem to suffer from catastrophic forgetting, and this is a very important advantage.

Weaknesses

* The proposed method is designed for a Transformer-based model with auto-regressive generative decoder architectures. As such, it may not be easy to adopt STAR for CTC or RNN-T-based ASR models (as the authors also discussed in Appendix A). * It would be important to discuss the differences and similarities between STAR and noisy student-based training [1][2]. NST also employs pseudo-labeling, heavy filtering, and iterative refinement. * [1] Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition * [2] Improved Noisy Student Training for Automatic Speech Recognition

Questions

* The attentive score A (Equation 5) seems to be affected by the total length of the generated transcript. The longer the transcript, the (potentially) higher the attentive score. In contrast, confidence scores are bounded to [0, 1]. Maybe missing a normalization term in Equation 5? * Any intuitive reasons not to incorporate cross-attention scores? * Comparing Table 1 and 2, the numbers of “Frozen” are the same but “Self-train” and “STAR” are different. For example, TED3 WER is (5.2, 4.9, 4.1) in Table 1 but (5.2, 5.3, 5.0) in Table 2. What’s the difference? * How many beams are used in the beam search? Are the beam search-based pseudo-labels also used for self-training baselines? * It would strengthen the paper’s claim if the pseudo-label and STAR score could be cross-transferred between different models (for example, Whisper-Large generates training resources for Whisper-Tiny).

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

* The paper adequately addresses the limitations.

Thank You for the rebuttal

Thank You for the rebuttal. I am keeping my score which is already an accept. The paper takes a decent step towards advancing the domain.

Thank you for the response

Thank you for addressing my questions. I am keeping my score toward accept, and I believe this paper can inspire many following studies.

Reply the rebuttal

Thank you for the rebuttal. I am keeping my score which is already an accept. Please make sure the explanations can be reflected in the revised paper.