Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs

Summary

This paper proposes a unified architecture and training method for auditory/visual speech recognition. Building upon this model, the authors introduce a semi-supervised pseudo-labeling method to leverage unlabeled audio-visual data, as well as self-supervised pre-training to enhance model performance. Experiments indicate that the model achieves state-of-the-art performance on A/V/AVSR.

Strengths

1. This work for the first time proposes an effective model and training procedure for unifying auditory and visual speech content recognition, which is of high novelty and practical significance. 2. The author conducted comprehensive and extensive ablation studies, verifying the characteristics of the model and the effectiveness of each step in the training paradigm. The experimental results are robust and credible, offering significant guidance for related research.

Weaknesses

The article has no obvious flaws, but there are some questions that I hope the authors can clarify (see questions).

Questions

1. How is the weight of the teacher model in self-supervised pretraining initialized? Is it initialized randomly or with pretrained weight on another task? 2. Did the author make a comparison between the teacher-student self-supervised pretraining in the paper with masked-autoencoding training of audio and/or visual features? Is the proposed pretraining method superior? 3. Did the author investigate the effect of different masking ratios?

Rating

7

Confidence

2

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discuss the limitation of their work in appendix A.

Summary

This paper proposes a training methodology for a *single* model which can use *either* audio, visual, or audiovisual features as input for automatic speech recognition. This is done by enforcing a training batch always includes (feature,label) pairs of all three modalities, using a 1D/2D ResNet-18 feature extractor for audio and video, respectively. These features are processed by a Transformer encoder-decoder model to obtain an ASR prediction. Furthermore, the authors explore a semi-supervised fine-tuning approach and a self-supervised initialization stage, both using a student-teacher approach, and within the same unified methodology. This allows the authors to produce a model which is competitive with state-of-the-art models while using a significantly less data.

Strengths

I think the proposed method is interesting for researchers in the audio-visual ASR domain and will spur future work. The paper is well-written with clear English, barring some questions I have stated below. The authors do a good job presenting their results, referring to details in the appendix where required. The ablation experiments clearly show readers how their proposed methodology behaves and why certain design decisions were made. The authors also shared their code and model checkpoints, which significantly increases the reproducibility and impact of this paper.

Weaknesses

The model architecture seems a bit unclear to me. Specifically, line 88 states the use of a transformer encoder-decoder model. However, line 104 states a single FC layer on top of the encoder for vocabulary predictions, while line 107 states to use the decoder output sequence, which is subsequently not used as $1 - \lambda_{ctc}=0$. So the decoder is not actually used during fine-tuning? How is inference actually done? I see no mention of a fixed random seed for running experiments, are all model initialized equal? This seems important as the paper does not have error bars/does not run experiments multiple times Minor editing comments: * Table titles must appear above the table as per the formatting instructions. * The table/figure combinations on Page 6 are confusing. Could you separate the figures as not part of a (sub)table? * A small description of LRS3 would be desirable for those not familiar with the dataset (e.g., how many hours does the unlabeled portion have (line 190), what is the data source, how was it collected, how large is the test set?) * line 97: 0.4 and 0.6 seconds for each second of ...

Questions

In which settings/experiments is the transformer decoder used? In table 3 (A), is there a reason for not trying targets A + V + AV, as during fine-tuning? You state in line 103 that features from the 3 modalities are concatenated along the batch dimension for efficient processing. However, Table 1 (B) shows that random sampling of modalities performs much worse, requiring 3x more epochs for similar performance. So it seems to me it's not only done for efficient processing, but also for effective optimization? Also, do [13, 15] in line 179 share parameters for each task or not? According to Table 4 they do not, but if you use random sampling of modalities, how does this explain their relevance to Table 1 (B)? What is the CTC attention in Table 2 (C)? Is this simply equation 3 with $\lambda_{ctc} < 1$? I might have missed it, but it seems to me the method section does not explain these 2 different loss types?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The methods requires all data to be audio-video paired. An interesting future direction could be the inclusion of audio-only data in the framework.

Summary

This paper proposes USR, a unified speech recognition model that leverages pseudo labels during fine-tuning. It introduces a single model capable of handling three tasks—ASR, VSR, and AVSR—simultaneously, delivering state-of-the-art performance.

Strengths

1. The paper is well-organized. Although the USR system is relatively complex, the paper presents each module with detailed descriptions and clear illustrations, making it easy for readers to follow. 2. The experiments, including ablations, are extensive. All experimental details are included, making it easy to reproduce the results. 3. The USR system leverages pseudo labels during the fine-tuning stage. While pseudo labeling is not a novel technique in ASR or AVSR, USR enhances the performance of ASR, VSR, and AVSR through carefully designed training procedures. The illustration of the pseudo labeling process is also clear. 4. The system achieves nearly state-of-the-art performance across all tasks. 5. The literature review is thorough.

Weaknesses

1. While not a unique weakness to this paper, the complexity of training current SSL-based VSR or AVSR systems remains a challenge. Introducing additional modalities significantly increases complexity compared to speech-only SSL systems. Notably, the reduction in GPU hours is minimal compared to previous works, and the convergence speed is exceedingly slow. Future work should address these issues. 2. Performance is highly sensitive to certain configurations, such as the ratios of pseudo labels and the use of EMA. However, the paper lacks an analysis of why this sensitivity occurs or suggestions on how to mitigate it. These are common weaknesses in related work. 3. The results do not consistently achieve state-of-the-art performance. The authors should experiment with other hyperparameters, such as learning rates, during fine-tuning to improve outcomes. 4. Failure cases were not discussed too much.

Questions

1. During pretraining, have you explored using audio-only targets? If so, what was the performance like compared to AV targets? How does it compare to AV-HuBERT? 2. Why do you incorporate all three features (audio, video, audio-visual) during fine-tuning? Is there a rationale or experimental evidence supporting this approach? 3. There's no need to adhere strictly to the architectures like AV-HuBERT or AV-data2vec. Consider experimenting with more advanced video encoders since visual features are often not well-extracted in previous studies. 4. For pseudo label sampling, why opt for a greedy search? Have you considered trying soft sampling instead?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations have been discussed in the paper.

Summary

This paper unifies the ASR, VSR, and AVSR tasks in a single model and shows the performance benefits of a single model in LRS3 data. There are several attempts at unifying these three models, but I think this is the first successful trial of realizing it. The paper proposes an effective training strategy to avoid losing performance on each task. Together with their self-supervised training, the model archives SOTA performance in a similar range of the training data.

Strengths

- the first successful method of realizing the ASR, VSR, and AVSR tasks in a single model while maintaining/improving the performance for each task - Good reproducibility based on the code release, use of the public data, and detailed experimental configurations/analyses. - Easy to read. Although the technique is a little bit complicated with a lot of terms depending on the architecture (CTC, attention, modality, training modes (self-supervised/supervised), the paper always provides some rationales (e.g., from the reference or experiments) to justify their methods - detailed ablation experiments support their design choices and strategies. - The paper also shows the effectiveness with multiple databases (LRS3, LRS2, and WildVSR)

Weaknesses

- the technical novelty is not very strong. Most techniques are well-known or straightforward (e.g., the use of CTC, pseudo-label filtering, etc.).

Questions

- Page 4, line 110: I'm a bit confused about "We set $\lambda_{\text{ctc}}$ to 1." Do you mean that you always set $\lambda_{\text{ctc}}$ to 1? No attention weights? Is it related to Table 2-d? Please clarify it. - Equation (4): Why didn't you prepare a different weight for a and av? - Section 3.2, Filtering: Did you use the same threshold for CTC and ATT? The dynamic range of c and a could be different, and I'm not sure that using the same threshold is optimal. - Section 4: Did you only use a Transformer architecture? How about using a Conformer architecture? - It is not a question but a suggestion. I recommend you emphasize the results of the multiple databases in the abstract to claim the generalization of this work across the database.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The paper has independent sections about limitations and Societal Impact, which describe the current issue due to the computational cost, the importance of the VSR, and the risk of general speech recognition technology.

Thanks for your response

Thanks the authors for actively addressing my concerns. I think the current score already reflects my positive evaluation of the paper, so I choose to maintain the existing score.