Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers

Summary

This paper proposes a new ASR model that connects a self-aligned encoder and the light text-only recurrence of RNN-T. The proposed model can be trained with label-wise cross entropy loss, which is computationally efficient than RNN-T training. The authors show the limitation of the model to inference on long-form audios and give a special inference configuration to mitigate the issue. Experiments on Librispeech and larger-scale ASR datasets demonstrated the close performance compared to other ASR models. The authors also show the audio-text alignment in the self-attention weights of a certain layer, which could be said to perform “self-transduction”.

Strengths

The paper proposed a new ASR model for better training and inference efficiency. The idea can be similar to performing down-sampling on encoder side but is more aggressive (to token-level). The strengths of the paper are: 1. The proposed model consumes way less GPU memory than RNN-T and achieves much lower latency than AED. 2. The analysis of the text-audio alignment behavior in self-aligned encoder. 3. A modification on Aligner without training to enable long-form audio decoding.

Weaknesses

The major concerns are the presentation and the limitation of the proposed model. 1. The explanation of the aligner modification for long-form audio is somewhat hard to follow. It may be better to have figures, or equations involved in the paragraph. 2. As the claim of the paper is that Aligner encoder achieves better efficiency than RNN-T and AED, it would be better to summarize the experimental numbers in Section 4.6 into a table for reader's convenience. People might be interested in those strong numbers. 3. The proposed model may have limited use cases e.g. offline ASR because the encoder cannot be implemented for streaming with the current design. 4. A related work that replaces RNN-T loss with the cross-entropy loss and reduces the GPU memory usage. (https://arxiv.org/pdf/2307.14132).

Questions

Several additional questions: 1. If I remember correctly, the AED model can achieve no worse WER than RNN-T on Librispeech in literature, I am wondering did the authors look into other reasons why AED in Table 3 is much worse than RNN-T (except the long-form problem)? 2. Any results comparing rotary positional embedding and relative positional embedding for speech encoder on long-form audio? The authors mentioned that RNN-T used RPE, is it the reason why RNN-T is good at long-form audio?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The main limitation is the use case of the proposed method, aka, non-streaming applications. Even though, the method can be helpful for many different purposes.

Summary

The paper introduces Aligner, which is to take the best parts from RNN-Transducer and AED (Attention Encoder-Decoder) models. The idea comes from the intuition that the transformer encoder with self-attention can already learns to align the input and the output -- which is explicitly modeled by previous approaches like applying techniques to ensure the monotonic alignment or doing dynamic programming to find the alignment. Aligner just simply train with cross-entropy loss, and the results surprisingly show that the encoder can internally learn to align the input and the output during the forward pass. Experimental results showed that Aligner can perform comparably with existing baselines, and show the computational efficiencies.

Strengths

- The motivation about the computational efficiencies of existing models and the idea of developing Aligner is very clear. - The idea is derived from the observation about internal alignment of Transformers, which give meaningful insights to the readers. - It successfully tackles the limitations of existing ASR models with simple and intuitive ways, and the results show that it works well. - The paper includes sufficient amount of evaluation results and showing the limitations (in 4.5.3) as well, providing insights to the readers about these models much. - I feel the writing is also very clear.

Weaknesses

- The idea is yet confirmed in the model fine-tuning setups; as recent Whisper model [1] shows that large-scale pretraining can leads to performant ASR models, I hope this Aligner work applied to the large-scale pretraining setups and show the effectiveness as well. Note that I feel this is not a critical weakness of this paper. - Honestly, I am not following the most recent state-of-the-art models for ASR systems, and not sure if there are some missing baselines that the authors should also compare with. I am willing to listen to other reviewers' opinions about this. [1] Robust Speech Recognition via Large-Scale Weak Supervision, Radford et al, 2022

Questions

No questions for now.

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are adequately addressed.

Summary

This paper proposes a new speech recognition (or, more generally, sequence-to-sequence) architecture. This architecture performs an alignment process between input and output features via self-attention mechanisms in an encoder. The decoder network is a simplified version of the combination of the RNN-T or AED decoder, and its relationship is detailed in Section 2.2. The primary advantage of this method is reducing the computational cost by avoiding the dynamic programming to adjust the input and output length in RNN-T or cross-attention to consider all possible scores across the input frame and output token. The experiments show the comparable performance of the proposed method to RNN-T and AED, but it significantly reduces the computational costs.

Strengths

- Novel speech recognition architecture or, more generally, novel sequence-to-sequence architecture. - Comparable performance to other SOTA speech recognition architecture (AED and RNN-T) while reducing the computational complexity. - Interesting analysis of the alignment behaviors and detailed ablation studies

Weaknesses

- Weak reproducibility due to the use of non-public data for main ASR experiments and the lack of source code release. Note that I did not penalize this point in my initial judgment. But if this part is improved, I'll raise my score.

Questions

- Section 2.1: Do you need $U \leq T$? Can we apply this method for $T \leq U$? This would happen in the general sequence-to-sequence problem like MT or TTS. - Section 4.2: Did you try it with the other encoder layer than the Conformer (e.g., vanilla transformer)? I'm curious because the alignment properties of this method might depend on the convolution operation in the Conformer. - Section 4.5.2: I'm not sure how each attention head behaves. Can you discuss a bit more about how this behavior is different across the head? - Section 4.5: Do you have some results on MT or AST? Suggestions - I recommend the authors emphasize the practical benefit of the computational complexity of this method in the abstract. - Sections 4.5.3 and 4.5.5: These sections provide the main experimental results in the appendix, which are not recommended. This way, it breaks the page limit rule and is unfair compared with the other papers, which put all main results in the main body. These sections should be rewritten to avoid using the appendix results. Note that some supplemental use of the appendix is no problem (e.g., Table 5 is a good example. This is too detailed and may not be so crucial for understanding the main idea of this paper, but it is essential for reproducibility. So, it is adequate to be located in an appendix section.).

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

- Since this is not based on a hard alignment approach (RNN-T and CTC are based on hard alignment), I'm curious about the hallucination issues often observed in AED or decoder-only architectures. For example, OpenAI's Whisper is based on AED (soft alignment). It has a serious hallucination issue (despite its outstanding performance), and I think this is a potential limitation of the soft alignment-based approaches in general. I want the authors to discuss this aspect. This method probably has an advantage over AED due to its shallow decoder architecture, but I'm not very sure.

Summary

A new simplified encoder-decoder model is presented without the attention. The decoder is generating the labels auto-regressively as usual until end-of-sentence (EOS). In contrast to attention-based encoder-decoder (AED) models, the attention is replaced by simply taking the same frame in the encoder - i.e. in decoder step u, it will take frame u also from the encoder. Thus the output sequence can never be longer than the input sequence. The idea is that the encoder with self-attention can already realign the information as necessary to output it label by label. The remaining encoder output frames after EOS are ignored, but of course all intermediate encoder frames are used due to self-attention. It's an interesting test to see whether the self-attention is enough to already learn this. And the answer is yes, it can learn this. Experiments are performed on three speech recognition tasks: * Librispeech with 960h train data * Voice Search with 500kh train data * YouTube videos with 670kh train data In all cases, a Conformer encoder is used. Word pieces are used as output labels. The self-attention weights are analyzed and it is observed that the realignment happens in layer 14. This is also verified in another way, by freezing the first N layers of the encoder, randomly resetting the other encoder parameters, and adding a RNNT on top of it and training that, and then generating the RNNT soft alignment. With N>14, the alignment looks like the identity mapping.

Strengths

Interesting idea and model. The work shows that this simple idea seems to work, even though its performance stays behind the other existing models (CTC/RNNT/AED). Interesting analysis on the attention weights and retraining the encoder partly and looking at the RNNT soft alignment.

Weaknesses

No source code to reproduce the results? No references are given to Voice Search and YouTube data, so it's impossible to reproduce and verify the results.

Questions

How is the convergence rate in comparison to CTC, RNNT, AED? How is the alignment behavior early in training? Table 3, what is dev, is that dev-clean, dev-other, or both combined? Table 3, it seems a bit weird to me that the RNN-T is so much better than the AED model. I would actually expect the opposite. For example: * A Comparison of Sequence-to-Sequence Models for Speech Recognition, https://www.isca-archive.org/interspeech_2017/prabhavalkar17_interspeech.html * On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition, https://www.microsoft.com/en-us/research/uploads/prod/2020/11/template-5fa34dc776e7f.pdf Both show that AED is better than RNN-T. And this is what I have seen on many other occasions as well. Was this expected to you? Why? Or if not, how do you explain it? What happens when f_pred is just a feed-forward network without the recurrence (dependence on g_{i-1}), i.e. you would get a model with only the last label as context? For RNN-T, it has been shown that this performs equally well as using the whole history. It would be interesting to see how it behaves for this model. (For an AED model, this is not really possible because the cross attention mechanism needs it.)

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

-

Discussion period starts

Dear Reviewers, The authors have already posted their rebuttal to address review concerns. It is now the discussion period (Aug. 7 - 13). Please read the author response, and conduct the discussion. Thank you! Thanks, AC

Thanks for your explanation

The answers are valuable (especially for the head and hallucination discussions), but they are not related to the overall discussions. I already gave the accept score and I want to maintain it. I'm looking forward to MT or AST experiments and the open-source implementation of this method.

Reply to the rebuttal

Thanks for the rebuttal. The paper can benefit from the revisions proposed by the authors. I am willing to raise the score to 7 (Accept). In terms of the AED vs. RNN-T, it would be great to clarify the settings and explain the gap. My impression is that RNN-T typically uses a streaming encoder for online purpose, while AED uses non-streaming decoder for offline ASR. AED would always be better in terms of WER. Maybe the authors used the same decoder, and I didn't check the paper again. The authors can use several sentences to clarify the gaps in the revised version of the paper.