Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation

End-to-end automatic speech recognition (ASR), unlike conventional ASR, does\nnot have modules to learn the semantic representation from speech encoder.\nMoreover, the higher frame-rate of speech representation prevents the model to\nlearn the semantic representation properly. Therefore, the models that are\nconstructed by the lower frame-rate of speech encoder lead to better\nperformance. For Transformer-based ASR, the lower frame-rate is not only\nimportant for learning better semantic representation but also for reducing the\ncomputational complexity due to the self-attention mechanism which has O(n^2)\norder of complexity in both training and inference. In this paper, we propose a\nTransformer-based ASR model with the time reduction layer, in which we\nincorporate time reduction layer inside transformer encoder layers in addition\nto traditional sub-sampling methods to input features that further reduce the\nframe-rate. This can help in reducing the computational cost of the\nself-attention process for training and inference with performance improvement.\nMoreover, we introduce a fine-tuning approach for pre-trained ASR models using\nself-knowledge distillation (S-KD) which further improves the performance of\nour ASR model. Experiments on LibriSpeech datasets show that our proposed\nmethods outperform all other Transformer-based ASR systems. Furthermore, with\nlanguage model (LM) fusion, we achieve new state-of-the-art word error rate\n(WER) results for Transformer-based ASR models with just 30 million parameters\ntrained without any external data.\n