Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition

Recently, attention-based encoder-decoder (AED) models have shown high\nperformance for end-to-end automatic speech recognition (ASR) across several\ntasks. Addressing overconfidence in such models, in this paper we introduce the\nconcept of relaxed attention, which is a simple gradual injection of a uniform\ndistribution to the encoder-decoder attention weights during training that is\neasily implemented with two lines of code. We investigate the effect of relaxed\nattention across different AED model architectures and two prominent ASR tasks,\nWall Street Journal (WSJ) and Librispeech. We found that transformers trained\nwith relaxed attention outperform the standard baseline models consistently\nduring decoding with external language models. On WSJ, we set a new benchmark\nfor transformer-based end-to-end speech recognition with a word error rate of\n3.65%, outperforming state of the art (4.20%) by 13.1% relative, while\nintroducing only a single hyperparameter.\n