Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus

End-to-end (E2E) automatic speech recognition (ASR) systems lack the distinct\nlanguage model (LM) component that characterizes traditional speech systems.\nWhile this simplifies the model architecture, it complicates the task of\nincorporating text-only data into training, which is important to the\nrecognition of tail words that do not occur often in audio-text pairs. While\nshallow fusion has been proposed as a method for incorporating a pre-trained LM\ninto an E2E model at inference time, it has not yet been explored for very\nlarge text corpora, and it has been shown to be very sensitive to\nhyperparameter settings in the beam search. In this work, we apply shallow\nfusion to incorporate a very large text corpus into a state-of-the-art E2EASR\nmodel. We explore the impact of model size and show that intelligent pruning of\nthe training set can be more effective than increasing the parameter count.\nAdditionally, we show that incorporating the LM in minimum word error rate\n(MWER) fine tuning makes shallow fusion far less dependent on optimal\nhyperparameter settings, reducing the difficulty of that tuning problem.\n