End-to-end automatic speech recognition (ASR) systems are increasingly\npopular due to their relative architectural simplicity and competitive\nperformance. However, even though the average accuracy of these systems may be\nhigh, the performance on rare content words often lags behind hybrid ASR\nsystems. To address this problem, second-pass rescoring is often applied\nleveraging upon language modeling. In this paper, we propose a second-pass\nsystem with multi-task learning, utilizing semantic targets (such as intent and\nslot prediction) to improve speech recognition performance. We show that our\nrescoring model trained with these additional tasks outperforms the baseline\nrescoring model, trained with only the language modeling task, by 1.4% on a\ngeneral test and by 2.6% on a rare word test set in terms of word-error-rate\nrelative (WERR). Our best ASR system with multi-task LM shows 4.6% WERR\ndeduction compared with RNN Transducer only ASR baseline for rare words\nrecognition.\n