Language modeling (LM) for automatic speech recognition (ASR) does not\nusually incorporate utterance level contextual information. For some domains\nlike voice assistants, however, additional context, such as the time at which\nan utterance was spoken, provides a rich input signal. We introduce an\nattention mechanism for training neural speech recognition language models on\nboth text and non-linguistic contextual data. When applied to a large\nde-identified dataset of utterances collected by a popular voice assistant\nplatform, our method reduces perplexity by 7.0% relative over a standard LM\nthat does not incorporate contextual information. When evaluated on utterances\nextracted from the long tail of the dataset, our method improves perplexity by\n9.0% relative over a standard LM and by over 2.8% relative when compared to a\nstate-of-the-art model for contextual LM.\n