Recently, text world games have been proposed to enable artificial agents to\nunderstand and reason about real-world scenarios. These text-based games are\nchallenging for artificial agents, as it requires an understanding of and\ninteraction using natural language in a partially observable environment.\nAgents observe the environment via textual descriptions designed to be\nchallenging enough for even human players. Past approaches have not paid enough\nattention to the language understanding capability of the proposed agents.\nTypically, these approaches train from scratch, an agent that learns both\ntextual representations and the gameplay online during training using a\ntemporal loss function. Given the sample-inefficiency of RL approaches, it is\ninefficient to learn rich enough textual representations to be able to\nunderstand and reason using the textual observation in such a complicated game\nenvironment setting. In this paper, we improve the semantic understanding of\nthe agent by proposing a simple RL with LM framework where we use\ntransformer-based language models with Deep RL models. We perform a detailed\nstudy of our framework to demonstrate how our model outperforms all existing\nagents on the popular game, Zork1, to achieve a score of 44.7, which is 1.6\nhigher than the state-of-the-art model. Overall, our proposed approach\noutperforms 4 games out of the 14 text-based games, while performing comparable\nto the state-of-the-art models on the remaining games.\n