Our world is open-ended, non-stationary, and constantly evolving; thus what\nwe talk about and how we talk about it change over time. This inherent dynamic\nnature of language contrasts with the current static language modelling\nparadigm, which trains and evaluates models on utterances from overlapping time\nperiods. Despite impressive recent progress, we demonstrate that Transformer-XL\nlanguage models perform worse in the realistic setup of predicting future\nutterances from beyond their training period, and that model performance\nbecomes increasingly worse with time. We find that, while increasing model size\nalone -- a key driver behind recent progress -- does not solve this problem,\nhaving models that continually update their knowledge with new information can\nindeed mitigate this performance degradation over time. Hence, given the\ncompilation of ever-larger language modelling datasets, combined with the\ngrowing list of language-model-based NLP applications that require up-to-date\nfactual knowledge about the world, we argue that now is the right time to\nrethink the static way in which we currently train and evaluate our language\nmodels, and develop adaptive language models that can remain up-to-date with\nrespect to our ever-changing and non-stationary world. We publicly release our\ndynamic, streaming language modelling benchmarks for WMT and arXiv to\nfacilitate language model evaluation that takes temporal dynamics into account.\n