Pre-training large language models has become a standard in the natural\nlanguage processing community. Such models are pre-trained on generic data\n(e.g. BookCorpus and English Wikipedia) and often fine-tuned on tasks in the\nsame domain. However, in order to achieve state-of-the-art performance on out\nof domain tasks such as clinical named entity recognition and relation\nextraction, additional in domain pre-training is required. In practice, staged\nmulti-domain pre-training presents performance deterioration in the form of\ncatastrophic forgetting (CF) when evaluated on a generic benchmark such as\nGLUE. In this paper we conduct an empirical investigation into known methods to\nmitigate CF. We find that elastic weight consolidation provides best overall\nscores yielding only a 0.33% drop in performance across seven generic tasks\nwhile remaining competitive in bio-medical tasks. Furthermore, we explore\ngradient and latent clustering based data selection techniques to improve\ncoverage when using elastic weight consolidation and experience replay methods.\n