Transformer-based pretrained language models (PLMs) have started a new era in\nmodern natural language processing (NLP). These models combine the power of\ntransformers, transfer learning, and self-supervised learning (SSL). Following\nthe success of these models in the general domain, the biomedical research\ncommunity has developed various in-domain PLMs starting from BioBERT to the\nlatest BioELECTRA and BioALBERT models. We strongly believe there is a need for\na survey paper that can provide a comprehensive survey of various\ntransformer-based biomedical pretrained language models (BPLMs). In this\nsurvey, we start with a brief overview of foundational concepts like\nself-supervised learning, embedding layer and transformer encoder layers. We\ndiscuss core concepts of transformer-based PLMs like pretraining methods,\npretraining tasks, fine-tuning methods, and various embedding types specific to\nbiomedical domain. We introduce a taxonomy for transformer-based BPLMs and then\ndiscuss all the models. We discuss various challenges and present possible\nsolutions. We conclude by highlighting some of the open issues which will drive\nthe research community to further improve transformer-based BPLMs.\n