Transformer-based pretrained language models (T-PTLMs) have achieved great\nsuccess in almost every NLP task. The evolution of these models started with\nGPT and BERT. These models are built on the top of transformers,\nself-supervised learning and transfer learning. Transformed-based PTLMs learn\nuniversal language representations from large volumes of text data using\nself-supervised learning and transfer this knowledge to downstream tasks. These\nmodels provide good background knowledge to downstream tasks which avoids\ntraining of downstream models from scratch. In this comprehensive survey paper,\nwe initially give a brief overview of self-supervised learning. Next, we\nexplain various core concepts like pretraining, pretraining methods,\npretraining tasks, embeddings and downstream adaptation methods. Next, we\npresent a new taxonomy of T-PTLMs and then give brief overview of various\nbenchmarks including both intrinsic and extrinsic. We present a summary of\nvarious useful libraries to work with T-PTLMs. Finally, we highlight some of\nthe future research directions which will further improve these models. We\nstrongly believe that this comprehensive survey paper will serve as a good\nreference to learn the core concepts as well as to stay updated with the recent\nhappenings in T-PTLMs.\n