A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks

Deep neural networks are typically trained under a supervised learning\nframework where a model learns a single task using labeled data. Instead of\nrelying solely on labeled data, practitioners can harness unlabeled or related\ndata to improve model performance, which is often more accessible and\nubiquitous. Self-supervised pre-training for transfer learning is becoming an\nincreasingly popular technique to improve state-of-the-art results using\nunlabeled data. It involves first pre-training a model on a large amount of\nunlabeled data, then adapting the model to target tasks of interest. In this\nreview, we survey self-supervised learning methods and their applications\nwithin the sequential transfer learning framework. We provide an overview of\nthe taxonomy for self-supervised learning and transfer learning, and highlight\nsome prominent methods for designing pre-training tasks across different\ndomains. Finally, we discuss recent trends and suggest areas for future\ninvestigation.\n