Self-supervised learning solves pretext prediction tasks that do not require\nannotations to learn feature representations. For vision tasks, pretext tasks\nsuch as predicting rotation, solving jigsaw are solely created from the input\ndata. Yet, predicting this known information helps in learning representations\nuseful for downstream tasks. However, recent works have shown that wider and\ndeeper models benefit more from self-supervised learning than smaller models.\nTo address the issue of self-supervised pre-training of smaller models, we\npropose Distill-on-the-Go (DoGo), a self-supervised learning paradigm using\nsingle-stage online knowledge distillation to improve the representation\nquality of the smaller models. We employ deep mutual learning strategy in which\ntwo models collaboratively learn from each other to improve one another.\nSpecifically, each model is trained using self-supervised learning along with\ndistillation that aligns each model's softmax probabilities of similarity\nscores with that of the peer model. We conduct extensive experiments on\nmultiple benchmark datasets, learning objectives, and architectures to\ndemonstrate the potential of our proposed method. Our results show significant\nperformance gain in the presence of noisy and limited labels and generalization\nto out-of-distribution data.\n