Despite the success of a number of recent techniques for visual\nself-supervised deep learning, there has been limited investigation into the\nrepresentations that are ultimately learned. By leveraging recent advances in\nthe comparison of neural representations, we explore in this direction by\ncomparing a contrastive self-supervised algorithm to supervision for simple\nimage data in a common architecture. We find that the methods learn similar\nintermediate representations through dissimilar means, and that the\nrepresentations diverge rapidly in the final few layers. We investigate this\ndivergence, finding that these layers strongly fit to their distinct learning\nobjectives. We also find that the contrastive objective implicitly fits the\nsupervised objective in intermediate layers, but that the reverse is not true.\nOur work particularly highlights the importance of the learned intermediate\nrepresentations, and raises critical questions for auxiliary task design.\n