A common practice in unsupervised representation learning is to use labeled\ndata to evaluate the quality of the learned representations. This supervised\nevaluation is then used to guide critical aspects of the training process such\nas selecting the data augmentation policy. However, guiding an unsupervised\ntraining process through supervised evaluations is not possible for real-world\ndata that does not actually contain labels (which may be the case, for example,\nin privacy sensitive fields such as medical imaging). Therefore, in this work\nwe show that evaluating the learned representations with a self-supervised\nimage rotation task is highly correlated with a standard set of supervised\nevaluations (rank correlation $> 0.94$). We establish this correlation across\nhundreds of augmentation policies, training settings, and network architectures\nand provide an algorithm (SelfAugment) to automatically and efficiently select\naugmentation policies without using supervised evaluations. Despite not using\nany labeled data, the learned augmentation policies perform comparably with\naugmentation policies that were determined using exhaustive supervised\nevaluations.\n