RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank

Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid\ndevelopment, with the emergence of many method variations but only few\nprincipled guidelines that would help practitioners to successfully deploy\nthem. The main reason for that pitfall comes from JE-SSL's core principle of\nnot employing any input reconstruction therefore lacking visual cues of\nunsuccessful training. Adding non informative loss values to that, it becomes\ndifficult to deploy SSL on a new dataset for which no labels can help to judge\nthe quality of the learned representation. In this study, we develop a simple\nunsupervised criterion that is indicative of the quality of the learned JE-SSL\nrepresentations: their effective rank. Albeit simple and computationally\nfriendly, this method -- coined RankMe -- allows one to assess the performance\nof JE-SSL representations, even on different downstream datasets, without\nrequiring any labels. A further benefit of RankMe is that it does not have any\ntraining or hyper-parameters to tune. Through thorough empirical experiments\ninvolving hundreds of training episodes, we demonstrate how RankMe can be used\nfor hyperparameter selection with nearly no reduction in final performance\ncompared to the current selection method that involve a dataset's labels. We\nhope that RankMe will facilitate the deployment of JE-SSL towards domains that\ndo not have the opportunity to rely on labels for representations' quality\nassessment.\n