Understanding self-supervised Learning Dynamics without Contrastive Pairs

While contrastive approaches of self-supervised learning (SSL) learn\nrepresentations by minimizing the distance between two augmented views of the\nsame data point (positive pairs) and maximizing views from different data\npoints (negative pairs), recent \\emph{non-contrastive} SSL (e.g., BYOL and\nSimSiam) show remarkable performance {\\it without} negative pairs, with an\nextra learnable predictor and a stop-gradient operation. A fundamental question\narises: why do these methods not collapse into trivial representations? We\nanswer this question via a simple theoretical study and propose a novel\napproach, DirectPred, that \\emph{directly} sets the linear predictor based on\nthe statistics of its inputs, without gradient training. On ImageNet, it\nperforms comparably with more complex two-layer non-linear predictors that\nemploy BatchNorm and outperforms a linear predictor by $2.5\\%$ in 300-epoch\ntraining (and $5\\%$ in 60-epoch). DirectPred is motivated by our theoretical\nstudy of the nonlinear learning dynamics of non-contrastive SSL in simple\nlinear networks. Our study yields conceptual insights into how non-contrastive\nSSL methods learn, how they avoid representational collapse, and how multiple\nfactors, like predictor networks, stop-gradients, exponential moving averages,\nand weight decay all come into play. Our simple theory recapitulates the\nresults of real-world ablation studies in both STL-10 and ImageNet. Code is\nreleased https://github.com/facebookresearch/luckmatters/tree/master/ssl.\n