ScatSimCLR: self-supervised contrastive learning with pretext task regularization for small-scale datasets

In this paper, we consider a problem of self-supervised learning for\nsmall-scale datasets based on contrastive loss between multiple views of the\ndata, which demonstrates the state-of-the-art performance in classification\ntask. Despite the reported results, such factors as the complexity of training\nrequiring complex architectures, the needed number of views produced by data\naugmentation, and their impact on the classification accuracy are understudied\nproblems. To establish the role of these factors, we consider an architecture\nof contrastive loss system such as SimCLR, where baseline model is replaced by\ngeometrically invariant "hand-crafted" network ScatNet with small trainable\nadapter network and argue that the number of parameters of the whole system and\nthe number of views can be considerably reduced while practically preserving\nthe same classification accuracy. In addition, we investigate the impact of\nregularization strategies using pretext task learning based on an estimation of\nparameters of augmentation transform such as rotation and jigsaw permutation\nfor both traditional baseline models and ScatNet based models. Finally, we\ndemonstrate that the proposed architecture with pretext task learning\nregularization achieves the state-of-the-art classification performance with a\nsmaller number of trainable parameters and with reduced number of views.\n