Summary
This paper investigates the complementary benefits of combining self-training and contrastive pretraining for domain adaptation under distribution shifts. Through an empirical study on 8 benchmarks, the authors demonstrate that applying self-training (FixMatch) after contrastive pretraining (SwAV) yields substantial accuracy gains over either approach alone. To understand this synergy, the authors analyze a simplified theoretical setup indicating that while contrastive pretraining can amplify signal along invariant features, it may retain dependence on spurious source-only correlations. On the other hand, self-training can effectively unlearn these spurious dependencies and improve linear transfer of representations, achieving optimal target performance. The authors further apply linear probing on representations to verify their theoretical findings empirically. Overall, this work highlights the potential of combining self-training and contrastive learning to address challenges posed by distribution shifts, grounded by both empirical evidence and theoretical analysis.
Strengths
**Originality**: The paper explores the promising yet underexplored direction of combining self-training and contrastive learning to address distribution shifts. The extensive empirical study systematically investigates their complementary benefits across diverse benchmarks. The theoretical analysis offers unique insights into their synergistic effects.
**Quality**: The empirical study is relatively extensive, spanning 8 datasets with careful controls. Despite simplifying assumptions, the theory makes falsifiable predictions that are precisely analyzed and align well with observations. Additional probing experiments further validate the theoretical intuitions.
**Clarity**: The paper is clearly presented overall. The problem setup and motivation are lucid, and the methods and experiments are thoroughly explained. The authors effectively convey the core conceptual messages and insights from their theoretical analysis.
**Significance**: Distribution shifts ubiquitously challenge ML model generalization in practice. This paper highlights the significant potential of combining self-training and contrastive learning to address this problem, demonstrating substantial empirical gains and proposing explanations grounded in both theory and experiments. Overall, I think this paper will be of interest to the ML research community of distribution shifts.
Weaknesses
**Small Pre-training Set**: The authors have chosen to train models using contrastive learning (CL) from scratch on a selection of small datasets from BREEDS and WILDS. The authors have justified this approach by stating, "We have opted not to use off-the-shelf pretrained models (e.g., on Imagenet [67]) to avoid confounding our conclusions about contrastive pretraining." However, in practice, pretraining CL models on small datasets is not a common practice, given the availability of powerful off-the-shelf CL models that have been pretrained on larger datasets (e.g., from SimCLR, iBot to DINOv2). If the paper's results only apply to CL models pretrained on small datasets and do not generalize to more powerful contemporary CL models, the utility of this work could be limited. I would recommend that the authors conduct studies of modern off-the-shelf CL models to strengthen this work. For instance, off-the-shelf ImageNet-pretrained CL models could be used as initialization, followed by self-training, and then evaluation on various shifted datasets (numerous ImageNet shifted datasets are available). Please note, it is not necessary to strive for results during the rebuttal period. This paper is already commendable in my view, I merely want to offer suggestions that could further enhance its impact.
**Limited Combination of CL+ST Algorithms** In this study, the authors have exclusively utilized SwAV+FixMatch, which are indeed effective algorithms. However, the presentation would be more compelling if a wider range of combinations were explored. Given that the authors employ Barlow Twins in their theoretical analysis, would it not be more fitting to also include Barlow Twins as a CL algorithm in the empirical studies?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
See Weaknesses.