Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation

Summary

The authors claim to have addressed the new problem of semi-supervised learning with feature distribution mismatch (FDM-SSL) and to have proposed a new method called Self-Supervised Feature Adaptation (SSFA) for this problem. The proposed SSFA consists of semi-supervised learning module and a feature adaptation module. The semi-supervised learning module optimizes the classifier based on joint minimization of supervised/unsupervised losses (to pseudo-labels) and a self-supervised auxiliary loss (for rotation prediction) with weak/strong data augmentation. The feature adaptation module aims to generate more reliable pseudo labels by updating the feature extraction backbone based on the self-supervised auxiliary loss on the unlabeled samples. Comparative experiments with existing semi-supervised learning are conducted and the effectiveness of the proposed method is claimed.

Strengths

This paper addresses the FDM-SSL problem, which has few studied examples. The method is simple and its superiority over the major semi-supervised learning methods in the FDM-SSL problem is experimentally demonstrated.

Weaknesses

A. Problem Novelty The authors claim in the abstract and conclusion sections that one of the novelties of this paper is in that they tackle the novel problem of semi-supervised learning with feature distribution mismatch. However, this claim is not valid because previous work [a] has addressed a very similar problem. The differences between these two should be emphasized and clarified. Furthermore, there are also several closely related problems that have been considered in the past (e.g., semi-supervised domain generalization [b] and few-shot domain generalization [c]), but discussion of their relevance is lacking. [a] Aminian et al., An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift. ICML, 2022. [b] Zhou et al., Semi-Supervised Domain Generalization with Stochastic StyleMatch. NeurIPS Workshop, 2021. (Extended version is published at IJCV) [c] Yuan et al., A novel forget-update module for few-shot domain generalization. Pattern Recognition, 2022. B. Experiments Comparisons with existing methods for the related problems I listed in Weakness A above are missing. Furthermore, since the problem of FDM-SSL can be considered a variant of domain generalization, there should be comparisons with existing domain generalization (and more recent domain adaptation) methods. Due to the lack of such an evaluation, the effectiveness of the proposed method is not adequately demonstrated. C. Method Novelty The novelty of the proposed method is lacking. It is basically composed of common techniques in semi-supervised learning (such as weak/strong data augmentation or pseudo labeling with confidence thresholding). The feature adaptation module, which performs fine-tuning of the feature extraction backbone based on self-supervised auxiliary loss, has not been explored in semi-supervised learning, but the method itself is borrowed from [30].

Questions

I have no specific questions for now. I would like to ask the authors to point out any misunderstandings I have about the above points I have raised as weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Its limitations are discussed in the concluding section. It would be interesting to add a discussion of the performance of the proposed method in the presence of another often discussed realistic distribution mismatch, class distribution mismatch (open set semi-supervised learning).

Summary

The paper challenges the traditional assumption of semi-supervised learning: the feature distributions of labeled and unlabeled data are consistent. It claims that this assumption rarely holds in realistic scenarios, as: (a) unlabeled samples could contain various corruptions; (b) unlabeled samples could contain unseen styles. For this, the authors propose Self-Supervised Feature Adaptation (SSFA), a generic framework for improving SSL performance when labeled and unlabeled data come from different distributions. SSFA consists of a semi-supervised learning module and a feature adaptation module, where the semi-supervised learning module performs semi-supervised and self-supervised learning

Strengths

(1) The paper is well written and easy to follow, presentation is good. (2) The proposed method is simple and effective. It consistently improves over its vanilla SSL algorithm counterpart. (3) The paper conducts extensive experiments on two realistic scenarios and ablation study to validate the contribution of each component and the overall method.

Weaknesses

(1) The proposed method has limited novelty. Incorporating self-supervised learning into semi-supervised learning, or leveraging self-supervised learning for domain adaptation, both are not new conceptions. (2) Some important technical details are missing in the experimental setting section. For example, what is the number of shared layers and the self-supervised learning algorithm by default. These are important for reproducibility.

Questions

(1) For Table 1, I am curious about the proposed method's performance on standard SSL benchmark, or when the ratio is low (like 0.1), this is a measure of whether the proposed method is robust to any scenarios, even if the noise is not that much. (2) The ablation study shows that the number of shared layers won't affect much as long as not sharing all, I would suggest the authors to conduct experiments on more backbones and get some empirical conclusion on how to set this hyper-parameter.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitation of this work is that the performance of SSFA is affected by the shared parameters between the main task and the auxiliary task. The ablation study shows that the performances do not differ much as long as not all parameters are shared. The number of layers to share should be a hyper-parameter to tune.

Summary

In this work, the authors propose a generalized Self-Supervised Feature Adaptation (SSFA) framework for FDM-SSL (Feature Distribution Mismatched Semi-Supervised Learning) that does not require prior knowledge of the distribution of unlabeled data. The SSFA framework aims to address distribution mismatch by decoupling pseudo-label predictions from the current model. It consists of two modules: the semi-supervised learning module and the feature adaptation module. The authors draw inspiration from previous work on auxiliary tasks and incorporate an auxiliary self-supervised task into the SSL module to train alongside the main task. In the feature adaptation module, the current model, primarily trained on labeled data, is updated by utilizing the self-supervised task to adapt the feature extractor before making predictions on unlabeled data. This adaptation allows for the generation of more accurate pseudo-labels, which in turn assist the SSL process.

Strengths

- Addition of the feature adaptation module to adapt the model (trained on labeled data) to the unlabeled data distribution before generating pseudo labels seems to help with the distribution shift problem between labeled and unlabeled data. - Extensive empirical evidence provided to show that proposed SSFA method significantly improves classification accuracy on CIFAR100 and OfficeHome and Office31 datasets. Domain-level and class-level feature visualization also suggest successful feature adaptation across different domains and learning of more distinguishable representations across different classes. Ablations studies investigating the effect of different self-supervised tasks, number of shared layers between auxiliary and main task, and the sensivity of the method to different confidence thresholds are also useful.

Weaknesses

- Lemma 1 is fine but convexity is a very strong assumption, which does not hold true in real-world deep loss functions. Some empirical evidence is provided, but it is somewhat counterintuitive to think that minimizing self-supervised empirical loss will also minimize supervised empirical loss. - Using weakly augmented samples for a supervised loss and strongly augmented samples for an unsupervised loss is not novel and first introduced in the FixMatch paper. Technical novelty is somewhat incremental though feature adaptation module can still be considered a significant improvement.

Questions

It was not very clear from the text about how the feature adaptation model is being trained. The module is being trained for one iteration and then is being discarded. This part is confusing. Is the module trained one unlabeled sample at a time or using all unlabeled samples? If the module needs all unlabeled samples to be trained this may not reflect a real-world use case scenario and will limit the applicability of the approach. Figure 1 needs to be updated to reflect these important details. What kind of self-supervised loss is used in equation 5? Please define G in Lemma 1. What does subscript m in the loss function in Lemma 1 indicates? It was not previously used in defining loss function. Please correct the typos in Table 1. "Traditioal", Abundant/Scarce could replace Plenty/Lack. ------------- Your responses have clarified some of the concerns I had with Lemma 1. In particular, I acknowledge your explanations regarding the training process of the feature adaptation module, and the flexible nature of the self-supervised loss function. I am inclined to update my initial score to reflect these additional insights your rebuttal offered. Thanks.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Conclusions allude to some algorithmic limitations yet do not discuss societal impact. No negative societal impact beyond what is already present in generic machine learning algorithms has been observed.

Summary

The authors propose an interesting and novel semi-supervised learning method with auxiliary self-supervised feature adaptation for addressing issues with heterogeneity in labelled and unlabelled data. In fact, the authors focus on heterogeneity across domains, which is not something traditionally considered in the SSL literature, but is more in the field of unsupervised domain adaptation. The authors combines ideas of feature adaptation from UDA with consistency regularization and pseudo-labelling from SSL to bridge the two fields. The results presented seem much better than the baseline methods.

Strengths

1. Table 1 is useful and sets the tone of the paper showing the problem settings and constraints of each type of approach. 2. The performance metrics reported in Table 2 and 3 are very impressive compared to baselines.

Weaknesses

1. However, the caveat is that the baseline methods cannot do both SSL and UDA, which makes comparisons of performance metrics harder. However, SSFA does seem to be a novel method bridging the two fields. Minor points: 1. Typo in Table 1: traditional 2. Might be useful to point out more forcefully Eq 2 is traditional in SSL, but Eq 7 is actually used, especially as SSFA Algorithm with equation references is in Supplement. Maybe it would be useful to not call both Eq 2 and Eq 7, L_u. 3. How was the cluster representation in Figure 4 made?

Questions

1. The Feature Adaptation Module seems like 1-directional (unlabelled to labelled) and 1step transformation, as the unlabelled are aligned to the labelled. Have the authors tried using more steps in the optimization to allow more adaptation, or does that rely on the loss and step calibration? 2. The authors show the effect of different fixed thresholds in Eq 7 in Figure 5. Have the authors tried an adaptive threshold as in FlexMatch? 3. While the authors have compared SSFA to more recent and state-of-the-art models in SSL such as FreeMatch and SoftMatch, the authors have not compared against the arguably the state-of-the-art in UDA such PMTrans. How does SSFA compare against PMTrans or any of the more recent leaders the Domain Adaptation leaderboard on Office-31 and Office-Home? 4. Moreover, SSFA is especially impressive for low number of labelled data against UDA methods. How does it do in low labelled data settings against more recent UDA methods?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors have addressed some limitations of their method

Summary

This paper focuses on Feature Distribution Mismatch SSL (FDM-SSL), which considers a mismatch between the labeled and unlabeled data. In this task setting, the challenges include the scarcity of labeled data and the mixed distributions of the unlabeled data. The paper proposes a framework named Self-Supervised Feature Adaptation (SSFA), which adapts the feature extractor to the unlabeled distribution. Experiments show that SSFA can improve SSL performance on labeled, unlabeled, and unseen data.

Strengths

1. The paper studies a new SSL problem named Feature Distribution Mismatch SSL. The presented SSFA method is simple and effective to address this problem. 2. Extensive experiments have been conducted to evaluate the performance of SSFA. 3. The paper is well written and presented. For instance, Table 2 is clear, it reveals the difference between task settings.

Weaknesses

1. The proposed SSFA method is built upon existing SSL and unsupervised domain adaptation (UDA) methods. In SSFA, the SSL module and Feature Adaptation Module are inherited from the benchmark SSL and UDA methods, respectively. The optimization objectives $L_x$, $L_u$, and $L_{aux}$ are widely used in SSL methods. The major novelty is the Feature Adaptation Module, which uses the unlabeled data as input to generate pseudo labels $q_b'$ instead of the original pseudo labels $q_b$ generated from the labeled data. I acknowledge that this work presents some novelties in proposing and addressing the challenges in domain-shifted SSL. However, the method itself is straightforward in my opinion. 2. The assumptions of Lemma 1 are somewhat strong, i.e., the loss is expected to be convex and smooth, the learning rate is fixed to $\frac{\epsilon}{\beta G^2}$. In addition, the result of Lemma 1 is counter-intuitive --- 'The empirical risk of the main task can be theoretically reduced to 0 by optimizing the empirical risk of self-supervised task.' Due to these aspects, I kindly doubt that this theoretical result could work well in practice. And, the derivation only guarantees that the empirical risk is decreasing along with gradient descent, but not 'reduced to 0'. Please check this. 3. I am a bit confused about the SSFA's theoretical performance on 'unseen data'. In Sections Introduction and Experiments, the authors claimed that SSFA can significantly improve SSL performance on unseen data. However, the method only adapts models to the *unlabeled* data during training. Why does SSFA improve the performance on unseen data? More theoretical evidence is encouraged to be presented to better support this claim. 4. I suggest the authors show some visualizations of the gap between labeled and unlabeled data, e.g., what are the image corruptions in the experiments? What's the style change from OFFICE-31 dataset to OFFICE-HOME dataset? I find some related details in Appendix's texts. However, it would be more friendly to show some visualizations of the data gap, since domain shifting is the main challenge studied by this work.

Questions

See Weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

None

Question to authors' rebuttal

Thanks to the authors for their efforts in answering my questions. Some of my questions have been resolved. I have a question to the answer to A. Problem Novelty. Despite the fact that some past work like [a] has already considered the feature distribution mismatche between labeled and unlabeled samples, we can see that this paper contains claims that contradict this point, for example, "In this work, we introduce a novel setting to formalize the feature distribution mismatch between the labeled and unlabeled samples" in the abstract. Such statements overclaim the contributions of this paper and may mislead readers, so they should be corrected. Does the author agree with this? If yes, I would like to see a brief plan on how it could be fixed.