(FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning

Federated Learning (FL) is a distributed machine learning framework that trains accurate global models while preserving clients' privacy-sensitive data. However, most FL approaches assume that clients possess labeled data, which is often not the case in practice. Federated Semi-Supervised Learning (FSSL) addresses this label deficiency problem, targeting situations where only the server has a small amount of labeled data while clients do not. However, a significant performance gap exists between Centralized Semi-Supervised Learning (SSL) and FSSL. This gap arises from confirmation bias, which is more pronounced in FSSL due to multiple local training epochs and the separation of labeled and unlabeled data. We propose $(FL)^2$, a robust training method for unlabeled clients using sharpness-aware consistency regularization. We show that regularizing the original pseudo-labeling loss is suboptimal, and hence we carefully select unlabeled samples for regularization. We further introduce client-specific adaptive thresholding and learning status-aware aggregation to adjust the training process based on the learning progress of each client. Our experiments on three benchmark datasets demonstrate that our approach significantly improves performance and bridges the gap with SSL, particularly in scenarios with scarce labeled data.

Paper

References (47)

Scroll for more · 35 remaining

Similar papers

Peer review

Reviewer ytwd6/10 · confidence 4/52024-07-02

Summary

This work addresses a very practical challenge against successful FL deployments, which is of unlabeled data at FL clients. Furthermore, the problem is set in the regime of low count of labeled samples at server. The proposed solution to train a model in semi-supervised manner includes having an adaptive confidence threshold for each client to get pseudo-labels of more samples in the initial stages of training, then updating the model by perturbing the weights and training them on high-confidence pseudo labels; and lastly by aggregating the model weights through a learning status aware hyperparameter. The results show superiority of the proposed method against existing state-of-the-art methods for federated self-supervised learning, and shows why naive application of centralized self-supervised methods are not cut for the disjoint nature of server and clients in FL.

Strengths

1. The paper is written very well. The flow of logic is mostly clear. 2. The issue this paper is tackling is very important (especially low labeled sample count at the server), and the solution proposed is elegant (clearly states why and how the existing centralized semi-supervised methods are not enough, which brings a unique solution for the FL setting). 3. Strong results with comprehensive experiments.

Weaknesses

1. As expanded in Questions, some parts of the methodology is unclear. As an example, why do we need Eq 10 (unsupervised training objective) at the stage of adaptive thresholding? I thought we are just getting the confidence threshold for pseudo-labels and sending it to the server. Another example where the methodology was slightly fuzzy is around line 193, "While we use client-specific adaptive threshold, we use a high fixed threshold to get high-confidence data samples". What if that fixed threshold is not set correctly? The goal of dynamic client threshold was to avoid the sub-optimal results of a fixed threshold, wouldn't the same issues arise for this adversarial perturbations? 2. The authors have tried two datasets: CIFAR10 and SVHN. I wonder how this translates to harder classification problems with more classes (where a few classes might not have representative samples at server and many clients just do not have samples related to those classes at all) or predicting classes or next words on natural language datasets.

Questions

1. In Line 68, "Clients with lower learning status (e.g., whose models are less certain about their predictions) receive higher aggregation weights, ensuring their updates are more significantly reflected in the global model." It's unclear why do we need the above? Shouldn't low confidence show bad generalizability of the trained model? If yes, I didn't catch how the authors are preventing the model from learning on "wrong" input-output pairs. 2. Related to Question #1, in what cases would the learning status be low? And what was the intuition behind giving high importance to those low status clients during the aggregation? What if this leads to some other clients getting low learning status then? 3. Why does $(FL)^2$ need strongly augmented samples for SACR? What happens if we just use weakly-augmented samples instead? 4. A minor suggestion: Figure 2 can benefit from numbers to show the flow of what happens after what. It was difficult to figure out where to start reading the diagram from.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

I do not see a limitations section. Although the authors mention that the limitations are mentioned in Conclusion, I am not sure what they are referring to (not having theoretical analysis does not sound like a limitation of the proposed method, more like potential future work).

Reviewer YhMN5/10 · confidence 4/52024-07-08

Summary

This paper focuses on the federated semi-supervised learning (FSSL) scenario, which is a more challenging problem in FL. There are two different scenarios in FL, labels-at-server and labels-at-clients and this paper tackles the former issues. The author found the gap between SSL and FSSL is confirmation bias. To diminish this bias, this paper proposes client-specific adaptive threshold, modified SAM, and learning status-aware aggregation for a new aggregation scheme. The experiments of $(FL)^2$ demonstrate improvement on two benchmark datasets.

Strengths

This paper is well-organized and easy to follow with clear contribution.

Weaknesses

- Lack of motivation and insights. It is not clear how the performance is influenced by confirmation bias Moreover, while the technologies discussed in this paper are not new, the argument that their particular combination can mitigate confirmation bias lacks persuasiveness. - The related work about labels-at-clients is outdated. Some new works should be included, e.g., [R1][R2]. - The experiments of this paper are insufficient. E.g., How $(FL)^2$ tackles the confirmation bias is not mentioned. This paper should add more ablation studies to prove the bias is decreased rather than only the performance. There are many reasons to get the improvement. Reference:\ [R1] Li, Ming, et al. Class balanced adaptive pseudo labeling for federated semi-supervised learning. In CVPR, 2023.\ [R2] Zhang, Yonggang, et al. Robust Training of Federated Models with Extremely Label Deficiency. In ICLR, 2024.

Questions

- I go through the source code and find that implementing FedMatch may lack some components, e.g., parameter decomposition for disjoint learning and K-Dimensional Tree for helper selection. So the incomplete baseline is not convincing to compare with the proposed methods. - Why the performance of SemiFL under SVHN dataset with Balanced IID, 250 labeled samples is lower than 40 samples which is counterintuitive. - The performance of SemiFL under CIFAR10 dataset with Unbalanced Non-IID, and Balanced IID is 10.0. This seems to be a training problem.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

- The dataset is limited, there are also CIFAR-100 and FMNIST which are most used in FL research. - The limitations of this work are not well-discussed.

Reviewer dbte7/10 · confidence 4/52024-07-10

Summary

This paper studies the federated semi-supervised learning (FSSL) problem. A significant gap between the centralized semi-supervised learning and FSSL is found due to the confirmation bias. To address this issue, the current paper proposes a new FSSL algorithm, by incorporating three new ideas, namely client-specific adaptive threshold, sharpness-aware consistency regularization, and learning status-aware aggregation. Experimental results show that the proposed method significantly improves the performance of existing FSSL algorithms.

Strengths

+ This paper is very well-written. The proposed method and the underlying idea are clearly explained. + The idea of using client-specific adaptive thresholds is very neat and well-motivated by mitigating the confirmation bias. + The paper proposes the sharpness-aware consistency regularization to address the issue of generalizing to wrongly labeled data points when using sharpness-aware minimization. + The numerical experiments show that the proposed method outperforms the existing methods by a significant margin consistently. + The paper also numerically assesses the contribution of each component and studies the impact of incorrect pseudo-labels, providing further insight into the success of the proposed method.

Weaknesses

+ The proposed method is based on heuristic reasoning and lacks theoretical justification. It would be nice if the authors could provide some theoretical justification for some components of the proposed method. + The numerical studies are not that extensive; only two public datasets are used for benchmarking. It would be more convincing if the authors could provide more thorough comparison studies over different datasets across different settings.

Questions

+ How does the proposed method perform in Figure 1? Does it perform comparably to the centralized SSL method?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The paper only briefly mentions the lack of theoretical formulation as one limitation of the work.

Reviewer LSrU4/10 · confidence 4/52024-07-13

Summary

The paper proposes a new method for federated semi-supervised learning tasks where only the server has a small amount of labeled data. The paper combines 3 different methods to tackle the problem and claims to reduce the confirmation bias issue with the method proposed.

Strengths

The paper is well-written and gives established motivations.

Weaknesses

- The method seems to be just a combination of FreeMatch and FlatMatch under the FL case. - It is better to give more motivation to the LSSA method - Only on 2 datasets while most of the existing SSL /FSSL methods use three datasets. Based on a lot of existing semi-supervised learning literature, it is quite common to at least also include the CIFAR100 dataset. It will be better to also incorporate other type of dataset (not only image) to show the robustness of the method - Authors claim that the new method can effectively reduce the confirmation bias, so it would be better to have experiments specially design to show this part. It would be better to show that the method can successfully label the hard data compared to the baseline.

Questions

- Section 5.3: I find it hard to understand why to compare with ‘correctly pseudo-labeled data’. First, under the semi-supervised case, we won’t know the label, hence there is no point to compare test only on the correct pseudo-label data. This can only induce bias. Second, if we compare with CAT and CAT+SACR(all data), CAT only performs consistently better. This negate the idea of SACR, isn’t it?

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Same as above

Authorsrebuttal2024-08-12

Thank you for your response

Thank you for your response to our rebuttal. We sincerely appreciate your effort and time in going through our rebuttals and raising your concern. We want to be more explicit in addressing your remaining concerns as follows: ### Regarding the confirmation bias As highlighted in previous work on Semi-Supervised Learning and others that mentioned confirmation bias [2, 3, 4, 5, 6], one approach to reducing confirmation bias is to filter out noisy pseudo-labels and retain only the high-quality ones. For instance, Fixmatch [2] claimed that it effectively reduced confirmation bias by generating higher-quality pseudo-labels. Similarly, Mean Teacher [3] suggested that improving target quality can help mitigate confirmation bias. Additionally, the Softmatch [4] paper highlighted that incorrect pseudo-labels often contribute to the occurrence of confirmation bias. Based on these previous works, we believe that demonstrating a higher correct label ratio effectively shows how well our method filters out incorrect pseudo-labeled samples. Consequently, since $(FL)^2$ has a significantly lower rate of incorrect pseudo-labels, the influence of these on the unsupervised loss is much less pronounced compared to SemiFL. This suggests that the training signals propagated by these pseudo-labels are more reliable in $(FL)^2$ due to lower contribution of noisy pseudo-labels - our unsupervised loss is dominated mainly by correct pseudo-labeled samples compared to SemiFL. Nevertheless, to further evaluate this, we **designed and conducted an experiment to compare the level of overfitting of $(FL)^2$ and SemiFL to wrongly pseudo-labeled data in a scenario with extremely scarce labeled data.** We measured this by inspecting Rsoft, RLoss, and RAccuracy. Rsoft is the average confidence of the wrongly pseudo-labeled data, similar metric measured in [1]. RLoss and RAccuracy means the training loss and accuracy on the wrongly pseudo-labeled data, respectively. High Rsoft and RAccuracy, and low Rloss indicate that the model is overfitting to the incorrectly pseudo-labeled data. The experiment used the CIFAR-10 dataset under a balanced IID setting with only 10 labeled samples at the server, keeping all hyperparameters the same as in the main experiments, except for the number of communication rounds, which we set to 500. |Method|Label Ratio (%) |Rsoft (After softmax) |RLoss|RAccuracy (%) | |:---:|:---:|:---:|:---:|:---:| |SemiFL|100|99.85|0.0015|100| |$(FL)^2$|86.86|84.53|0.4008|87.09| The results clearly show that SemiFL overfits significantly to the wrongly pseudo-labeled data compared to $(FL)^2$. This demonstrates $(FL)^2$’s robustness against incorrect pseudo-labels, particularly in scenarios with extremely scarce labeled data at the server, outperforming SemiFL. Thus, we conclude that $(FL)^2$ is more effective in reducing confirmation bias in low-label scenarios. Thank you again for your insightful suggestion to explore overfitting with incorrectly pseudo-labeled data. We will ensure that the results of this experiment are included in the camera-ready version. - [1] Arazo, Eric, et al. "Pseudo-labeling and confirmation bias in deep semi-supervised learning." 2020 International joint conference on neural networks (IJCNN). IEEE, 2020. - [2] Sohn, Kihyuk, et al. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." Advances in neural information processing systems 33 (2020): 596-608. - [3] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017). - [4] Chen, Hao, et al. "SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning." The Eleventh International Conference on Learning Representations. - [5] Nassar, Islam, et al. "All labels are not created equal: Enhancing semi-supervision via label grouping and co-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. - [6] Zhang, Bowen, et al. "Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling." Advances in Neural Information Processing Systems 34 (2021): 18408-18419.

Authorsrebuttal2024-08-12

Thank you for your response

Thank you for your comment regarding motivation and innovation. We would like to explain further about $(FL)^2$'s motivation and novelty. ### Novelty of $(FL)^2$ We would like to further explain the novelty of $(FL)^2$ in detail. Use of client-specific threshold (CAT) unlike FreeMatch - $(FL)^2$ provides a precise measure of the client's learning status rather than relying on an EMA-based estimation like FreeMatch, since $(FL)^2$ directly calculates learning status using all unlabeled data. - $(FL)^2$ is computationally efficient. Instead of continuously updating the learning status every batch like FreeMatch, $(FL)^2$ only requires one interaction of learning status calculation over unlabeled samples per communication round. Additionally, since we utilize the global pseudo-labeling scheme proposed in SemiFL, we must go through all the unlabeled data anyway to pseudo-label them. Additional computation for CAT fits naturally into the existing workflow. Utilization of SAM under FSSL - We developed a novel SAM objective specific to FSSL setting. We first applied the Sharpness-Aware Minimization (SAM) objective in a FSSL setting and discovered that applying it to all pseudo-labeled data degrades performance even though SAM shows strong generalization ability across different tasks. Instead, we introduced a novel approach that selectively applies the SAM objective to a small subset of high-confidence pseudo-labeled data, while using an adaptive threshold to incorporate a larger portion of unlabeled data. Our ablation study (Table 2 of the paper), which emphasizes the importance of each component, demonstrates that this combination of techniques effectively reduces confirmation bias and achieves high performance. ### Motivation of LSAA We would like to further clarify the motivation of LSAA. - In a centralized SSL, methods like FlexMatch and FreeMatch introduced the use of different thresholds for each class within a dataset and showed their effectiveness. The rationale is that different classes pose varying levels of learning difficulty, so lower thresholds are assigned to more challenging classes—those with a lower learning status—to facilitate more effective learning from these harder classes. - In the context of FSSL, the learning difficulty can vary across clients. This variation arises for two main reasons. First, since the server has access to only a small labeled dataset, clients whose data closely resembles the server’s data will face lower learning difficulty, while those with more distinct data will encounter higher difficulty. Second, due to the non-iid distribution of data across clients, the learning difficulty naturally differs among them. - We propose LSAA to take account of different learning difficulties of clients, where LSAA - assigns higher aggregation weights to clients with higher learning difficulty, enabling the global model to learn more effectively from these clients. In contrast, previous FSSL approaches did not account for these variations in learning difficulty and instead relied on fixed aggregation weights.

Authorsrebuttal2024-08-13

Thank you again for your critical and constructive feedbacks to our papers and rebuttals. If there is still anything that is not clear about our justification regarding the confirmation bias and the novelty/motivation of our work, we would be happy to provide more clarification. If you have any further concerns or questions, please do not hesitate to let us know as it will be extremely important for our revised manuscript. Thank you very much!

Reviewer YhMN2024-08-08

Thanks for the efforts and additional experiments. My concerns have been addressed, I would like to raise my score to borderline accept

Reviewer LSrU2024-08-09

Thanks for more experiments and results. I still have some concerns regarding innovation and motivations. Also, I'm not entirely convinced of the authors' answer regarding the confirmation bias. The authors mentioned about the pseudo label accuracy and correct label ratio, which seems to be the same thing (the accuracy of pseudo labeling). However, the definition of the confirmation bias proposed by the reference is about whether or not the model is overfitting to the wrong pseudo-labels. Therefore, how the proposed methods can mitigate the confirmation bias still lacks persuasiveness. Therefore, I would maintain my score for now.

Reviewer dbte2024-08-12

Response to Author Rebuttal

Thank you very much for your detailed responses. Including theoretical justification would strengthen the paper. I would maintain my score.

Authorsrebuttal2024-08-13

Thank you for your response!

Thank you again for the valuable suggestions and comments. If you have any remaining concerns, please let us know!

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC