Thank you for your response
Thank you for your response to our rebuttal. We sincerely appreciate your effort and time in going through our rebuttals and raising your concern. We want to be more explicit in addressing your remaining concerns as follows:
### Regarding the confirmation bias
As highlighted in previous work on Semi-Supervised Learning and others that mentioned confirmation bias [2, 3, 4, 5, 6], one approach to reducing confirmation bias is to filter out noisy pseudo-labels and retain only the high-quality ones. For instance, Fixmatch [2] claimed that it effectively reduced confirmation bias by generating higher-quality pseudo-labels. Similarly, Mean Teacher [3] suggested that improving target quality can help mitigate confirmation bias. Additionally, the Softmatch [4] paper highlighted that incorrect pseudo-labels often contribute to the occurrence of confirmation bias. Based on these previous works, we believe that demonstrating a higher correct label ratio effectively shows how well our method filters out incorrect pseudo-labeled samples. Consequently, since $(FL)^2$ has a significantly lower rate of incorrect pseudo-labels, the influence of these on the unsupervised loss is much less pronounced compared to SemiFL. This suggests that the training signals propagated by these pseudo-labels are more reliable in $(FL)^2$ due to lower contribution of noisy pseudo-labels - our unsupervised loss is dominated mainly by correct pseudo-labeled samples compared to SemiFL.
Nevertheless, to further evaluate this, we **designed and conducted an experiment to compare the level of overfitting of $(FL)^2$ and SemiFL to wrongly pseudo-labeled data in a scenario with extremely scarce labeled data.** We measured this by inspecting Rsoft, RLoss, and RAccuracy. Rsoft is the average confidence of the wrongly pseudo-labeled data, similar metric measured in [1]. RLoss and RAccuracy means the training loss and accuracy on the wrongly pseudo-labeled data, respectively. High Rsoft and RAccuracy, and low Rloss indicate that the model is overfitting to the incorrectly pseudo-labeled data. The experiment used the CIFAR-10 dataset under a balanced IID setting with only 10 labeled samples at the server, keeping all hyperparameters the same as in the main experiments, except for the number of communication rounds, which we set to 500.
|Method|Label Ratio (%) |Rsoft (After softmax) |RLoss|RAccuracy (%) |
|:---:|:---:|:---:|:---:|:---:|
|SemiFL|100|99.85|0.0015|100|
|$(FL)^2$|86.86|84.53|0.4008|87.09|
The results clearly show that SemiFL overfits significantly to the wrongly pseudo-labeled data compared to $(FL)^2$. This demonstrates $(FL)^2$’s robustness against incorrect pseudo-labels, particularly in scenarios with extremely scarce labeled data at the server, outperforming SemiFL. Thus, we conclude that $(FL)^2$ is more effective in reducing confirmation bias in low-label scenarios.
Thank you again for your insightful suggestion to explore overfitting with incorrectly pseudo-labeled data. We will ensure that the results of this experiment are included in the camera-ready version.
- [1] Arazo, Eric, et al. "Pseudo-labeling and confirmation bias in deep semi-supervised learning." 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.
- [2] Sohn, Kihyuk, et al. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." Advances in neural information processing systems 33 (2020): 596-608.
- [3] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017).
- [4] Chen, Hao, et al. "SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning." The Eleventh International Conference on Learning Representations.
- [5] Nassar, Islam, et al. "All labels are not created equal: Enhancing semi-supervision via label grouping and co-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
- [6] Zhang, Bowen, et al. "Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling." Advances in Neural Information Processing Systems 34 (2021): 18408-18419.