OwMatch: Conditional Self-Labeling with Consistency for Open-World Semi-Supervised Learning

Summary

The paper introduces OwMatch, a novel method for open-world semi-supervised learning (SSL). This approach incorporates conditional self-labeling and open-world hierarchical thresholding. Additionally, the paper provides theoretical analyses that demonstrate the unbiasedness and reliability of the label assignment estimator. Rigorous experimentation confirms that the proposed method achieves a significant improvement in performance.

Strengths

1. The paper provides adequate theoretical analyses for the proposed novel method. 2. The paper is well-presented with clear organization throughout. 3. The experiments are adequate and achieve better performance compared to existing methods.

Weaknesses

1. The paper exhibits limited innovation. The two strategies proposed—Conditional Self-labeling and Open-world Hierarchical Thresholding—are commonly employed in existing OwSSL (Open-world Semi-Supervised Learning) and GCD (Generalized Category Discovery) methodologies. Specifically, the Conditional Self-labeling strategy mirrors the optimal transportation label assignment method detailed in references [1-4], and the Hierarchical Thresholding closely resembles the Momentum Prior Update described in references [3-4]. 2. The method assumes that the number of unknown classes is pre-determined and fails to address the identification of unknown classes. The estimation of the number of unknown classes is a critical issue in both OwSSL and GCD fields. 3. The comparison with other algorithms is insufficient. Recent publications such as OpenLDN, TRSSL, and OpenCon from 2022 are included; however, more recent methods, particularly those utilizing pretrained ViT (Vision Transformer) architectures, should also be compared. Additionally, comparisons with methods addressing imbalance GCD are necessary. [1] Towards Realistic Semi-Supervised Learning [2] OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning [3] ImbaGCD: Imbalanced Generalized Category Discovery [4] Bootstrap Your Own Prior: Towards Distribution-Agnostic Novel Class Discovery

Questions

See weakness

Rating

5

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

No

Summary

This paper proposes OxMatch, a semi-supervised learning (SSL) algorithm for an open-world setup where unlabelled data might come from outside of the labeled class distribution. The authors combine consistency regularization and self-labeling techniques to address the main challenges in open-world SSL (OwSSL). In particular, they use a conditional self-labeling approach to improve the label assignment stage (i.e. computation of pseudo-labels) and a hierarchical thresholding scheme to weight the contribution of different samples depending on the state of the model (i.e. the rate at which each class is being learned). Additionally, the authors provide a theoretical analysis of the conditional self-labeling estimator.

Strengths

- The paper is theoretically sound and the authors properly justify the proposed elements. - The experimental section is complete and provides insights into the contribution of the different elements of the model.

Weaknesses

- The presentation and phrasing of some paper parts should be addressed. The explanation of the method is slightly confusing and difficult to follow at some parts. - The paper addresses confirmation bias but this is not introduced at any point. Similarly, the text refers to Figure 1 as an example of the mitigation of confirmation bias (line 48) but this is not explained or elaborated. I would suggest addressing this in the main text and writing a more comprehensive caption for Figure 1. - Line 138 refers to Figure 3 for an illustration of issues with self-labeling and confirmation bias. This is not clear. Whether this refers to Figure 1 or Figure 3, it needs some more explanation. - In general, it is a good practice to write self-contained captions for figures. These should explain the elements in the figure and highlight their relationship with the main text. - Captions of tables should also be reviewed to include details of the metrics being displayed in the table. for better readability I would suggest highlighting the relevant results.

Questions

None

Rating

5

Confidence

3

Soundness

4

Presentation

2

Contribution

2

Limitations

The authors already mention the main limitations fo the proposed approach: the method assumes that the class distribution is known a priori.

Summary

This paper proposes OwMatch, a new approach for open-world semi-supervised learning (OwSSL). The key contributions are: (1) A conditional self-labeling method that incorporates labeled data into the clustering process to reduce confirmation bias and misalignment. (2) A hierarchical thresholding strategy to balance learning difficulties across different classes. (3) Theoretical analysis of the unbiasedness and reliability of the conditional self-labeling estimator. (4) Extensive experiments demonstrating state-of-the-art performance on multiple benchmarks. The method combines ideas from self-supervised and semi-supervised learning, using consistency regularization and self-labeling tailored for the open-world setting where unlabeled data may contain novel classes.

Strengths

1. The paper introduces a novel combination of conditional self-labeling and open-world hierarchical thresholding to address the challenges in OwSSL. Hierarchical thresholding is proposed to address the issue of different learning pace of seen and novel classes and seems to work well. 2. The paper is well-written and clearly structured, with detailed explanations of the methodology, theoretical analysis, and experimental results. The figures and tables effectively illustrate the key points and support the claims made by the authors. 3. The theoretical analysis provides a solid foundation for the proposed method, demonstrating its unbiasedness and reliability in label assignment. The empirical results show significant performance improvements across various datasets, indicating the method's robustness and effectiveness.

Weaknesses

1. The Open-world Semi-supervised Learning setting is very similar to or the same as Generalized Category Discovery, both assuming novel classes exist and part of the data has labels in seen classes. The paper lacks discussion and comparison with closely-related work in generalized category discovery, such as GCD[1], SimGCD[2], and BaCon[3]. 2. A critical ablation study is lacking/needed to verify the effectiveness of the proposed open-world hierarchical thresholding (OwAT). OwAT is an incremental adaptation to existing self-adaptive thresholding in FreeMatch[4] by considering the different group learning speeds in novel and seen classes. In the existing ablation study, it is shown that OwAT boosts model performance, but I think the previous self-adaptive thresholding contributes most of the improvement and it is unclear how much the proposed hierarchical design really helps. Besides, since the self-adaptive thresholding adjusts thresholds based on the learning status of different classes, I think it can automatically consider the different learning paces of novel and seen classes and assign different thresholds, which also questions the effectiveness of the proposed hierarchical design. 3. The proposed conditional self-labeling aims to incorporate labeled data into the online clustering process. But its goal and function seem to overlap with the supervised loss, which is used in this work. How is the performance (1) without the supervised loss, (2) without the conditional part of self-labeling, (3) without both? 4. Currently, the code is not available to verify the performance and ablation study. [1] Vaze, Sagar, Kai Han, Andrea Vedaldi, and Andrew Zisserman. "Generalized category discovery." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7492-7501. 2022. [2] Wen, Xin, Bingchen Zhao, and Xiaojuan Qi. "Parametric classification for generalized category discovery: A baseline study." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16590-16600. 2023. [3] Bai, Jianhong, Zuozhu Liu, Hualiang Wang, Ruizhe Chen, Lianrui Mu, Xiaomeng Li, Joey Tianyi Zhou, Yang Feng, Jian Wu, and Haoji Hu. "Towards distribution-agnostic generalized category discovery." Advances in Neural Information Processing Systems 36 (2023): 58625-58647. [4] Wang, Yidong, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang et al. "Freematch: Self-adaptive thresholding for semi-supervised learning." arXiv preprint arXiv:2205.07246 (2022).

Questions

Please refer to questions in the weakness section.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Limitations and potential impact are well discussed in the paper.

Summary

The paper has a novel idea, clear problems, and rich experimental results. It is a good article. However, there are some problems that need to be further optimized.

Strengths

The idea is clear and the problem is prominent. The experimental results prove the superior performance of the proposed method in open world scenarios.

Weaknesses

Some results need to be supplemented and explained Requiring some proof of principle, theory or experiment.

Questions

1. References should not be used as sentence components, such as reference [31]. 2. I would like to know how the proposed method reduces the bias, not just the final result. Can the author provide some principle experimental analysis? 3. Maybe I overlooked it. I did not find any experimental results on Tiny ImageNet that are consistent with Table 1. 4. From the confusion matrix, at 100 iterations, for unseen data, only the sixth category can be predicted correctly, while the other categories are all wrongly predicted. This is different from what the author described. 5. Why does the prediction effect of seen classes become worse after adding the ConSL module? 6. The authors should try to align the experiments. The experimental data for each experiment is incomplete and is based on three of the datasets.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

As above