InstanT: Semi-supervised Learning with Instance-dependent Thresholds

Summary

This paper introduces a new thresholding methodology for Semi-Suervised Learning. This paper proposes InstanT, which uses an instance-dependent threshold for each unlabeled data (Fig1). This algorithm shows the improvement on multiple semi-supervised benchmark datasets.

Strengths

Pros. - This paper is well-written and easy to follow. - This paper is well-motivated (Fig 1). - This paper proposes InstanT, and it is very well derived. - This paper shows extensive experiments (Especially it seems to train all other semi-supervised learning baselines using ViT models) and shows the effectiveness of this method.

Weaknesses

Cons. - (Minor) This paper uses a pre-trained ViT model. In pre-training, I think the classes in Cifar10/Cifar100 could be the subset of the pre-training dataset. If These classes are already trained, they could be already well clustered and easy to train with a very small labeled dataset. (It also shows the performance improvement in the original training setting in the Appendix) - (Minor) It would be great if it reports the supervised learning results (full labeled dataset) - (Major) This paper evaluates their algorithm in very small datasets (Cifar10/Cifar100/STL-10). Therefore, I don't know how it robustly works in large datasets such as ImageNet. Nowadays, most of the papers evaluate their algorithms on ImageNet. (In this case, it seems to train the model from scratch. (not using pre-trained ViT model)

Questions

Please see the weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Please see the weaknesses.

Summary

This paper focuses on semi-supervised learning (SSL) and proposes the study of instance-dependent thresholds to make incorrect pseudo-labels have higher thresholds.

Strengths

This paper studies an important problem in SSL, and tries to propose a dynamic and adaptive threshold to guarantee the pseudo-label quality for instances. Also, it gives a theoretical analysis on estimating the threshold. The experiments are sufficient and the results surpasses SOTA. In the results, it is obvious to see that the algorithm has a considerable improvement in Top-1 accuracy. The ablation study shows that every part of this algorithm is useful.

Weaknesses

Theorem 2 is proved with Bayes rule with Assumption 2. But if the dynamic value k_t > 0, Theorem 2 does not hold. It is incorrect P(X>=a+b)>=P(X>=a) for any b>0. I think there may be some mistakes in Assumption 2 and Theorem 2. The theorems and analyses based on Assumption 2 may be wrong. In section 4.1 the authors do not give any theoretical reasons for the definition of k_t and \tau(x_u). Whether k_t satisfies Assumption 2 or not? In Equation 8, the authors use an approximation \hat{T}(x) to estimate T(x). But the analysis is based on the real value of T(x). If the label noise is large or the number of samples is not enough, the error between the real value of T(x) and the approximation \hat{T}(x) cannot be ignored. Some analysis should be provided here.

Questions

As mentioned in the part of weaknesses, the authors may make a mistake in Theorem 2, which makes that the paper is not convincing. The theoretical analysis part of the paper should be considered more seriously. The dynamic threshold is determined by the transition matrix T(x). But in practice it is impossible to estimate it without any bias. The bias may influence on the final result. Is it possible to provide an error bound in the analysis?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

No

Summary

This paper presents a new approach for selecting confident examples called InstanT, which uses instance-dependent thresholds for assigning pseudo-labels to unlabeled data. Unlike existing methods that apply the same threshold to all samples, InstanT considers the instance-level ambiguity and error rates of pseudo-labels, assigning higher thresholds to instances more likely to have incorrect pseudo-labels. The paper demonstrates that this approach provides a probabilistic guarantee for the correctness of the assigned pseudo-labels. This innovative method may offer a new perspective on SSL.

Strengths

1.The paper finds a significant and challenging problem in semi-supervised learning (SSL) that has not been adequately tackled by existing methods. Traditional SSL methods typically use a single loss threshold to select confident examples, implicitly assuming that examples with the same loss have the same likelihood of pseudo-label correctness. However, this assumption does not always hold, as there can be hard but confident examples that have larger loss values but correct pseudo-labels. 2.The innovation of this paper is noteworthy. This paper innovatively proposes to estimate the probability of examples being incorrect and applies instance-dependent thresholds based on these estimates. This approach is more nuanced and potentially more effective as it takes into account the individual characteristics of each example. It is the first to propose the estimation of instance-dependent thresholds in SSL. This novel approach represents could open up new avenues for research in the field. 3.The authors also provide sufficient theoretical analysis about their proposed method. They present a theorem that shows that for samples that satisfy their instance-dependent threshold function, the likelihood of the pseudo-labels being correct is lower-bounded. This provides a solid foundation for the proposed method and helps to convince readers of its assumption and correctness.

Weaknesses

1.Certain aspects of the paper could benefit from further explanation and clarification. Specifically, the relationship between the Quality-Quantity Trade-off and the effective dynamic value is not clearly articulated in the main body of the paper. These are key components of the proposed method, and their interaction could significantly impact the performance of the method. 2.The paper does not sufficiently discuss the limitations of the proposed method. Every method has its limitations and potential drawbacks, and a thorough discussion of these is crucial for a balanced and comprehensive presentation of the work. 3.The empirical improvement of the proposed method is marginal when the amount of labeled data increases. This suggests that the method's performance may not scale well with larger labeled datasets. 4.The experimental evaluation of the method is based on only three datasets. This limited number of datasets may not provide a comprehensive evaluation of the method's performance.

Questions

1.What might be the potential drawbacks or limitations of this method? How could these impact its applicability or performance in certain scenarios or with certain types of data? 2.The experimental evaluation of the method is based on only three datasets. Could the authors explain their choice of datasets and how representative these datasets are of the types of data the method would encounter in real-world applications? Would the performance of the method vary significantly if tested on other datasets? 3.The authors may need to clarify the relationship between the Quality-Quantity Trade-off and the effective dynamic value. How do these two factors interact within the context of their proposed method? 4.The paper discusses the balance between the quality and quantity of pseudo-labels. Could the authors provide some intuition of how this balance can be achieved in practice? 5.This paper is highly related to the problem of learning with instance-dependent label noise. To clearly demonstrate the effectiveness, can authors provide a comparison with methods for handling instance-dependent label noise?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The main paper seems to lack a comprehensive discussion on the limitations of the proposed method. It would be beneficial for the authors to conduct a thorough review of their method to identify any potential limitations. I suggest including a separate section in the main paper dedicated to discussing these limitations, which would provide a more balanced and complete perspective of the proposed method.

Summary

This paper proposes a semi-supervised method with instance-dependent thresholds (InstanT), which can assign different thresholds to individual unlabeled data based on the instance-dependent label noise level and prediction confidence. Also, this paper provides a theoretical analysis of the proposed InstanT. Extensive experiments have shown the effectiveness of the proposed method.

Strengths

1. This paper proposes an Instance-dependent Thresholding strategy for semi-supervised learning. Also, the proposed method can vouch for the reliability of pseudo-labels it assigns with a theoretical guarantee. 2. This paper is easy to read and well organized. 3. The experimental results and the proof of Theorems in this paper seem to be solid.

Weaknesses

1. This paper is devoted to addressing the SSL problem with dynamic thresholding. However, there is a lack of some SOTA methods to be compared with the proposed method. For example, [1] Guo et al., Class-imbalanced semi-supervised learning with adaptive thresholding, ICML, 2022. [2] Yang et al., Class-aware contrastive semi-supervised learning, CVPR, 2022. [3] Wang et al., Freematch: Self-adaptive thresholding for semi-supervised learning, ICLR, 2023. [4] Chen et al., Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning, ICLR, 2023. 2. Most previous SSL methods have reported their performance on both datasets: SVHN and ImageNet. It is therefore suggested that the authors supplement some experiments to evaluate the generalization performance of the proposed method. 3. It is suggested to omit some symbols whose meanings have been explained in the previous context, so as to increase the brevity of this paper. 4. The proposed method is motivated by the learning with noisy labels. Therefore, it is necessary to review some related methods to give readers a whole picture.

Questions

Please refer to the Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations of this paper should be discussed.

Summary

- Assumption for instance-dependent threshold setting and theoretical proof based on it - Transition matrix modeling (estimator design) to reduce label error through instance-dependent threshold function - This paper shows good performance on various datasets (CIFAR10, 100, STL-10)

Strengths

1) A new approach in SSL called instance-dependent threshold setting 2) Solid theoretical proof 3) Excellent performance, especially in environments with little labeled data

Weaknesses

1) Transition matrix modeling for threshold calculation for each instance is required (complexity is expected to increase). 2) There is no consideration for class imbalance, which is mainly dealt with in SSL. If there is a class imbalance problem, it may not be a good idea to set different thresholds for each instance. 3) Comparative papers (FlexMatch, Dash, AdaMatch, etc.) are papers published before 2022. Comparisons with recent papers have not been made. In addition, AdaMatch, which is mainly compared, is a paper dealing with domain adaptation issues.

Questions

1) The running time in Table 2 was 1s lower than Fix and AdaMatch. Do you have the result in terms of the amount of calculations and the number of parameters? 2) If the reason for applying Distribution Alignment in InstanT-II in Table 3 is to eliminate class imbance, what is the reason why performance improves when DA is applied in ablation study? (The experimental environment in the paper appears to be a situation with low imbalance using the same number of labeled data for each class). 3) Usually SSL papers show error rates of CIFAR10/100. How does the error rate come out? 4) What about the results on larger sized images (e.g. ImageNet)? 5) Papers on instance dependent pseudo-labeling already exist. Is there any difference from that? For example, the paper SimMatch: Semi-supervised Learning with Similarity Matching (2022 CVPR) also uses instance dependent similarity. This paper theoretically proves the validity of using instance-level threshold, but is there any excellence or novelty compared to the similarity method used in the above paper? 6) The results of the comparative papers presented in the proposed paper and the results of the papers below are different. Is it an accurate comparison? Is there a reason why the second paper below is not included in the result table? - SimMatch: Semi-supervised Learning with Similarity Matching (2022 CVPR) - SOFTMATCH: ADDRESSING THE QUANTITY-QUALITY TRADE-OFF IN SEMI-SUPERVISED LEARNING (2023 ICLR) 7) Please also respond to the weaknesses pointed out.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Thank you for responding to our rebuttal!

Dear reviewer zzrr, we sincerely thank you for reading through all of our rebuttals, we are strongly encouraged by your acknowledgments of our contributions! We're committed to continuously refining our paper, thus we welcome any new insights and suggestions that you may still have during the rolling discussion phase.