Doubly Robust Self-Training

Summary

This paper presents a very simple approach to semi-supervised learning that utilizes both labeled and unlabeled datasets. When there is a large amount of unlabeled data available, following the same distribution as the labeled dataset, the most effective method to leverage this unlabeled data for training is self-training, where pseudo labels are generated and used for training. The main limitation of self-training is that the performance can degrade when the pseudo-labels produced by the predictor for the unlabeled set are not accurate. In this paper, the authors propose a very simple method to overcome this limitation by replacing the conventional loss for the labeled dataset and the unlabeled dataset in self-training. This modification allows the model to be trained effectively in both cases, whether the pseudo-labels are correct or not, leading to improved performance in all scenarios. The effectiveness of this approach is experimentally demonstrated on image classification benchmarks and 3D object detection benchmarks.

Strengths

Overall, the method is very simple, intuitive, and quite novel. The paper is well-written and it provides a well-derived explanation using equations for both cases where the pseudo labels are accurate and when they are not. Additionally, the method is thoroughly analyzed from a theoretical perspective.

Weaknesses

Although the theoretical derivation demonstrates the soundness of the method, there is doubt regarding its effectiveness in experiments. 1. Especially, in the experiments (sec. 3.1), the authors use curriculum-based loss in each epoch. With $\alpha_t < 1 $, the proposed method will behave like self-training (exactly when $\alpha_t = \frac{n}{m+n}$) and the behavior/effectiveness of the proposed method could not be well represented in this case. 2. Image classification experiments were conducted using ImageNet-100. However, when the labeled set ratio is 100%, the top-1 accuracy of DaVIT and ResNet50 is remarkably lower at 47.8% and 46.7%, respectively, compared to the reported top-1 accuracy in other papers. Following the paper of DaVit and ResNet, the top-1 accuracy on the more complex task of ImageNet1k, compared to ImageNet-100, using DaVit-tiny and ResNet is 82.8% and 79.26%, respectively. Following [1], the top-1 accuracy of various resnet50-based methods for ImageNet-100 consistently surpasses 70%. It appears that the baseline has not been sufficiently well-trained. 3. There is a lack of comparison with other semi-supervised methods. While this method compares with the basic self-training loss, it is necessary to compare it with various methods that utilize pseudo-labels for self-training [2], [3], [4]. Most semi-supervised image classification methods have performed experiments on benchmark datasets such as ImageNet1k and CIFAR100. These methods achieve a top-1 accuracy of 72% or higher when using a labeled dataset of 10% on ImageNet1k. Therefore, it is necessary to compare the effectiveness of this method on these benchmark datasets. 4. Similarly, in semi-supervised 3D object detection, it is necessary to compare the results with existing baseline methods [5], [6]. 5. Although this paper proposes a method for the distribution mismatch case in Section 2.4, estimating the probability values of each sample, p(x) and q(x), for each distribution is not easy. Practical methods for this issue have not been provided, and experiments on this aspect are also lacking. Practical algorithms and experiments need to be provided in order to address this concern. [1] Zelin Zang, et al. DLME: Deep Local-flatness Manifold Embedding. ECCV 2022 [2] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, 2020 [3] Kihyuk Sohn, David Berthelot, Zizhao Li, Chun-Liang Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In IEEE Conference on Computer Vision and Pattern Recognition, 2020 [4] Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V Le. Meta pseudo labels. In IEEE Conference on Computer Vision and Pattern Recognition, 2021 [5] He Wang, Yezhen Cong, Or Litany, Yue Gao and Leonidas J. Guibas. 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2021 [6] Na Zhao, Tat-Seng Chua and Gim Hee Lee. SESS: Self-Ensembling Semi-Supervised 3D Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2020

Questions

see above.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Future work is discussed.

Summary

The paper proposed a doubly robust loss for self-training. The proposed loss is analysed and shown to have preferable theoretical properties.

Strengths

1. The idea is interesting: a simple change from 1/(m+n) to 1/n (line 51 - 53) lead to a doubly robust loss function for self-training. 2. The writing is clear and easy to follow

Weaknesses

1. While the proposed doubly robust loss for self-training enjoys theoretical advantages, directly minimizing the loss during network training leads to instability. The actual loss used in line 219 is very different especially in the early epochs. 2. Lack of comparison to other stronger semi-supervised learning baselines. For example, the authors discuss MixMatch and FixMatch as related work but did not compare with them in experiments. minor: 1. I don't think the description in line 79-87 is precise. MixMatch/FixMatch does not pre-trained a teacher model on 'labeled' data.

Questions

NA

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

The authors propose a very simple yet effective modification to the original loss for self-training by re-weighting terms of the loss function making. This change effectively balances between using the pseudo-labels when the predictor is strong and learning to not use it when it is unreliable, making it doubly robust. They provide a sound theoretical analysis and empirical evaluation on classification and object detection tasks substantiating their claims.

Strengths

- Strengths - The paper is well written and easy to follow with sufficient background and motivated examples given to present the chain of reasoning well. - For linear predictor, the proposed loss is unbiased with lower variance which is strictly better than self-training - The results on both classification and 3D object detection highlight clear improvement over standard self distillation - The technical novelty in the paper is limited with just a small modification to the overall loss. But the theoretical insights including guarantees for general loss and positive experimental results make it a meaningful contribution. The simplicity of the modification also make it much more likely to be adopted and have higher impact.

Weaknesses

- Weaknesses - The analysis provided is for very simplistic settings of linear predictor or mean-predictions and it’s unclear how much of it translates to realistic settings of over-parameterized deep nets trained on SGD. - For image classification, the baselines used in the paper are meaningful but thorough comparison with other state of the art self-imitation methods like noisy-student etc is lacking. - The evaluation is restricted to lower data regimes as the model shines when the training data is limited, making their evaluation a bit more contrived as compared to real world scenarios. It seems the proposed loss should shine more when unlabelled data scales, so some experimentation on the impact on performance as the unlabelled data scales would also be interesting to see - One important use of self-distillation is using unlabelled data to show domain adaptation to that domain. I would like to see experiments for that as well.

Questions

* In Section 2.1, the intuitive interpretation for when the predictor is bad is not super clear and paper would benefit from some elaboration. If just m goes to infinity, how would the two terms cancel out * In the case of distribution mismatch, what’s the motivation for changing the first term to average of (n) sample instead of (m+n)? Also how are $\pi (x)$ ie importance weights chosen? * Did the authors explore tuning the predictor $\hat{f}$ on the labelled examples before generating the pseudo-examples to train the student? Typos and possible errors - Inconsistent notation : line 155 should be $\hat{\theta}$ instead of $\theta^{*}$ - In line 161, I believe the upper bound should be $6/n(var[Y] ..) $ instead of $4/n$. Please verify! - in equation in line 171, it should be $\theta_{SL}$ instead of $\theta_{DR}$

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed some limitations. Other suggestions are listed above.

Summary

This paper proposes a pseudo-labeling approach that balances out the supervised signal between the labeled and incorrect pseudo-labeled datapoints during the training process. The aim is to only account for the pseudo-labels when they are correctly labeled, which may happen when the covariate distribution of the unlabeled dataset and the labeled dataset matches. They show some analysis of the proposed loss and results in ImageNet100 (a subset with 100 random classes from ImageNet-1k) and mini-ImageNet100 and nuScenes dataset.

Strengths

The proposed method is clearly written, and the paper can easily be understood. The paper with all necessary details and deep explanations for experimental results. Comprehensive algorithmic analysis and clear motivation. The proposed method effectively improves over vanilla pseudo-labeling [1] [1] D.-H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop : Challenges in Representation Learning (WREPL), 07 2013.

Weaknesses

Novelty and missing prior work: Pseudo-labeling is the defacto method for entropy regularization techniques in semi-supervised learning problems. This is a paper that tackles the semi-supervised learning problem. Recent work has explored different ways to mitigate the error propagation from the teacher model and confirmation bias present in pseudo-labeling approaches. For example, [2,3,4] investigate the thresholding effect via fixed and curriculum based approaches, with flexible thresholds that are dynamically adjusted for each class according to the current learning status. With such prior exploration and no comparison with any of that work, it's difficult to assess the importance and impact of this work, which seems limited and incomplete. The paper in its current state also fails to provide technical details to validate fair comparisons in the main text. Furthermore, no ablations for any of the technical selections are conducted. The alternative loss proposed for distribution mismatch (Section 2.4) is only shown in the method but not in the empirical section. [2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In IJCNN, pages 1–8, 2020. [3] Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6912–6920, 2021. [4] Zhang B, Wang Y, Hou W, Wu H, Wang J, Okumura M, Shinozaki T. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems. 2021 Dec 6;34:18408-19.

Questions

Prior literature [2,3,4,5] has shown that predefined threshold values impact the overall performance in pseudo-labeling. Using 0.3 as the threshold seems to be too permissive, allowing too many noisy pseudo-labels. How sensitive is this threshold in your setup? [5] Oliver A, Odena A, Raffel CA, Cubuk ED, Goodfellow I. Realistic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems. 2018;31.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

No limitation section is provided.

Finished Additional Experiment 4

Dear reviewers, We have finished the additional experiments 4 on the ablation studies on the detection thresholds. As is shown in the table, setting $\tau = 0.5$ does not improve compared with the case of $\tau=0.3$ in our default choice. And our doubly robust estimator still remains the best estimator under the most appropriate choice of $\tau$. We will supply more experiments in the final version. Thank you again for the great comments! Please let us know if you have any further suggestions. | Labeled Data Fraction | $\tau = 0.3$ | | | | $\tau = 0.5$ | | | | |:----------------------|:-------------:|:-:|:-:|:-:|:-------------:|:-:|:-:|:-:| | | Labeled+Pseudo | | Doubly Robust Loss | | Labeled+Pseudo | | Doubly Robust Loss | | | mAP↑ | NDS↑ | mAP↑ | NDS↑ | mAP↑ | NDS↑ | mAP↑ | NDS↑ | | 1/24 | 7.56 | 18.01 | **8.18** | **18.33** | 7.15 | 15.82 | 4.37 | 13.17 | | 1/16 | 11.15 | 20.55 | **12.30** | **22.10** | 11.05 | 21.22 | 8.09 | 19.70 |