Summary
This paper introduces an active learning framework that is based on a combination of deep ensemble, confidence margin, and self-training techniques:
1. **Sample Selection Based on Deep Ensemble (checkpoint ensemble):** fine tune N models using SGD with different random seeds. An ensemble of these models is used to compute the average confidence for each sample. Those unlabelled test samples with the lowest confidence margins are selected for human labelling.
2. **Active Learning:** All the N models are fine-tuned on these selected test samples and the original training dataset using Cross Entropy loss to adapt to test distribution and prevent forgetting previously learned information.
3. **Self-Training:** After active learning, unlabeled samples from the test set with confidence exceeding a certain threshold (e.g., 0.95) are selected for self-training. The average label provided by the ensemble of models is assigned as the sample's pseudo label. Then the authors train the N models using these samples with KL divergence loss.
4. **Computational Efficiency on Large Test Sets:** When dealing with an extensive unlabeled test set, to reduce computational costs, the average confidence and pseudo labels for each test sample sample are updated only after multiple epochs.
The resultant N models trained through this framework exhibit high ensemble accuracy. Additionally, their average confidence proves effective when applied to selective classification tasks.
Strengths
**Originality/Significance**: The author presents an effective framework for active learning under distribution shift, which also mitigates overconfidence.
**Quality**: This paper utilizes a wide range of experimental datasets, covering various types such as image and text. From this perspective, the empirical evidence is quite comprehensive, and the ablation study is also supportive.
**Clarity**: Overall, the paper is easy-to-read. Each element, such as the introduction of a particular loss function, is accompanied by intuitive justifications.
Weaknesses
**Originality/Significance**: 1) Method: there has been some literature such as [1] utilizing model ensemble's uncertainty scores, e.g., average confidence or its variants like confidence margin to select samples for active learning. On the other hand, laeveraging high-confident pseudo-labels for self-training to enhance its accuracy has also been mentioned in many literatures, such as [2]. Combining these methods indeed can improve performance, but in terms of the bringing new method/ideas to these fields, I would find it rather limited. 2) Framework/Task: Regarding the new framework (i.e. task) proposed by the authors that combines selective classification and active learning seems somewhat unconvincing: one pertains to the training framework, while the other is related to the inference phase. The author's objective is, in my view, to enhance model accuracy during training while also considering its confidence calibration aspect to mitigate overconfidence, the latter of which the ensemble will naturally satisfy. Besides, I think this goal sometimes is already built in the active learning's framework since they usually require an accurate confidence score to do sample selection.
[1] Beluch, William H., Tim Genewein, Andreas Nürnberger, and Jan M. Köhler. "The power of ensembles for active learning in image classification." In CVPR 2018.
[2] Lee, Dong-Hyun. "Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks." In Workshop on challenges in representation learning, ICML, 2013.
**Clarity**: The introduction part and Sec 3.1-3.3 can be somewhat confusing, making it challenging for readers to quickly grasp how selective classification and active learning are integrated. For example, in Figure 1, the low-confidence samples chosen by selective classification are also put into the human labelling. Readers will wonder how these labeled samples at the inference stage benefit the model and will guess whether it is a dynamic system. Full understanding is achieved only upon seeing the specific algorithm implementation.
Questions
1. For Eq. 10, how is the ground truth label in the KL divergence obtained? Is it directly using the average confidence from the ensemble as its label, or is it based on majority voting?
2. The last sentence in Sec 4 seems to indicate that this method cannot generalize to other test samples that haven't appeared in Ux. Is my interpretation accurate in believing that the model can directly utilize the learned ensemble models to make new predictions on unseen test data points?
3. How is checkpoint ensemble implemented? How does it fundamentally differ from deep ensemble? What makes checkpoint ensemble unique? Is my understanding correct that the approach of checkpoint ensemble involves fine-tuning N different models using SGD and varying random seeds, and then using T-round active learning to train every ensemble model? What does "checkpoint" indicate?
Rating
5: marginally below the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.