Algorithm Selection for Deep Active Learning with Imbalanced Datasets

Summary

This paper proposes a novel algorithm called TAILOR that adaptively chooses the active learning algorithm from the candidate set of multiple algorithms. Novel reward functions are proposed that encourage choosing the algorithm such that both class diversity, as well as informativeness, are maintained while selecting unlabeled samples for the annotation. A strong theoretical justification is provided to demonstrate the effectiveness of the proposed TAILOR algorithm using regret analysis. The extensive evaluation conducted on various multiclass (imbalanced) and multilabel datasets demonstrates the ability of the proposed technique to produce comparable/better accuracy compared to the data-specific best candidate active learning strategy.

Strengths

1. This paper has unified the strength of existing active learning algorithms into the single TAILOR framework. This avoids the burden of finding an effective active learning algorithm for a given specific problem. 2. The authors have provided sufficient novelty in this paper. Specifically, considering the deep active learning setting, the reward function design along with the bandit setting seems novel and non-trivial to me. 3. The extensive experimentation is conducted considering multiple datasets under multiclass and multilabel settings. This enhances the credibility of the effectiveness of the proposed TAILOR framework. 4. The theoretical analysis showing better regret bounds for the proposed TAILOR algorithm compared to linear contextual bandits seems to be a very important contribution.

Weaknesses

1. The experimentation is conducted using a single ResNet-18 architecture. Having experimental results on multiple architectures, especially more powerful ones, such as ResNet101, ViT would be helpful to even strengthen the effectiveness of the proposed technique. 2. The impact of \gamma (line 281) is missing. I would expect to see the performance variation with respect to different \gamma values (some ablation study would be helpful). 3. One of the crucial components is the reward that is designed to maximize the participation of the data samples from the minority (rarest) classes in an imbalanced data setting. It would be interesting to see how the involvement of those minority (rarest) class data samples affects minority class performance. An ablation study comparing the rarest class performance of the proposed TAILOR framework with regard to different baselines would be interesting.

Questions

1. In the rebuttal, it would be interesting to see experimental results mentioned in the weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

See Weaknesses.

Summary

This paper proposes a new method for algorithm selection in active learning. The problem is treated as a multi-armed bandit problem and the proposed method is based on Thompson sampling. Extensive experiments are conducted on several datasets, including both multi-class and multi-label classification setting.

Strengths

1. The paper is well-written and easy to follow. 2. The new reward functions designed for label balance are interesting. 3. The authors provide regret analysis and the proposed method enjoys better regret bound than the standard linear contextual bandit algorithm. 4. The experimental results are comprehensive, including several benchmarks and two settings. TAILOR is comparable to the best candidate algorithm and even performs better than the best candidate for multi-label setting.

Weaknesses

1. My major concern is the motivation behind active learning algorithm selection. For all the multi-class datasets, BADGE is almost always the best algorithm. Are there any datasets where BADGE does not perform well? 2. What's the time efficiency of TAILOR? Is TAILOR less efficient than the single algorithm? Minor: The figure is somewhat difficult to read the specific numbers. It would be better to provide an additional table.

Questions

1. Are there any datasets where BADGE does not perform well? 2. Can BADGE be adapted for multi-label setting? 3. What's the time efficiency of TAILOR? Is TAILOR less efficient than the single algorithm?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

The paper proposed TAILOR, a Thompson Sampling framework for active learning algorithm selection for unlabeled, possibly imbalanced datasets by framing it as a multi-arm bandit problem. The authors compared with random Meta and another meta-framework ALBL, along with other AL methods on 10 multi-class multi-label vision datasets and concluded their framework new art.

Strengths

Originality: The work is deemed original based on its discussion with related work. Clarity & quality: The work is well written with great graphics like Figure 1 to illustrate key ideas. The appendix is also very helpful information. Significance: The work is relevant to the broader community of active learning and computer vision. The technical contribution it makes is solid in terms that the TAILOR does show improved AL efficacy compared with individual AL methods.

Weaknesses

I have a suggestion for clarity: Add / explain what is the pool of AL algorithms that TAILOR considers at the very beginning of the paper (perhaps the intro part) as the plot shows 81 algs are considered (which can raise confusion for readers) My major concern about the paper lies in the necessity of the different pieces of the reward function. I think an ablation study on the three different types of rewards function would do great help towards readers' understanding of relative contributions of their modules.

Questions

1. As an active learning practitioner, my experience with active learning strongly signifies a non-stationarity in the effectiveness of different AL methods at different label levels (they are also path dependent, not only the number of labels but which labels were annotated till this iteration matters for the next round reward). Therefore I would like to ask how the authors bake this non-stationarity into the current formation. 2. What exactly are the 40 algs in the uncertainty (40 algs)? I didn't find it in Appendix F.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No potential negative social impact of this work is observed. The authors are encouraged to address their limitations more.

Summary

Selecting the most appropriate active learning algorithm for a given dataset poses a significant challenge when applying active learning in real-world scenarios. This challenge stems from the fact that the performance of different active learning strategies could vary significantly across various scenarios and datasets. The paper introduces an interactive and adaptive algorithm selection strategy inspired by the concept of multi-armed bandit. By employing carefully designed reward functions, the proposed strategy can identify a subset of acquisition functions that yield maximum rewards. The authors prove that their selection algorithm exhibits a tighter bound on the Bayesian regret. Experimental evaluations conducted on both multi-class and multi-label classification tasks illustrate the promising performance of the proposed approach in terms of accuracy.

Strengths

* Given the diverse nature of datasets, it is crucial to carefully consider and evaluate various active learning algorithms to achieve optimal performance and effectiveness. The concept of reducing the selection of acquisition functions to a multi-armed bandit problem is interesting as it allows the election process to incorporate accumulated rewards, providing a valuable approach for algorithm selection. * The proposed method demonstrates adaptability to both multi-class and multi-label classification tasks by utilizing well-designed reward functions that suitably capture the task-specific requirements. * Additionally, the authors present a proof establishing that the proposed selection algorithm outperforms an existing algorithm with a higher bound on Bayesian regret, although the reviewer did not thoroughly examine the soundness of the proof.

Weaknesses

* The class imbalance issue is dealt with through a proper designed reward function, discussed in section 4.1. The idea of the inversely weighting each class based on its samples is not supervising, given that the weight scheme has been widely used in imbalanced semi supervised learning. The main concern is rather the empirical studies. Even though Figure 3(c) seems to show that the proposed method could deal with the imbalance problem, there is lack of studies about how tolerate the proposed method to different class imbalance ratios. Instead of showing the number of samples in the rarest class, it is better to plot the imbalance ratio. Furthermore, it is not clear how the imbalance issue is handled in the multi-label classification tasks. * With respect to Assumption 3.1, it basically assumes to choose the top B samples ranked based on, for example uncertainty/least confidence scores. However, there are other methods, in particularly, some diversity-based AL methods, that choose the samples using clustering, which cannot be directly converted into an iterative selection process. * The efficiency aspect of the proposed method is not adequately discussed in the paper. Consideration of efficiency factors, such as computational cost or time complexity, would provide a more holistic evaluation of the proposed method's practicality and real-world applicability.

Questions

* For each iteration of acquisition, B samples are selected from B acquisition strategies, it sounds like an ensemble approach, doesn’t it? Different acquisition functions might capture, for example, different types of uncertainties, some to do with the model parameters, some to do directly with the model performance. Will this contribute to the performance difference? * The reward function focuses on the class diversity in order to deal with the class imbalanced issue. It would be interesting to design some reward function that is closely related to the model performance. * It is good to show the higher bound. However, can the author prove the convergence of the classifier, as done in WMOCU (Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xiaoning Qian. Uncertaintyaware active learning for optimal Bayesian classifier. In International Conference on Learning Representations, ICLR 2021, 2021) * In figure 3, there is no substantial difference between TAILOR and some other methods, including BADGE, can the author explain this? Meanwhile, BADGE is a diversity-based method using clustering, how this can be adopted in TAILOR with Assumption 3.1?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The author briefly touches on the limitations in section 6.3 and section 7. As mentioned above, it is also good to discuss more about the computational cost which could potential benefit the AL practitioners.

Summary

This paper proposes an algorithm, TAILOR, that iteratively and adaptively selects candidate active learning algorithm to gather class-balanced examples. Experimental results demonstrate that TAILOR achieves comparable or better performance than the best of the candidate algorithms.

Strengths

The paper presents a new active learning algorithm and provides some theoretical analysis. Good empirical results are reported.

Weaknesses

The presentation of the paper leaves much to be desired, and its contribution appears to be quite limited. Additionally, the paper contains improper or insufficiently supported claims, such as “meta algorithm”, “focus on class imbalanced setting”, and “the first adaptive algorithm selection strategy”. 1. The organization and content selection are unsatisfactory. (1) The introduction is not well motivated or presented. The framework illustrated in Figure 1 is an adaptive active learning procedure, and it is difficult to build connection with a multi-armed bandit problem from such a figure. It is not very proper to report a result figure in the introduction as well. (2) Algorithm 1 is not the contribution of this paper but was included on page 4 with a lot of space. Since most contents of section 3.2 are from the literature, the presentation can be largely compressed. (3) Much space of the paper is not on the main contribution of the paper. The proposed algorithm TAILOR is only presented in 4.3. (4) Throughout the overall paper (in particular section 4), method presentation, previous work review, and contribution claims are jumbled together without a clear and logical structure of presentation. 2. The presentation of the paper lacks sufficient clarity. (1) In 4.2, the notations and concepts are not well explained. For example, what is the 1-sub-Gaussian distribution? What does it mean by stating “Nature reveals weighting vector v^t”? (2) Section 4.2 also fails to present a clear and principled connection of the proposed active learning setting to the bandit problem. (3) The Equations are not well explained. For example, the “\lor” operator in the class diversity reward is not described. 3. Authors claim the algorithm as a meta algorithm. The TAILOR algorithm in Algorithm 2 didn’t show any meta-learning process. The meta concept needs to be clarified. 4. The paper claims on focusing on class imbalanced setting, but fails to provide any substantial discussion or analysis on this aspect, except for conducting experiments on datasets with class-imbalance. 5. The reward functions proposed in 4.1 are heuristic and lack principled connections to the model performance. There are no discussions on how to choose reward functions on different datasets. 6. The claim of “the first adaptive algorithm selection strategy for deep active learning” is not proper. Adaptive active learning has been studied in the active learning literature. Most active learning strategies developed in the literature are not dependent on the classifiers, and hence there is no need to distinguish linear models from deep models from the active learning perspective. In particular, the proposed active learning method has no special ties to deep models. 7. Many related works on adaptive active learning or meta active learning are missing, e.g., [1,2,3]. [1] “Adaptive Active Learning for Image Classification”. 2013. [2] “Active Learning with Multi-label SVM Classification”. 2013. [3] “Meta-Learning for Batch Mode Active Learning”, 2018.

Questions

Please see the weaknesses above.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

N/A

Author-Reviewer Discussion

Thanks to the authors for submitting a detailed rebuttal. Can the reviewers please read the author response and let them know if they have any further questions / need any additional clarifications? Best regards, - Your AC.

Thanks for the Rebuttal

Thanks for the rebuttal, I would like to keep my current score. Also, I would like to see the ablation study result and large architecture result in the revised paper.