Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment

Summary

The authors identify five main pitfalls that have impacted previous Active Learning (AL) empirical evaluations. They propose a novel evaluation framework that avoids these pitfalls, and they study both Active Learning and its potential synergies with Semi-Supervised and Self-supervised learning. The paper provides a comprehensive empirical evaluation, together with copious appendices.

Strengths

The paper's main contribution is to carefully catalogue and map the various critical aspects of a proper AL evaluation, aka The Five Pitfalls; in contrast, prior work has properly addressed at most 3 of the 5 pitfalls. The thorough empirical evaluation emphasizes the proper way to create an evaluation framework for active learning, and how to also take into account Semi- & Self- Supervised methods. The paper is fairly easy to read and follow.

Weaknesses

The paper can be further improved along the readability dimension. First of all, in its current form, the paper has "too much content in too little space" Even though, at first glance, the current split between "the main paper" and "the appendices" seems reasonable, the paper would greatly benefit by compressing Section 2.3 (which could be then added "as is" in an APPENDIX). On the other hand, Section 4 is hard to follow because the graphs in Fig 2 are too small to read, so the prose is not really supported by Fig 2. One possible way to address this issue would be to move most of the 3-page Section 4 to the APPENDICES and replace it with (A) a brief, 1-page summary of the results - no figures, and (B) a new section with an INTUITIVE, PRESCRIPTIVE section that crisply summarizes the main findings: "the ideal evaluation framework for active learning for image classification is the following ..." Second, the idea of "rolling-out Active Learning" to "unseen data sets different from the labeled development data" (first introduced at lines 32-33) is mentioned 13 times without being properly explained. In this reviewer's opinion, the idea is worth being presented in the main part of the paper, but the authors should invest the necessary 2-3 paragraphs to crisply & intuitively explain it. OTHER COMMENTS: - active learning does not begin/end with DL and image classification. As such, the authors could easily take the 1-2 sentences required to explicitly define the scope of their investigation.

Questions

1) the four "motivating papers (lines 23-24)" [6, 19, 38, 39] use, besides CIFAR-10/100, ImageNet and Tiny ImageNet; why didn't you consider these datasets, given that some of those papers' mistaken claims are made on them? 2) on line 198-199, you claim that the three CIFARs are "developmental datasets," while the other two are used for roll-out. While this is a great idea, an even better one would be to (i) drop the CIFARs, which are too "nice & easy" to begin with, (ii) add another 2-3 non-CIFAR, real-world datasets, and (iiI) perform either Cross-Validation or Leave-One-Out on the 5 datasets - say, by taking turns in using 1-4 of them as dev datasets, and the remaining one(s) as test dataset. Did you consider such an approach? 3) I am not sure if this is just a flaw of the empirical evaluation, or is should be added as a 6th pitfall, but it is always bothersome to see that the CIFAR-10/100 evaluations of an active learner ends way earlier than the SOTA performance of 99% and 96%, respectively (in this paper, the learning curves end around 94% and 76%, respectively). After all, AL is NOT supposed to be for "quickly reaching mediocre performance," but rather "creating the best model with less data than required by random sampling." It could be that this is due to the uniform distribution of the CIFARs (thus making them unrealistic & unsuitable for this evaluation), but this issue should be addressed head-on rather omitted.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

See Weaknesses & Questions above

Summary

The submission presents a large collection of experimental results for four active learning methods (entropy, core-set, BALD, BATCH) applied to image classification using ResNet-18 models, investigating the effect of differences in the data distribution (e.g., "roll-out data" from a different domain), size of initial labeled dataset, size of query batch, and integration of self- and semi-supervised learning. Special care is taken to optimize the supervised baseline and the active learning methods (on the development datasets).

Strengths

The submission presents an extensive collection of results and appears to be the only study so far that jointly considers all the five "pitfalls" referred to in the manuscript. The poor performance of semi-supervised learning on the roll-out datasets is remarkable, and the results bolster the case for the usefulness of active learning. It is very useful to see an independent evaluation of BADGE that indicates its performance is robust.

Weaknesses

Two of the main findings do not appear to be novel: the usefulness of careful parameter tuning and the beneficial effect of self-supervised learning (although this new work perhaps provides more extensive empirical support). The results are based on a single neural network architecture (ResNet18) and a single performance metric on each of the two groups of datasets (plain accuracy on the development datasets and balanced accuracy on the roll-out datasets). Only a single semi-supervised learning algorithm is considered (a version of FixMatch). Considering the statement that "Semi-SL performs best on toy datasets but does not seem to generalize to more complex settings.", this seems problematic. The experiments seem to jointly vary the batch size and the size of the initial labeled dataset. It seems difficult to disentangle the effect of these two parameters based on the presentation of the results in the paper, and it is unclear whether all the claims regarding the effect of these two parameters are supported. The submission implies that active learning is particularly useful when dealing with imbalanced data. However, it does not appear to consider techniques applied in standard supervised learning to deal with imbalanced data. Full results for semi-supervised learning on MIO-TCD and ISIC-2019 should be included. ----- The rebuttal has satisfactorily addressed these comments. -----

Questions

N/A

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

Active Learning is a crucial problem that focuses on selecting a subset of examples from an unlabeled dataset to be labeled. Primarily community focus on two pillars for defining acquisition functions -- uncertainty and diversity based acquisition methods. With emergence of the highly powerful pretrained models and alternative paradigm such as self SL and semi SL, the usecase of when to use AL is getting questionable. That being said, this work begins with listing down pillars at which AL scheme should be evaluated, starting from the hyperparameters to proper comparison against SSL baselines. Paper list down total 5 "pitfalls" and show how existing community has not been evaluating the proposed AL methods correctly. Paper then extensively evaluate the common AL QMs on different settings, and then giving concluding remarks on some of the existing methods, and guidance on when is AL beneficial.

Strengths

The paper is well written, easy to follow and targets a well motivated problem. Previous AL works often become cynical, and when not carefully studied the mentioned pitfall, it can give a false sense of label efficacy. Conducted experiments are solid, and the community would benefit with it.

Weaknesses

Weakness are more of some additional experimental suggestions. Given the abundance of pretrained CLIP models, I think adding them would also have been a great idea. Secondly, in terms of difficulty, paper could've added some large scale datasets such as Imagenet variants. Lastly, one additional pillar could also be studying the OOD robustness. This was recently studied in the AL paper -- "Continual Active Learning" which aimed to speedup the active learning, and proposed additional evaluation based on robustness. Lastly, there is a typo on Fig 2a, HL, where 10000 is written as 1000. - Das et al. 2022 Continual Active Learning. https://openreview.net/forum?id=GC5MsCxrU- / https://arxiv.org/abs/2305.06408

Questions

Please refer to weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Please refer to weakness section.

Summary

The authors in this work try to address the common issues as to why active learning has not been in use compared to other methods like semi-supervised or self-supervised learning. The authors argue that the active learning methods are not evaluated in robust manner and in a cross-task generalisation setting which leads to lower performance compared to semi-supervised and supervised learning. Hence, they also provide an evaluation framework to overcome these issues. The code for all their experiments is also made available. This work is useful to anyone who is new to the field of active learning and are deciding whether or not active learning would be useful for their task and dataset. The authors list five common pitfalls in the evaluation of active learning settings and suggest means to address these issues. Some of these issues are lack of evaluated data distribution settings, lack of evaluated starting budget, lack of evaluated query sizes, neglection of the classifier configuration and neglection of alternative training paradigms. Some conclusions of this work, like active learning could provide substantial gains in class-imbalanced settings, reiterating that BADGE is the most robust QM etc can help the readers intending to use AL in their works.

Strengths

The authors conduct a thorough ablation study to evaluate the active learning gains by changing the parameters of only one of these issues and keeping the rest constant. This has helped in filling the research gap between theoretical and practical works to use smaller query sizes as a potential solution for the instabilities of certain QMs like BALD. It has also helped in recalibrating the potential gains from active learning measured by previous works which could be attributed to hyperparameter settings. By introducing a practical way to obtain the best hyperparamter settings, this work shows the actual gains of active learning without overstating it.

Weaknesses

Since it was mentioned in the limitation that the training may not have been optimal for semi-SL method, how can the conclusion that "semi-SL performs best on toy datasets but does not seem to generalize to more complex settings" be made? Presentation - - Some lines contain too many pieces of information in a single line which can be organised in a more concise and clearer manner - A breif description of the citation would help in following the work better. For ex, writing a few words on what [6] is in line 313 would make the presentation better - The abbreviation ST was used in line 186 before clarifying the meaning in line 235 which can be confusing

Questions

- Can you clarify the relation between the starting budget and the labelling regimes(high-label,low-label etc) - Model selection for ST and Self-SL models is based on the best validation set epoch, while for Semi-SL models the final checkpoint is used. - Why? - Do the cost for the checks of the generalizability of AL in real-world settings itself negate the purpose of AL?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have clearly mentioned the limitations of their studies.

Summary

The paper identifies the pitfalls in current evaluation frameworks for AL, and proposed solutions that lead to a new evaluation framework (codebase) to tackle these problems. Empirical results reveal insights on the current state of AL methods.

Strengths

- The paper aims to tackle a very timely and important topic on how to more systematically and realistically evaluate AL methods. - The paper identifies several pitfalls in current AL evaluation and proposed solutions to overcome them. - The paper presents quite thorough experimental results accompanied by code releasing for future work to build on. - The key takeaways provide insights on the current state of AL literature.

Weaknesses

- Line 71 AL validation paradox: It is not very clear to me why AL parameters cannot be adapted to the dataset at hand. Why can't one adapt or even change query method in an online fashion like in [1]? I agree that one should not "use the performance" in a hindsight to choose the best AL method but that doesn't imply that we are only allowed to use fixed "pre-configured AL settings" for all new dataset? - While AL validation paradox appears to be the most important pitfall in AL evaluation, it's not immediately clear to me as how P1-5 addresses the validation paradox. Can the authors discuss the connection between AL paradox to P1-5? - While the paper has emphasized the importance of testing AL methods on new datasets and on realistic settings, the datasets included in the current experiments are limited in number, modality (only images) and relatively less realistic (e.g., CIFAR). Results/takeaways can be more convincing if more realistic datasets are used, and even include modalities other than image. [1] Active Learning by Learning. Hsu et al. 2015.

Questions

See above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses its limitation in its experimental setup.

Thanks for the rebuttal

I'd like to thank the authors for the rebuttal; However, I'd like to retain my rating.