Summary
This work introduces a method named FreeSel, which is designed to efficiently select information-rich data points from an unlabeled pool. These selected data points can then be annotated and utilized for training deep learning models, thereby mitigating the costs and resources needed to annotate an entire dataset. The method primarily consists of two key components: feature extraction using a large-scale pretrained model (DINO), and the construction of semantic patterns based on intermediate features. Subsequently, a distance-based selection strategy is employed to identify the most diverse and informative data samples.
The authors assert that the proposed approach addresses three critical aspects: Generality, by decoupling data selection from task-specific models; Efficiency, by enabling the selection of samples in a single pass; and Non-Supervision, by eliminating the need for annotations until the data selection process is complete.
While the results on multiple datasets and tasks indicate that the proposed methodology holds promise, the paper seems to lack some important comparisons. Additionally, characterizing this work as an alternative to traditional active learning may not be entirely accurate.
Strengths
1. The idea of employing a large-scale pre-trained model for data selection is interesting. This is especially pertinent in today's environment, where there is a proliferation of large-scale foundational models. Harnessing these models to select data samples for training of downstream models could prove to be highly advantageous for addressing a wide range of problems.
2. The paper is well-structured and easy to understand
3. The inclusion of experiments across multiple tasks and datasets provides a comprehensive view of the empirical performance of the proposed approach
Weaknesses
1. The primary concern regarding this work pertains to its positioning. The paper seems to be more aligned with the domain of subset selection rather than active learning (AL).
* AL methods, as the name suggests, actively learn which samples to select in a manner that is specific to the model at hand. While I concur with the authors' perspective that this iterative method is resource-intensive, drawing a direct comparison between the proposed "passive" selection method and traditional active learning in terms of training time may not be entirely appropriate.
2. There is a need for additional baselines to more effectively evaluate the merits of the proposed approach.
* For instance, in the absence of any iterative active learning, if a random subset of data is selected and a model is trained on it, how does the accuracy of this model compare to one trained on samples selected using FreeSel?
3. There appears to be some conflation between the method (FreeSel) and the problem setting (data selection using large-scale pre-trained models). For example, the principle of Non-supervision (Line 48) seems more pertinent to the methodology than to the problem setting.
4. To gain a clearer understanding of the novelty and contributions of this work, it would be beneficial to compare the method with existing subset selection methods.Such comparisons do not necessarily need to be empirical but could include theoretical or conceptual comparisons.
5. It would be beneficial for the authors to clarify the positioning of FreeSel in relation to traditional active learning and to provide additional comparisons to strengthen the evaluation.
References for consideration:
1. [Kaushal et al., Learning From Less Data: Diversified Subset Selection and Active Learning in Image Classification Tasks](https://arxiv.org/pdf/1805.11191)
2. [Chang et al., On Training Instance Selection for Few-Shot Neural Text Generation](https://arxiv.org/pdf/2107.03176)
3. [Birodkar et al., Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need](https://arxiv.org/pdf/1901.11409)
4. [Ramalingam, et al., Less is more: Selecting informative and diverse subsets with balancing constraints](https://arxiv.org/pdf/2104.12835)
Questions
1. What is the rationale for choosing CoreSet in Figure, there are much more recent and useful active learning strategies that are specifically proposed for the problem of object detection?
2. No comparisons were made with the works listed in `Data Selection with Pretrained Models` paragraph of related work though they seem very relevant, is there any specific reason for this? Including comparisons with these pertinent works could enhance the comprehensiveness and depth of the evaluation.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
All the limitations are clearly discussed