Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurement

W1: Generalizing to other vision tasks (e.g. object detection and segmentation)

We thank Reviewer o3dH for the constructive feedback on __recognizing the motivation__, __acknowledging the significant finding that has not been previously overlooked in the literature__ as well as __appreciating the utility of our benchmark tool__. The comments have been instrumental in refining our work. We would like to address each concern separately and apologize for the delay in our response due to our commitment to conducting additional experiments. __W1: Generalizing to other vision tasks (e.g. object detection and segmentation)__ __Ans:__ We agree on the review that this study on few-class models should be extended to other vision tasks as mentioned in line 115 in our original submission. As the first work and benchmark tool to assist in application and research in the few-class regime, we intend to __remain our focus solely on image classification (IC) in this work__ with object detection (OD) and segmentation (Seg) as future work. This strategic decision aligns with the progressive advancements in the computer vision community, which often begins with IC (such as the ResNet [1], EfficientNet [2], ViT [3], MMPretrain [4]) before moving to more complex tasks of OD or S, such as the R-CNN series [5-6], EfficientDet [7], ViT for OD [8], MMOD[9] etc. To __assess the generalization of the few-class properties to OD and Seg__, we conducted additional __validation__ experiments using YOLOv8. The procedure is consistent with the method outlined in the Figure 1 caption. Specifically, for a specific $N_{CL}$ (2 in this example), we randomly sample the (2) classes from the full dataset of COCO, where each $N_{CL}$ consists of five subsets with seed numbers from 0 to 4. We performed experiments with $N_{CL} = $\{2, 5, 80\}. The YOLOv8-nano model was chosen since we focus on efficiency. Image size of 320x320 was used. Model performance was evaluated using the standard metric, mean average precision at an IoU threshold of 0.5 (__mAP@50__). The tables below summarize our results: --- __Table of Average Score↑__ | Model | $N_{CL}$ | OD mAP@50 | Seg mAP@50 | |----------|----------|----------|----------| | YOLOv8-nano F | 2 | 0.488 | 0.475 | | __YOLOv8-nano S__ | 2 | __0.538 (0.150+)__ | __0.482 (0.007+)__ | | YOLOv8-nano F | 5 | 0.456 | 0.435 | | __YOLOv8-nano S__ | 5 | __0.503 (0.047+)__ | __0.474 (0.039+)__ | | YOLOv8-nano F | 80 | 0.405 | 0.378 | --- __Table of Standard Deviation↓__ | Model | $N_{CL}$ | OD mAP@50 | Seg mAP@50 | |----------|----------|----------|----------| | YOLOv8-nano F | 2 | 0.161 | 0.180 | | __YOLOv8-nano S__ | 2 | __0.106 (0.055-)__ | __0.152 (0.028-)__ | | YOLOv8-nano F | 5 | 0.090 | 0.098 | | __YOLOv8-nano S__ | 5 | __0.069 (0.021-)__ | __0.084 (0.014-)__ | | YOLOv8-nano F | 80 | 0.195 | 0.200 | --- where each row summarizes the results of 5 models for a given $N_{CL}$. F: Full model; S: Sub-model. Since there is no "few class" for the full dataset ($N_{CL}=80$), the corresponding model consists only of the full model (using the downloaded pre-trained weights). The arrow signs shows the direction of a better value, e.g., ↑ means higher is better. Overall, these results in OD and S align with the main findings: _Sub-models attain higher upper-bound accuracy than full models_, as indicated by the scores highlighted with the (+) sign in Table of Average Scores↑; _The range of accuracy widens for full models at few-classes, which increases the uncertainty of a practitioner selecting a model for few classes. In contrast, sub-models narrow the range_, as shown in Table of Standard Deviation↓. We will append these results and discussions in the draft shortly. [1] He et al. "Identity mappings in deep residual networks." ECCV, 2016. [2] Tan et al. "Efficientnet: Rethinking model scaling for convolutional neural networks." PMLR, 2019. [3] Dosovitskiy et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv:2010.11929 (2020). [4] MMPreTrain Contributors. Openmmlab’s pre-training toolbox and benchmark. https://github.com/open-mmlab/mmpretrain, 2023. [5] He et al. "Mask r-cnn." ICCV. 2017. [6] Ren et al. "Faster R-CNN: Towards real-time object detection with region proposal networks." TPAMI 2016. [7] Tan et al. "Efficientdet: Scalable and efficient object detection."CVPR 2020. [8] Li et al. "Exploring plain vision transformer backbones for object detection." ECCV 2022. [9] Chen et al. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.

Author-year citations

Thank you for the suggestion of the correct use of author-year citations. We have updated the draft accordingly. For instance, "_Typical examples include 1000 classes in ImageNet Deng et al. (2009) for image classification, and 80 object categories in COCO Lin et al. (2014) for object detection. Previous benchmarks also extend vision to multimodal research such as image-text Lee et al. (2024); Le et al. (2024); Laurenc¸on et al. (2024); Bitton et al. (2022)._" has been corrected as "_Typical examples include 1000 classes in ImageNet (Deng et al., 2009) for image classification, and 80 object categories in COCO (Lin et al., 2014) for object detection. Previous benchmarks also extend vision to multimodal research such as image-text (Lee et al., 2024; Le et al., 2024; Laurenc ̧on et al., 2024; Bitton et al., 2022)._"

W2 & Q1: Discussion on fine-tuned models by pre-trained self-supervised models

__W2__: "*The experiments cover classification tasks based on supervised (pre-)training. However, there is an increasing trend that classification models are fine-tuning based on self-supervised pre-trained models. This paradigm is not covered in this study, and therefore, the findings are limited to the more traditional fully supervised paradigm. The authors could discuss these aspects or add experimentation with self-supervised pre-trained models (if available).*" __Q1__: "*(Related to W2) The finding that the scaling law w.r.t. model size is violated for submodes is interesting. I would be curious if this only applies to supervised training or also to self-supervised pre-training combined with minimal fine-tuning or linear probing. Did you conduct any experiments into this direction or do you have an intuition on that?*" __Ans__: We appreciate the suggestion of pre-trained models by Self-Supervised Learning (SSL) methods, which can potentially bring new insights into the Few-Class Regime study. We would like to first highlight the __research goal differences__: in the __Few-Class Regime__, our goal is to identify the __smallest baseline__ model that achieves the desired performance accuracy while utilizing the __minimal set of features__ necessary for the target __few-class applications__. This direction is fundamentally different from the SSL framework that aim to learn __general representations__ by leveraging the knowledge from large amounts of __unlabeled data__, typically via a a pretext task. This can further enable models adapted to a target application from a large well pre-trained SSL model. However, we hypothesize that such a (SSL) pre-trained model may have included __extraneous__ features that are not required in the target application, which can contradict with the aforementioned "efficiency" goal in the __Few-Class Regime__ (discussed on Line 384 in Section 4.2 RESULTS ON FC-SUB). To investigate the Few-Class problems, we strategically select sub-models as they are only trained on the target few-classes. We apologize for the lack of clarity regarding their relations and differences, and have addressed this issue in the revised draft. Please refer to Line 47 and Line 845 in A.3 EXTENDED RELATED WORK within the Appendix for a detailed explanation. The new addition is highlighted in blue. We have included additional study on comparing with fine-tuned (SSL) models in Table 2 in 4.4 COMPARISON WITH FINE-TUNED MODELS in the revision. Table 2 shows the Top-1 Accuracies for different configurations on CIFAR100. $N_{CL} \in$ \{2, 4, 100\}. Best scores are highlighted in bold. For ViT, we fine-tune a ViT-B model initialized with weights from the CLIP pre-trained backbone. A linear layer is added on top, and the model is trained for 10 epochs. This setup is indicated by the star symbol (*). --- __Table 2__ | Model Type | $N_{CL}$ | ResNet18 | ResNet50 | MobileViT-Small | ViT-B | |----------|----------|----------|----------|----------|----------| | Full | 100 | 76.11 | 73.71 | 73.83 | 32.54 | | Full | 4 | 75.10 | 72.20 | 72.35 | 36.15 | | Fine-tuned | 4 | 87.60 | __90.55__ | __90.00__ | __91.16__* | | Sub | 4 | __90.65__ | 90.15 | 89.45 | 85.40 | | Full | 2 | 75.00 | 71.30 | 71.80 | 40.80 | | Fine-tuned | 2 | 87.90 | 93.70 | 90.50 | 95.20* | | Sub | 2 | __96.30__ | __95.30__ | __95.50__ | __95.90__ | --- We conclude that the fine-tuned models exhibit patterns and trends consistent with the observations presented in Fig. 1. Note our focus of this work is to leverage the proposed difficult measurement method, FC-Sim, to efficiently estimate the achievable model accuracy, thereby assisting in model selection in the Few-Class Regime. Sub-models can offer insights into the minimal visual features required for a specific real-world scenario as they are trained exclusively on the target classes. In contrast, weights pre-trained on large full datasets -- whether through fully supervised or self-supervised manner -- may include extraneous features that are irrelevant to the target classes. We hereby prioritize sub-model study in this work.

W1: Adaptation Layers in Transfer Learning

We sincerely thank Reviewer 2XhF for appreciating the breadth of our work ("many models with many datasets"), the thoroughness and depth of the trained models, the strong correlation demonstrated between the proposed similarity measure and performance, as well as the clarity and quality of our presentation and writing. __W1.1__: _"The motivation behind few-class evaluations is not fully convincing. In practice, one will take a large model and fine-tune it (without the classification layer) to a target set of classes, hence obtaining a specialized model. "_ __Ans__: We agree on the commonly used fine-tuning method to adapt to a target sub classes. We would like to first highlight the __research goal differences__: in the __Few-Class Regime__, our goal is to identify the __smallest baseline__ model that achieves the desired performance accuracy while utilizing the __minimal set of features (MF)__ necessary for the target __few-class applications__. Fine-tuning, however, does __not explicitly__ take MF into consideration. We hypothesize that such a large pre-trained model may have included __extraneous__ features that are not required in the target application, which can contradict with the aforementioned "efficiency" goal in the __Few-Class Regime__ (discussed on Line 384 in Section 4.2 RESULTS ON FC-SUB). Therefore, we argue that full model evaluation is necessary to investigate the Few-Class problems. Please refer to Line 47 and Line 845 in A.3 EXTENDED RELATED WORK within the Appendix for a detailed explanation. The new addition is highlighted in blue. We have included __additional__ study on comparing with __fine-tuned__ pre-trained models in Table 2 in 4.4 COMPARISON WITH FINE-TUNED MODELS in the revision. We also include the results of transfer learning from a Self-Supervised pre-trained model ViT-B. Table 2 shows the Top-1 Accuracies for different configurations on CIFAR100. $N_{CL} \in$ \{2, 4, 100\}. Best scores are highlighted in bold. For ViT, we fine-tune a ViT-B model initialized with weights from the CLIP pre-trained backbone. A linear layer is added on top, and the model is trained for 10 epochs. This setup is indicated by the star symbol (*). --- __Table 2__ | Model Type | $N_{CL}$ | ResNet18 | ResNet50 | MobileViT-Small | ViT-B | |----------|----------|----------|----------|----------|----------| | Full | 100 | 76.11 | 73.71 | 73.83 | 32.54 | | Full | 4 | 75.10 | 72.20 | 72.35 | 36.15 | | Fine-tuned | 4 | 87.60 | __90.55__ | __90.00__ | __91.16__* | | Sub | 4 | __90.65__ | 90.15 | 89.45 | 85.40 | | Full | 2 | 75.00 | 71.30 | 71.80 | 40.80 | | Fine-tuned | 2 | 87.90 | 93.70 | 90.50 | 95.20* | | Sub | 2 | __96.30__ | __95.30__ | __95.50__ | __95.90__ | --- We conclude that the fine-tuned models exhibit patterns and trends consistent with the observations presented in Fig. 1. Note our focus of this work is to leverage the proposed difficult measurement method, FC-Sim, to efficiently estimate the achievable model accuracy, thereby assisting in model selection in the Few-Class Regime. __W1.2__: "_When does that happen in practice, and could you just not use small adaptation layers on top of frozen backbone for each of the subsets ?_" __Ans__: It happens when the usage environment __shares a subset of the object categories__ with the large pre-trained model, but __ground truth labels are unavailable__ in the target environment, which is very common when users would like to simply apply off-the-shelf models whose weights are pre-trained from large many-class datasets. Popular examples include deploying pre-trained YOLO models directly for industrial use cases, such as detecting general objects like pedestrians and vehicles. (For details on extending to object detection, please see Section A.12 in the revision and our Response to W1: Generalizing to other vision tasks (e.g., object detection and segmentation) for Reviewer o3dH. In this work, however, we focus specifically on image classification.) When a large model encounters novel classes in the deployment that were not included in its training dataset, ground truth labels are typically required in order to train new adaptation layers for classification. If we don't use any adaptation layers trained by the novel class ground truth labels, the model will simply output the class label with the highest confidence, but the correctness of this prediction cannot be guaranteed."

W2: Recommendation for practitioners

__W2.1__: "_One thing that is missing from the paper is a recommendation for practitioners on which vision model to use for someone interested in the few-class problem._" __Ans__: We have added A.2 BENCHMARK USAGE GUIDELINE in the revised draft. We would like to distinguish between the use cases of practitioners (P) and researchers (R): Ps are primarily interested in selecting the optimal, model while Rs would like to study many models in the Few-Class Regime. For P, users can compute the proposed SimSS score to identify and select the model that best satisfies their accuracy and hardware constraints. For R, we assume the main interest is the study of model comparison (full and sub-models in FC-Full and FC-Sub), and accessing difficulty measurements in FC-Sim. Our benchmark tool features in configuration files that cover various scenarios by allowing specifications such as $N_{CL}$, seed numbers and so forth. It provides streamlined interfaces, sparing users from managing such tedious implementations for users to conduct large-scale experiments, detailed in Section 3 FEW-CLASS ARENA (FCA). __W2.2__: "_Basically discussing in more details the results from Table 1 and providing comparison between models in Section 4.2 and 4.3._" __Ans__: We have already provided the details (in the original draft) of ten models on the ten datasets from Fig. 8 to Fig. 17 in Section A.8. Due to the large amount of result data, we summarized the comparisons in Fig. 3 and 4 in Section 4.2 and 4.3 and discussed the general observations. We encourage a more concrete question if possible. __W2.3__: "_One interesting question is, do models that perform really well on the many classes setup are the same that also perform well on the few-class setup ?_" __Ans__: __Not necessarily__. We show that the rankings differ dramatically for different models on various datasets in Fig. 2 (b) and Fig. 7.

W3: Novel findings

__W3__: "_Some of the findings in the paper are fully expected. The fact that a model specifically trained on the target subset of classes perform better that a larger model trained on a superset is not very surprising or novel._" __Ans__: Our paper presents in a progressive way: first revisit some well observed findings in prior work (which are expected) "_(a) Sub-models attain higher upper-bound accuracy than full models.", "(b) The range of accuracy widens for full models at few-classes, which increases the uncertainty of a practitioner selecting a model for few classes. In contrast, sub-models narrow the range._" The __scaling law__ is an emerging general law but mainly describing how models __scale up__: "_(c) Full models follow the scaling law [1] in the dimension of model size - larger models (darker red) have higher accuracy from many to few classes_." Then we present our __novel findings__ of __scaling down__ in the __Few-Class Regime__: "_(d) Surprisingly, the scaling law is violated for sub-models in the Few-Class Regime where larger models do not necessarily perform better than smaller ones._" described from Line 94 - 102 and Fig. 1. The role of __image similarity__ (measured by SimSS as a proxy of dataset difficulty) is __more pronounced__ in the __Few-Class Regime__ than in the Many-Class Regime, as demonstrated across ten datasets with a wide range of $N_{CL}$ in Fig. 18 in Section A.10. Notably, this type of dataset difficulty measurement has not been considered in existing scaling laws, as mentioned in Line 1467. [1] Jared et al. Scaling laws forneural language preprintarXiv:2001.08361,2020.

Q: Original research to unlock

__Q__: "_What original research do you expect will use this benchmark and what do you hope it will achieve or unlock ?_" __Ans__: By using this tool, we aim to explore key research questions in the Few-Class Regime: What is the simplest baseline model capable of meeting performance criteria within this context? What is an effective dataset difficulty measurement method to assist in model selection? How does the scaling law behave in this setting? When scaling up to a larger dataset, the scaling Law might guide us in scaling up models in terms of data size, model size etc. However, we show that image similarity (as a proxy of dataset difficulty) plays a more and more important role as $N_{CL}$ decreases to the Few-Class Regime, as demonstrated across all datasets in Fig. 18. We argue that simply downscaling a model blindly without considering class similarity may yield a model selection with sub-optimal efficiency. Therefore, our benchmark tool can assist in the next (down) scaling law that takes image similarity (as a way to measure dataset difficulty) into consideration for model selection. These issues have been discussed from Line 1463 - 1471. We have introduced a new argument on Line 47 in the revised draft.

Actionable suggestions

Dear Reviewer 6NMw, we really appreciate your valuable suggestions. The feedback of "_suggest authors to extend their analysis_" is somewhat vague for us, we would greatly appreciate any actionable suggestions for experiments to help extend our analysis and contribute to improving our work. Thank you!

W1: Few classes selection and semantic closeness

We thank Reviewer 6NMw for recognizing the utility of our benchmark, the behavioral analysis of models, and the advantages of the proposed similarity metric in correlating with model performance. We would address each concern separately. __W1__: "_...there are no details discussed on how these few classes have been selected and how semantically close these few classes to each other?_" __Ans__: __Few classes are randomly sampled by seed numbers from 0 - 4 by default.__ The details were specified on __Line 82__: "_Each model is tested on 5 subsets whose_ $N_{CL}$ _classes are randomly sampled from the original 1000 classes._", __Line 187__: "_Few-Class Arena generates the specific model and dataset configuration files for each subset, where subset classes are randomly extracted from the full set of classes, as specified by the seed number._" and __Line 412__: "_For reproducible results, we use seed numbers from 0 to 4 to generate 5 subsets for one_ $N_{CL}$ _by default._" For extension to Object Detection and Segmentation, these are detailed from __Line 1610 to 1613__: "_The procedure is consistent with the method outlined in Section3.3 and 3.4. Specifically, for a specific_ $N_{CL}$ (2 in this example), we randomly sample the_ $N_{CL}$ _classes from the full dataset of COCO, where each consists of five subsets with seed numbers from 0 to 4. We performed experiments with 2, 5, 80._" The proposed Similarity-Based Silhouette Score (SimSS) is designed to capture the __semantic cluster__ characteristics of few classes, in particular "_the (1) tightness of a class cluster and (2) distance to other classes of class clusters, are features that characterize the inherent class difficulty,_" described on __Line 315__. We kindly recommend that the reviewer examine __Fig. 18__ closely for detailed insights into the semantic closeness across ten datasets, spanning from the Many-Class to the __Few-Class Regime__.

W2: Transfer Learning

__W2__: "_The aspect of transfer learning has not been discussed. It is a common practice to finetune ImageNet pretrained models like ResNet50 or ViT, or recent foundation models like CLIP and DINOv2 to different downstream tasks that include adapting or finetuning them on few classes. The analysis presented in the paper is missing this exploration. Is it better to train the models from scratch on the few classes or finetuning these models work better for few classes? Does SimSS score also align on the finetuned models?_" __Ans__: For transfer learning, we conducted __additional__ experiments during the rebuttal period. Please kindly refer to the response to _W1: Adaptation Layers in Transfer Learning_ by Reviewer 2XhF and _W2 & Q1: Discussion on fine-tuned models by pre-trained self-supervised models_ by Reviewer o3dH. The empirical observation is that the performance of the finetuned models is close to the sub-models, therefore the SimSS scores should align with the finetuned models.

W3: How is C_hat acquired?

__W3__: "_To compute SimSS, a score called Nearest Inter-Class Similarity requires a nearest class (C_hat) to the target class (C), it is not clear how this C_hat is acquired._" __Ans__: As described on Line 326, it is "_a scalar describing the similarity among instances between class C and the closest class of each instance in C_". It is computed simply using a __max()__ function (where a higher similarity score indicates greater closeness) applied to the class candidate list (sim_c_p_ls) in our released code: max(sim_c_p_ls) https://github.com/fewclassarena/fca/blob/aa796880953a58f79b243a855d7aad3a221b8587/configs/_base_/sim.py#L220

Q1: Full model predictions in the Few-Class Regime

__Q1__: "_For FC-Full, when_ $N_{CL}$ _decreases, how to make sure that model predicts only few classes? Are the logits of those few classes are selected to get the prediction and discard logits of all other classes?_" __Ans__: __No modification of the logits (or any layers).__ The few classes are a subset of the full dataset, we let the full models output the labels (predictions) and compare them against the ground truth label to calculate the Top-1 accuracy.

Q2: Benchmark usage guideline

__Q2__: "_if a user has a custom dataset with few classes and want to find a model that works better on this custom dataset, it would be helpful to have an explanation on how this benchmark can assist the user in this case._" __Ans__: We have added A.2 BENCHMARK USAGE GUIDELINE in the revised draft. Please refer to the response to "Q1: Difficulty metric usage guideline" by Reviewer trsf and "W2: Recommendation for practitioners" by Reviewer 2XhF.

Responses to Weaknesses MO1 & M02

We sincerely thank Reviewer trsf for __acknowledging the limitations we identified in commonly used many-class datasets__, as well as __our efforts in open sourcing our code with detailed documentation__. We also appreciate Reviewer trsf's __recognition of our new findings in the few-class regime__ that challenge the prevailing assumptions and motivate us to explore further into this direction. This has led to the development of the first benchmark specifically designed for few-class problems. We would like to emphasize that, in addition to releasing our benchmark tool, we have also conducted extensive large-scale experiments. Our key findings, summarized in the Statement of Contributions encompassing a diverse range of architectures of CNNs and transformers, evaluated on 10 datasets with a total of 1591 training and testing runs. The original submission includes comprehensive results for pre-trained models, referred to as Full-models, which are summarized in Figure 3. These Full-models, including ResNet50, VGG16, ConNeXt V2 Base, Inception V3, EfficientNet V2 Medium, ShuffleNet V2, MobileNet V3 Small, ViT Base, Swin Transformer V2 Base and MobileViT Small have been evaluated on ten datasets for various number of classes. Detailed results for each model are presented in Figures 8-17 in the Appendix. __MO1: Citation format.__ __Ans__: We have corrected the citation format. Please check the revision. __MO2: L52: Main text does not seem to describe Figure 1 accurately?__ __Ans__: The main text summarizes the key findings in Figure 1. Sorry, I do not understand in what specific aspect the main text does not describe Figure 1. It would be great if you can elaborate the points concretely.

Q2: difficulty metric compute complexity, comparison to a single model training run

__Q2__: "_The proposed difficulty metric seems expensive to compute, with pairwise similarity scores required for large subsets of the data. How does this compare to the cost of conducting a single model training run?_" __Ans__: Since the difficulty metric involves pairwise computations, the time complexity is quadratic, which can be a problem when the number of classes is large. However, our target usage is in the Few-Class Regime with few classes. Based on our empirical testing on ten datasets, it takes around a few minutes to obtain results for $N_{CL}<10$. The cost is affordable compared to the hours or days required for model training. We would like to emphasize our motivation that without such a difficult score, it requires the re-evaluation of published models or even retraining (for many training and testing runs) to find an optimal model in an expensive architectural search space.