> Thank you for their detailed response! The additional results are encouraging and addressed my primary concerns, and I have updated my rating to borderline accept. I feel that some of the rebuttal experiments (ImageNet, DINO w/ clustering, SSL) should be expanded and included in the paper to make the arguments more convincing.
We thank the reviewer for the response and for raising the score. We are glad that our clarifications helped to increase the reviewer's confidence in our work. We will include these additional experiments in our paper and we thank the reviewer for the suggestions.
Below we address the remaining concern of the reviewer. We hope that our response will further help to increase the reviewer's confidence in our work.
>I am still a bit concerned that unsupervised learning is less critical when strong representation spaces exist ("if the classes are already linearly separable, then only a few labeled examples per class are needed to achieve high accuracy, which should not cost substantial human effort to annotate" in original review), though I see the potential of HUME as a bridge between unsupervised and semi-supervised learning in eliminating the need to label even a few examples. I would love to know the authors' view on this.
We thank the reviewer for an additional question. In response to the reviewer’s feedback, we have conducted additional experiments that compare HUME to few-shot learning baselines. Namely, starting from the strongest DINO representation space, we employ two different few-shot baselines: (i) following [1], we compute a prototype for each class on top of DINO representations using few-labeled examples and then assign examples to the nearest prototype (we refer to it as proto baseline), and (ii) we train a linear classifier on top of DINO representations using a few labeled examples (we refer to it as linear baseline). Remarkably, the results show that HUME consistently outperforms these two baselines in the 1-shot and 2-shot learning settings on all datasets. In the 3-shot learning setting HUME outperforms these baselines on the STL-10 dataset, matches performance on the CIFAR-10 dataset, and these baselines show better performance only on the CIFAR100-20 which in this setting requires *60 labeled examples*.
In particular, in the 1-shot learning setting HUME significantly outperforms proto baseline by 39%, 24% and 44% on the STL-10, CIFAR-10 and CIFAR-100-20 datasets, respectively. Similarly, HUME outperforms linear baseline by 39%, 23% and 37% on the STL-10, CIFAR-10 and CIFAR-100-20 datasets, respectively. In the 2-shot learning setting, HUME outperforms proto baseline by 18%, 7% and 6% on the STL-10, CIFAR-10 and CIFAR-100-20 datasets, and linear baseline by 22%, 8% and 4%, respectively. It is important to emphasize that in this setting these baselines use *20 labeled examples* for STL-10 and CIFAR-10, and *40 labeled examples* for CIFAR-100-20 while HUME works *fully unsupervised*. Only, in the 3-shot learning setting we start seeing the benefits of few-shot baselines but only on the CIFAR-100-20 dataset. In particular, on the STL-10 dataset HUME still outperforms proto and linear baselines by 9% and 11% respectively, and matches performance of these baselines on the CIFAR-10 dataset, and proto and linear baselines outperform HUME by 11% and 13% on CIFAR-100-20 dataset by using 60 labeled examples. On the STL-10 dataset, we observe that HUME keeps outperforming these baselines even in a challenging 5-shot learning setting where these baselines use 50 labeled examples. We would also like to emphasize that obtaining few-labeled examples per class is in reality hard because it requires a strong prior knowledge of knowing the complete space of all possible classes in advance and then labeling a few examples per class. In many real-world tasks, this is impractical and very difficult. We thank the reviewer for this question and we will also include few-shot baselines in our paper. We believe these results further strengthen our work.
Overall, our results show that although being completely unsupervised, HUME (i) outperforms *fully supervised* linear model on top of MOCO representations on the STL-10 dataset and matches the performance on the CIFAR-10 dataset (main paper results); (ii) outperforms challenging *few-shot learning baselines* ran on top of DINO representations in the 1-shot and 2-shot learning scenarios, and (iii) *semi-supervised method FreeMatch* using reliable samples from HUME matches the performance of the FreeMatch with ground truth labels.
[1] Snell et al. Prototypical Networks for Few-shot Learning. NeurIPS 2017.