No Representation Rules Them All in Category Discovery

Summary

This paper works on generalized category discovery, a setting that requires classifying unlabelled samples according to the taxonomy defined by the labeled set. This work identifies that a shortcut exists in previous benchmarks, and methods overlooking the classification metric defined by the labeled set could still perform well. It thus proposes the CLEVR-4 benchmark, in which the taxonomy/isolation of categories is separated as 'shape', 'texture' or 'color', or 'count'. Based on this benchmark, it evaluates the performance of previous methods in terms of pre-training/representation and pseudo-labeling and proposes $\mu$GCD. The technique shows better performance in both the proposed benchmark and prior benchmarks.

Strengths

*Originality & significance.* The proposed benchmark is absolutely novel and original. It identifies a key issue in current GCD benchmarks: unsupervised classification algorithms are generally specialized for semantics, and the task could be easily suited without relying on the labeled set. The benchmark is in line with a recent trend of aligning machine outputs with human requirements: what is the definition needed by humans? I am happy to see this benchmark and the revisiting of previous works that it enables. *Quality.* The benchmark is accompanied by careful evaluations of different aspects of previous methods (representation & classification), and analytical figures (fig. 3/6/7) that help understand the source of improvement. Ablation study and error bars are also provided, which is good and makes the contribution sound. *Clarity* The presentation is natural and fluent, and I did not find issues in delivery. *Reproducibility.* Technical details are well provided to facilitate reproduction. Hope to see the dataset soon.

Weaknesses

I have to say that my major concerns are anticipated and well discussed in the appendix. There is one small concern on methodology and I list it in the next section.

Questions

One concern on methodology is whether $\mu$GCD could be listed as a major contribution. It is somehow technically incremental to me and is more like SimGCD Plus (momentum teacher & weak-strong aug) rather than a new method. I understand it suits the proposed benchmark better, and I feel okay with the overall contribution of this paper, the only concern is whether could it be listed as a side-by-side contribution with the benchmark.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

have been discussed

Summary

This paper contributes one new dataset and one new method for category discovery. The dataset is designed in a way that each image can be clustered into 4 different clustering based on the attribute of the image (shape, count, texture, and color), this design can be used to reveal the difficulty of unsupervised clustering and the necessity of category discovery methods. The proposed method extends on the SimGCD method by adding an EMA teacher and strong-weak augmentations.

Strengths

1. I really like the discussion of the limitation of unsupervised clustering method on self-supervised representations. 2. The design of the Clevr-4 dataset is interesting, and could be a direction for future works.

Weaknesses

1. I would like to note a few related fields, (1) deep multiple clustering [R1, R2] which deals with unsupervised clustering for different clustering criteria, and (2) conditional similarity [R3, R4] 2. The proposed method seems to be a strighforward extension of the SimGCD method, I think it is not significant enough to be a claimed contribution. 3. I think the discussion on the experiment of previous method on Clevr-4 is not comprehensive, as mentioned in line 262-269, augmentations seem to play an important role for category discovery, then I think the discussion on Clevr-4 should contain some experiments of using different augmentation for differe splits. [R1] AugDMC: Data Augmentation Guided Deep Multiple Clustering, https://arxiv.org/abs/2306.13023 [R2] A Diversified Attention Model for Interpretable Multiple Clusterings, TKDE2022 [R3] Learning Similarity Conditions Without Explicit Supervision, CVPR 2019 [R4] Towards Latent Attribute Discovery From Triplet Similarities, ICCV 2019

Questions

1. I am interested in how would applying semi-supervised k-means algorithm on pretrained models in tab 2 performs, because using the semi-supervised k-means is IMO the easiest way to inject certain clustering criteria to the model. 2. Since the proposed method is similar to SimGCD, then how about applying the strong augmentation to SimGCD or GCD and compare with that performance? It seems that the strong augmentation plays an important role for the performance as shown in line 2 of tab 6. 3. I am also interested in the performance of tuning the augmentations used by SimGCD and GCD method on Clevr-4. As `misaligned augmentations degrade performance`, yet there are no experiment supporting this claim.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I think the methodology contribution of this paper is limited, the proposed method seems like a combination of FixMatch and SimGCD. The proposed Clevr-4 might be of interest to the community, but as I discussed in the weakness and questions section, I think currently the discussion on Clevr-4 is not comprehensive enough.

Summary

1. This paper tackles the problem of generalized category discovery (GCD) and identifies the drawbacks of existing methods that they are verified only with labels for a single clustering of the data. In such a case, the model may simply perform unsupervised clustering, not correctly using the available labels. 2. A synthetic dataset, named “Clevr-4” is proposed which contains four independent taxonomies, i.e., shape, texture, color or count. 3.The authors demonstrate the limitations of unsupervised clustering in the GCD setting and propose a new method, µGCD, based on mean teachers, which outperforms existing baselines on Clevr-4 and sets a new state-of-the-art on the Semantic Shift Benchmark (SSB).

Strengths

1. The motivation of this paper is clear and interesting. The authors propose that existing GCD methods may simply perform unsupervised clustering based on the natural grouping of the data. The issue is overlooked by existing methods and is indeed worth investigating. 2. A new benchmark is proposed which provides four independent sets of labels for a common dataset, which is valuable to extensively verify GCD methods’ ability to extrapolate the taxonomies specified by the labeled data using different pre-trained feature extractors.

Weaknesses

1. Additional recent methods should be compared: Pu N, Zhong Z, Sebe N. Dynamic Conceptional Contrastive Learning for Generalized Category Discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 7579-7588. 2. Fig.2 is misleading. Why there are three 'student predictions'? Does the arrows between them mean that the three losses act sequentially, rather than all together? 3. Since the proposed method adopts a two-stage training strategy, i.e., the proposed method could be regarded as an additional 100 epochs of training based on GCD, the training cost could become a concern.

Questions

1. Additional ablation study on lambda_1 which is used to balance the supervised and unsupervised loss terms, should be added to verify the hyperparameter sensitivity.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The motivation of verifying GCD methods on taxonomies not merely on the semantic labels is reasonable. The experiments and analysis are comprehensive. Limitations of the method would be limited technical novelty. Codes are not provided.

Summary

This paper addresses generalized class discovery (GCD) from a unique perspective. The authors argues that current GCD benchmarks are unable to ascertain whether models are using the available labels to solve the GCD task, or simply solving an unsupervised clustering problem. In light of this, this paper introduces a new dataset where the data can be clustered according to different rules, i.e., object count, shape or texture. This paper also proposes a simple method that effectively solves such a problem.

Strengths

- The motivation of this paper is sound. - The paper is well-written and easy to follow. - The contribution is convincing.

Weaknesses

- The introduced dataset is focused on synthetic data. It could be more convincing to evaluate on real-world data. - The proposed method is simple yet powerful, but it's unclear which mechanism solves the GCD problem when multiple clustering rules are available.

Questions

- The proposed is a simple reinvent of existing techniques that are common in semi- and self-supervised literature. As mentioned above, the question is how does such a simple method solves the proposed GCD problem when different clustering rules are acceptable? Thank the authors for the rebuttal, which addressed most of my concerns. Please include the modifications as mentioned in the discussion in the final version.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations have been carefully discussed in the paper.

Thanks for the response.

**1. Results of semi-supervised $k$-means:** Thanks for performing the experiments, I think this could be a valuable add-on for the paper. **2. Results on using aligned augmentations:** I think given the performance boost of using an aligned augmentation, this should definitely be mentioned in the main paper. As a reader of the paper, my first thought on the different taxonomy of categories is that augmentations could play a key role here. I would like to also note a few references on how augmentations influence the learned representations [a,b]. Adding these discussions would make the paper stronger. I would encourage the author to include these discussions, as they further complete the paper. Given that the response resolves my concern, and these changes to the paper is easy to make, I would remain positive about this paper. [a] Amortised Invariance Learning for Contrastive Self-Supervision, ICLR 2023. [b] Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks.

Global response after discussion period

After the discussion period, we are encouraged that the reviewers ‘remain positive’ about the paper (dGiY, aYTM, s1RT) and that we were able to ‘resolve the concern’ of the reviewer dGiY. We will incorporate the following to the manuscript: * Discussion of related work (**dGiY**) on: deep multiple clustering (DMC); conditional similarity (CS); and the study of augmentations in self-supervised learning (SSL). * Analysis on: the results with mis-aligned augmentations, and with semi-supervised $k$-means (**dGiY**, shown in Tab R1 and Fig R2); effect of the hyper-parameter $\lambda_1$ (**s1RT**, shown in Fig R1); and an exploration of how $\mu$GCD design choices effect how different taxonomies are learned (specifically with further feature space visualizations as in Fig 6, **V8PZ**). Furthermore, we will **clarify the scope** of our contributions (**dGiY**, **aYTM**). In this work, we have proposed a *new synthetic dataset* for controlled exploration of the category discovery problem, allowing evaluation of *multiple clusterings* of the same images. This is related to (and may find broader applicability in) the DMC, CS and SSL literature (reviewer **dGiY**). We have also *developed an effective algorithm for GCD*. This method *extends SimGCD* with components from the semi-supervised learning literature (as noted by **aYTM**). We note that, similarly to papers in related fields (e.g SimCLR, UNO, SimGCD), our methodological contribution is to design a *performant recipe* for GCD (Appendix L297), which we find outperforms SoTA on the introduced dataset (+4%) and on the SSB suite (+3%).