Representation Learning via Consistent Assignment of Views over Random Partitions

Reply to rebuttal comments 1/2

> Since CARL does not have the results on ImageNet, is the parameter of CARL tuned for ImageNet. As the reviewer mentioned, CARL was not trained on ImageNet. We did the hyperparameter search for the number of prototypes due to our experience executing CARL. When CARL was trained with 10000 prototypes, training collapsed. We kept the entropy weight as 2 (following CARL’s guidelines for other datasets) and reduced the number of prototypes to 3000 (where we observed no collapse). To fully address the reviewer’s concerns, we pre-trained multiple setups of CARL on ImageNet for 100 epochs. We fixed the number of prototypes as 10000 and ablated different values of lambda (entropy weight). Results are below. \begin{array} {rrrrrr} \hline \lambda &1&2&3&4&5\\\\ \hline CARL&C&C&65.1&64.8&63.9\\\\ \hline \end{array} As discussed, using 10000 prototypes causes collapse (**C**), and the workaround is to increase the entropy weight. For reference, CARP 100 epoch model with 2 views achieves 69.7%. As discussed in Section 3.1, learning that many prototypes at once with the entropy term, as in CARL, leads to underperformance. > Table 1 [...] CARL shows that it works well with 10,000 prototypes on CIFAR-100. Therefore, 10,000 prototypes should not be a big issue on ImageNet. We respectfully disagree with the reviewer on this point. The training of CARL on a much simpler dataset with only 100 classes cannot be extrapolated to a more complex dataset with 1000 classes (10 times more). Moreover, CARL’s paper discusses the relation of the number of prototypes and the number of classes (Fig. 3 in CARL’s paper) which is in line with the observed behavior in our experiments. **In CARP, we show that there is a lack of generalization in the usage of prototypes and propose a solution.** > [...] it is better to have CARP on CIFAR-100 for a fair comparison. The results comparing CARP and CARL on CIFAR-100 are below. \begin{array}{rrrrrrr} \hline & Cifar10&&Cifar100&&STL10&\\\\ \hline Ep. &100&200&100&200&100&200\\\\ \hline CARL&73.39&78.94&42.91&48.85&76.9&81.95\\\\ CARP&74.84&79.52&44.67&50.64&78.05&82.44\\\\ \hline \end{array} We pre-trained CARP on CIFAR-10/100 and STL-10 following the CARL’s guidelines (Table 2 on CARL’s paper). We report average top-1 (linear probing) results across 3 independent runs (same as CARL). The number of prototypes was set to 100, 300, and 300 for Cifar-10,-100, and STL-10, respectively (same as CARL), and the partition size was set to 50 for all datasets (no tuning was done to select this partition size). We report models trained for 100 and 200 epochs. CARP outperforms CARL in all datasets. These experiments showed that CARP works well for simpler datasets as well as in ImageNet. By default, CARP uses $\lambda=1$ while CARL uses larger values to avoid collapse, e.g., $\lambda$=2, which explains CARP’s improvements. > Moreover, the proposed method eliminates the parameter for the entropy while introducing an additional parameter for the partition size, which may not save tuning efforts. We agree with the reviewer that our method introduces a new hyperparameter (the partition size) and avoids the tuning of the entropy weight. However, **the entropy weight and the partition size have different tuning difficulties**. As discussed in Section 3.1, without the random partition pretext task, training collapses if the entropy term is too small, and accuracy is suboptimal if the entropy term is too high. On the other hand, CARP is robust to the choice of partition size, as shown in Table B.2 (appendix). > [...] Compared with CARP with multi-crop augmentation as in Table 1 in the submission, the gap [with CoKe] is quite marginal, i.e., 0.3% on Avg@k. A fair comparison is essential to draw any conclusion. As pointed out by the reviewer, when comparing CoKe with CARP with multi-crop, CARP still outperforms CoKe by a small margin (even though CoKe was trained for 800 epochs and CARP for 400). To clear the reviewer's concerns, we extended the previous comparison against CoKe to include all instances of CARP and an additional instance of CoKe trained for 800 epochs w/o multi-crop. CARP consistently outperforms CoKe by large margins. \begin{array}{lrcrrrrrrrrrrr} \hline Method&Ep&Pets&Flowers&Aircraft&Cars&Country&Food&STL&GTSRB&Avg @k&&&\\\\ &&&&&&&&&&10&20&100&200\\\\ \hline CoKe &1000&81.3&75.3&29.3&22.6&13.3&60&95.7&64.2&55&55.2&54.1&53.4 \\\\ CoKe (mc) &800&79.5&79.5&27&22.4&14.6&58.9&95.7&60.4&57.1&57.4&57.1&56.3\\\\ CARP&200&86.8&78.2&38.9&29.8&12.2&58.4&95.5&73.7&59.2&59.2&58.5&57.9\\\\ &400&86.8&80&\textbf{42.1}&\textbf{33.5}&12.3&58.4&95.9&75.3&60.4&60.5&59.7&59.2\\\\ &800&\textbf{87.3}&\textbf{81.2}&41.1&33.2&13.6&61.2&\textbf{97}&\textbf{76.4}&\textbf{61.2}&\textbf{61.4}&\textbf{60.4}&\textbf{59.7}\\\\ CARP (mc)&200&78.7&79.7&35&26.6&\textbf{14.5}&61.8&95.5&64.7&57.1&57.1&55.9&55\\\\ &400&83.9&80.3&34.8&27.1&14.2&\textbf{62.9}&95.5&62.8&57.6&57.7&56.8&56\\\\ \hline \end{array}

Reply to rebuttal comments 2/2

> [...] the phenomenon for the degenerated performance from the multi-crop augmentation is interesting and worth an additional discussion. We agree with this observation. We expect to study it further in the future. As it is now, this understanding is out of the scope of this work. > [...] For example, CoKe reports 76.6% NMI on CIFAR-10 as a clustering method, which is much better than the result in this work. [...] We highlight that CoKe’s results on CIFAR-10 were reported with an inter-dataset setup, meaning that it was trained and evaluated on CIFAR-10. In our experiments, CARP was pre-trained on the ImageNet dataset and evaluated on CIFAR-10 (transfer learning). The latter is a more challenging setup that evaluates the generalization and robustness of the learned representations. This explains the difference observed by the reviewer. > [...] The setting of clustering in this work can be highlighted in the caption of Table 2 to avoid misunderstanding. We reported the evaluation protocol in Appendix D.4 (as mentioned in Section 5.2). In addition, we will improve the caption in Table 2 to better convey the evaluation protocol.. > Unlike [7,8] that have different temperatures for pseudo labels and predictions, CoKe has the same temperature and the label is not sharpened. [...] We do not understand the reviewer’s comment given that CoKe clearly uses the temperature. First, all sets of variables are sharpened here: github.com/idstcv/CoKe/blob/main/coke/builder_double_view.py#L98C1-L102C76. Then, used the in the softmax here: github.com/idstcv/CoKe/blob/main/main_double_view.py#L321-L323. Then, in the loss function here: github.com/idstcv/CoKe/blob/main/main_double_view.py#L330C1-L337C97. We stated the differences between CARP’s and CoKe’s loss functions and the comparison against other methods, like SwAV [7] and DINO [8]. We would appreciate it if the reviewer could provide additional information about the concern raised in this question. > [...] While this work does not have the temperature, the statement can be more accurate. We will review our manuscript to improve the accuracy of how the new pretext task produces the sharpening and its relation with the temperature. As discussed in the other reviews, we will include the empirical evidence that the random partition process sharpens the prediction by design, as shown in the PDF rebuttal document. We hope to have addressed all the open questions about our work.

Reply to rebuttal comments

> [...] about the motivation: when and why do we need so many prototypes? **(Why)** Our requirement for more prototypes stems from their ability to leverage local information derived from our learning objective (unsupervised local view agreement). This local data is insufficient to grasp coarser (global) clusters, but it effectively captures changes within the data manifold, as evidenced by our findings. Based on our empirical experiments and intuitive understanding of our method, here's a breakdown of our earlier point: With fewer prototypes, each one must encompass a larger set of images exhibiting substantial variations. Consequently, these prototypes must encapsulate more abstract concepts to maintain coherence among the represented images. Conversely, when the number of prototypes matches the training set's image count, the clusters become specific representations, resembling a k Nearest Neighbor problem. Our experiments revolve around investigating how the learned representations behave as we alter the prototype numbers (while keeping them below the sample count). Due to the localized nature of our learning objective, operating on local views of the same image, learning more abstract representations becomes more challenging. To alleviate this challenge, we increase the prototype count. This adjustment lessens the demand for prototypes to grasp intricate abstractions, allowing them to focus on specific characteristics sufficient to represent the smaller set of images with minor variations. This over-clustering of the data space effectively harnesses our sole source of information: local information. The large number of prototypes, coupled with a stochastic random partitioning process, induces a noise-filtering mechanism that enhances representation robustness, as our experiments illustrate. **(When)** We aim to overparameterize within the bounds of having more prototypes than or equal to the true class count and fewer prototypes than the data points in the training set. In the unsupervised scenario, the class count remains unknown. Thus, our task involves exploring the dataset to strike the right balance among prototype count, the unknown class quantity, and the sample number. This process resembles k-means clustering, where data exploration guides the identification of optimal cluster patterns. Setting up fewer prototypes than true classes results in prototypes acting as higher-level categories. Conversely, matching the prototype and class counts yields generalized representations that remain consistent despite class variations. Exceeding true class numbers leads to more specific representations capturing local changes within the data manifold. > [...] for [datasets as big as ImageNet] the techniques proposed in this work are also not necessary as agreed by the authors We highlight that the **main point is not about the number of prototypes to be as high as possible**, but rather to **have a number high enough that allows us to exploit the locality of the views** (through the learning objective) **while supporting the number of classes** (and their complexity). We clarify that **we don’t mean** (in our rebuttal) **that the high number of prototypes is not necessary to obtain good performance** (as mentioned by the reviewer in this comment). But rather, we mean that **there is a trade-off** between the number of prototypes and classes (and their complexity through their multi-modality) **that must be understood** and considered to select the optimal number of prototypes that will ideally represent each class. However, in our unsupervised setup (and in real-world scenarios where the data is complex and hard to define), this number is not so easy to define and requires experimental evidence to set. In addition, we show in our experiments (Table B.1 appendix) that CARP is robust to the number of prototypes, and the difference in performance from 65k to 4k prototypes is marginal, which shows CARP's capacity to learn more abstract or more specific prototype representations. We hypothesize that this minimal change is related to the information we get from our learning objective. However, as we decrease the number of prototypes even more (< 4k), the learning problem becomes more complex because the prototypes require a higher level of abstraction, and our self-supervised objective does not provide a sufficient level of information. Consequently, the performance decreases. The understanding of this trade-off and to automatically find ways of keeping the relevant prototypes is future work that we intend to tackle. However, it is out of the scope of our current proposal. Nevertheless, we agree with the reviewer about its relevance. > Hope to see the motivation validated in larger-scale datasets. We intend to evaluate our method on larger datasets for our future work. However, to provide such results during the rebuttal is impossible, given our resources.

About DINO v2 number of prototypes.

Based on the following evidence from Dino v2 repository, we would like to emphasize that DINO v2 uses **65536 prototypes for ImageNet-1K pre-training** (github.com/facebookresearch/dinov2/blob/main/dinov2/configs/ssl_default_config.yaml#L45), not 4096 as previously stated. Moreover, to pre-train on even larger scale datasets such as ImageNet 22k (github.com/facebookresearch/dinov2/blob/main/dinov2/configs/train/vitg14.yaml#L2), **DINO v2 uses 131072 prototypes**. These numbers are aligned with our experimental results and intuitive explanation in the comment above.

Thanks to reviewer's work

We thank the reviewer for the relevant work reviewing our manuscript and for the final position towards acceptance of our work.

Thanks to reviewer's work

We thank the reviewer for the fair assessment, valuable insights, and suggestions for our work. We will incorporate the relevant suggestions into the final version.