Multi-label Cluster Discrimination for Visual Representation Learning

Response to Reviewer KbPe

Thanks for your valuable suggestions to improve our paper. We were glad to see your positive ranking at the beginning. Sorry for our late reply due to the time cost of additional experiments. We hope the revised version has solved all of your concerns on this paper. **Weakness1: detailed explanations for the mathematical symbols used in the text .** **A1:** In the revised version, we add detailed explanations for all mathematical symbols used in the equations and text. **Weakness2: In section 3.2, the authors claim efficient parallel computation and scalability of the model training process. However, is there quantitative data to support this point? Does the incorporation of clustering significantly improve the training time?** **A2:** In the revised Appendix A.1, we add a detailed explanation regarding the gradient calculation and we also include time cost comparison on the classification layer. As we can see, the proposed method enables an elegant separation of positive and negative gradient calculation, which can decrease the communication frequency of calling the allreduce operation. To compare the time cost on the classification layer, we train UNICOM, MLC, and MLCD on the LAION-400M dataset with one million classes. We use ViT-B/32 as the backbone, and the final embedding dimension is 512. The batch size is set as $32,800$ and the A100 GPU number is 80 in total distributed across 10 computation nodes. The time cost of the forward and backward steps regarding the backbone is $409$ms. For MLC and MLCD, the positive class number is set as 8. For the single-label classification method, UNICOM, the time cost on the classification layer is $75$ms. For MLC and MLCD, the time cost on the classification layer is $138$ms and $82$ms. The proposed multi-label cluster discrimination method slightly increases the time cost by 9.3% on the classification layer compared to the single-label cluster discrimination method, UNICOM. Compared to the widely used multi-label classification method, the proposed method obviously decreases the time cost by 45.6% on the classification layer. Therefore, the proposed decomposition of contrastive loss ensures efficient parallel computation across multiple GPUs from different computation nodes with minimal communication overhead. With the aid of efficient feature quantization (Johnson et al., 2019), it only takes around 10 minutes to complete the offline clustering step on the large-scale LAION-400M dataset. For multi-label training, there is a slight increase of 9.3% in the time cost of the classification layer compared to the single-label cluster discrimination method. **Weakness3: In section 3.2, you employ some distribution training techniques but details are not provided, which hinders the reproducibility of the work.** **A3:** In the revised Appendix A.1, we add a detailed explanation regarding the gradient calculation and the distribution training. In the supplementary material, we put our training code, which is visible to the public now and can be used for reproducing our method. **Q1: The performance reported in the CLIP paper differs from your reproduced version. In Tab 1 and Tab 2, which may influence the validation of improvement of your model. Have you checked the implementation and settings?** **A1:** Even though CLIP has generously open-sourced the models, they have not provided the evaluation tools. To this end, we contacted the UNICOM [1] authors to get their evaluation toolkit and all test datasets for the task of linear probe. In UNICOM, the authors have developed a GPU-accelerated logistic regression algorithm, facilitating batch-wise assessment of linear probe performance. In this paper, the performance of CLIP is also reported by using this evaluation toolkit. For the task of zero-shot classification, we adopted the same experimental settings as FLIP [2] and referred to the FLIP paper to report the zero-shot results. [1] An, Xiang, et al. Unicom: Universal and Compact Representation Learning for Image Retrieval. ICLR 2023. [2] Li, Yanghao, et al. Scaling language-image pre-training via masking. CVPR 2023.

Response to Reviewer g3TE

Thanks for your insightful suggestions to improve our paper. **Q1: Is it possible that the multiple labels assigned to the same images provide models with additional knowledge about the co-occurrence/relevance of different concepts (cluster-to-cluster relationships) rather than actually telling models what is involved in images (sample-to-cluster relationships)? pre-training on a dataset with known non-overlapping class structure, eg ImageNet, and see if the multi-label clustering still benefits.** **A1:** We added corresponding experiments and visualization in the revised version. | CASE | DATA | 0.5K | 1K | 2K | 4K | 8K | 20K | | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | | UNICOM | IN1K | 42.1 | 58.4 | 61.5 | 62.8 | 62.4 | 61.5| | MLCD | IN1K | 63.2 | 67.2 | 68.2 | **69.9** | 69.7 | 69.0| In Tab. 7 (a), we compare the proposed multi-label cluster discrimination and the single-label cluster discrimination on ImageNet with the clustered class number ranging from 0.5K to 20K. The clustering step is conducted by using the features predicted by the CLIP model. In the discrimination step, both MLCD and UNICOM employ the negative class center sampling with a ratio of $0.1$, and the positive number for MLCD is set as $8$. Pre-training is executed on the ImageNet training dataset by 100 epochs. The model backbone is ResNet-50. The evaluation is undertaken using a linear probe on the ImageNet validation set. As we can see, the proposed multi-label learning significantly surpasses UNICOM and achieves the best performance of 69.9% when the class number is 4K, which is four times of the true class number of ImageNet. In Fig. 5 of the Appendix, we visualize the top three labels for our training samples. When training with multiple labels, our method can not only capture the label correlations (e.g., paintbrush and oil painting, shield and armour, ladle and frying pan) but also learn complementary visual signals (e.g., different breeds of dogs, different locations of figs) to improve visual representation learning. **Q2: The modifications made to triplet loss make sense to me but their effects are unclear. How will the proposed model perform if all its designs are kept unchanged except for replacing L’_MLCD (Eq.6) with L_MLC.** **A2:** In the revised version, we add the experiments to verify the effectiveness of the proposed MLCD. |CASE | DATA | Finetune | Linear Probe | Zero Shot| | ------ | ------ | ------ | ------ | ------ | |MLC | LAION-400M| 80.9 | 76.9 | 63.9 | |MLCD | LAION-400M| 81.2 | 78.1 | 64.5 | Tab. 7 (a), we compare the performance of the vanilla MLC and the proposed MLCD on the ImageNet validation dataset. Pre-training is executed on the LAION-400M dataset by 32 epochs. The model backbone is ViT-B/32. Both MLC and MLCD employ the negative class center sampling with a ratio of $0.1$. MLCD outperforms MLC in all three settings: fine-tuning, linear classification, and zero-shot, confirming the effectiveness of the two additional optimization targets. In Appendix. A.1, we compare their gradient calculation and time cost on the classification layer. The proposed contrastive loss decomposition can significantly decrease the communication cost, facilitating distributed training on large-scale training data. **Q3: What are the blue and green cells standing for?** **A3:** For Fig. 1(a), blue cells stand for the similarity scores of positive image-text pairs. The similarities in the blue cells need to be increased during training. For Fig. 1(b), there are two training samples in the minibatch. The blue and green cells stand for the positive scores in their corresponding one-hot target logits. For Fig. 1(c), there are also two training samples in the minibatch. However, each sample has three positive labels. Thus, the target logits contain three positive similarity scores that need to be increased during training. **Q4: Whilst Fig.2 is the first figure being referred to, Fig.1 is simply mentioned as the illustration of visual representation learning but it lacks further explanation/discussion.** **A4:** In the revised introduction section, we refer to Fig. 1 first when we introduce instance discrimination methods and cluster discrimination methods. In the revised preliminary section, we add detailed explanations of Fig. 1 (a) and Fig. 1 (b). In Section 3.2, we also refer to Fig. 1 (c) when we introduce our method. **Q5: In Eq.1, I assume the pairwise similarity is cosine similarity if following CLIP, but without normalization of features, it is just an inner product. So I’m wondering if it is a mistake or my misunderstanding.** **A5:** We employed the normalization to all embedding features, and the pair-wise similarity is the cosine similarity. In the revised version, we added the keyword of normalization to features around Eq. (1).

Response to the weaknesses mentioned by Reviewer S2Uc

Thanks for your valuable comments and suggestions to improve this paper. **Weakness1: novelty and originality.** **A1:** In the instance discrimination methods (e.g., CLIP), each image-text pair represents one unique class. CLIP can provide richer forms of labels for a single image, e.g., objects, scenes, actions, and relations, at multiple levels of granularity. However, negative pairs that share similar semantics will be undesirably pushed apart in the embedding space. In the cluster discrimination methods (e.g., UNICOM), visually similar instances are pulled together by the classification step. However, UNICOM only defines a single pseudo-label for each image. To the best of our knowledge, we are the first to propose multi-label learning on the large-scale dataset (LAION-400M). The proposed multi-label cluster discrimination approach can not only capture the semantic structures in the data but also support the learning of multiple granularity of labels for a single image. In addition, we introduce a novel decomposition of contrastive loss, to avoid ambiguity during the optimization of $(s_j-s_i)$ as well as decrease the communication cost during distributed training. **Weakness2: moderate improvement.** **A2:** In this paper, we mainly compared our method with the latest state-of-the-art methods (e.g., UNICOM [1] and FLIP [2]). In the task of linear probe, our method outperforms UNICOM by 1.3% (Tab. 1). In the task of zero-shot classification, our method surpasses FLIP by 1.5% (Tab. 2). Vision representation learning is one of the most competitive research topics, and our improvement compared to this year's publications is indeed significant. [1] An, Xiang, et al. Unicom: Universal and Compact Representation Learning for Image Retrieval. ICLR 2023. [2] Li, Yanghao, et al. Scaling language-image pre-training via masking. CVPR 2023. **Weakness3: The proposed method is developed upon the feature embedding from the pre-trained CLIP model and it does not involve any textual information in the proposed multiple label loss.** **A3:** In this paper, we focus on improving visual representation by introducing multi-label cluster discrimination. Our method does not involve text learning. In Fig. 1, we compare our method with CLIP and UNICOM. CLIP consists of the image encoder and text encoder, while UNICOM and the proposed method only contain the image encoder. In Section 4.1, we mention how to train an additional text encoder following Locked-image Tuning (LiT)[3], which teaches a text model to read out good representations from a locked image model for zero-shot classification and image-text retrieval tasks. "To assess the performance of zero-shot classification and zero-shot image-text retrieval tasks, we employ contrastive learning to train a text encoder from scratch for 32 epochs with a frozen visual encoder following LiT (Zhai et al., 2022b). The structure of the text encoder is also identical to CLIP. " [3]Zhai, Xiaohua, et al. Lit: Zero-shot transfer with locked-image text tuning. CVPR 2022. **Weakness4: In Equation 5, two new items are further introduced into the multi-label loss. This is regarded as one of the key contributions by this paper. However, its efficacy does not seem to be clearly verified in the ablation study.** **A4:** In the revised version, we add the experiments to verify the effectiveness of the proposed MLCD. |CASE | DATA | Finetune | Linear Probe | Zero Shot| | ------ | ------ | ------ | ------ | ------ | |MLC | LAION-400M | 80.9 | 76.9 | 63.9 | |MLCD | LAION-400M | 81.2 | 78.1 | 64.5 | Tab. 7 (a), we compare the performance of the vanilla MLC and the proposed MLCD on the ImageNet validation dataset. Pre-training is executed on the LAION-400M dataset by 32 epochs. The model backbone is ViT-B/32. Both MLC and MLCD employ the negative class center sampling with a ratio of $0.1$. MLCD outperforms MLC in all three settings: fine-tuning, linear classification, and zero-shot, confirming the effectiveness of the two additional optimization targets. In Appendix. A.1, we compare their gradient calculation and time cost on the classification layer. The proposed contrastive loss decomposition can significantly decrease the communication cost, facilitating distributed training on large-scale training data.

Response to the questions arised by Reviewer S2Uc

**Q1: Please explain and experimentally demonstrate how this modification can facilitate distributed learning more clearly. “Eq. 6 is able to distribute the weights associated with one million class centers across all GPUs with minimal communication overhead.” Why?** **A1:** In the revised version (Appendix A.1), we add a detailed explanation regarding the gradient calculation and we also include a time cost comparison on the classification layer. As we can see, the proposed method enables an elegant separation of positive and negative gradient calculation, which can decrease the communication frequency of calling the allreduce operation. To compare the time cost on the classification layer, we train UNICOM, MLC, and MLCD on the LAION-400M dataset with one million classes. We use ViT-B/32 as the backbone, and the final embedding dimension is 512. The batch size is set as $32,800$ and the A100 GPU number is 80 in total distributed across 10 computation nodes. The time cost of the forward and backward steps regarding the backbone is $409$ms. For MLC and MLCD, the positive class number is set as 8. For the single-label classification method, UNICOM, the time cost on the classification layer is $75$ms. For MLC and MLCD, the time cost on the classification layer is $138$ms and $82$ms. The proposed multi-label cluster discrimination method slightly increases the time cost by 9.3% on the classification layer compared to the single-label cluster discrimination method, UNICOM. Compared to the widely used multi-label classification method, the proposed method obviously decreases the time cost by 45.6% on the classification layer. Therefore, the proposed decomposition of contrastive loss ensures efficient parallel computation across multiple GPUs from different computation nodes with minimal communication overhead. **Q2: Some technical details are missing, e.g., in Section 4.1. the number of classes (k) and number of positive centers (l).** **A2:** In the revised version, we add these technical details in Section 4.1. "In the following experiments, unless otherwise specified, the model used is ViT-L/14, the number of classes ($k$) is one million, and the number of positive labels ($l$) assigned to each image is $8$." **Q3: Table 3, all the best results need to be highlighted.** **A3:** We fixed this in the revision, and we also checked other tables. **Q4: the meaning of the y-axis of the charts should be provided to improve the clarity** **A4:** In Fig. 3, we visualize per-dataset differences in the performance. The Y-axis shows the performance difference. Most of the evaluation metrics are accuracy as shown in Tab. 10 (Appendix). Green bars indicate our model outperforms the baselines, while the orange bars depict our model is surpassed by the baselines. Our model outperforms UNICOM on 23 datasets on the task of linear probe and surpasses FLIP on 15 datasets on the task of zero-shot classification. In the revised version, we added a detailed explanation in the caption of Fig. 3.