We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass
Paper
Similar papers
Peer review
Summary
The paper proposes a simple alternative to CLIP-style pretraining that doesn't require a text encoder and can be done using only a text tokenizer. The goal is to provide a simpler yet more efficient alternative to vision-language model (VLM) pretraining, which is known to be very expensive. Additionally, the authors simplify the VLM pertaining setup into a classification task which is both novel and very intuitive. The authors demonstrate the efficiency of their method through experiments on both classification and vision-and-language downstream tasks. Additionally, they present a comprehensive set of ablations to dissect the performance gains of their proposed approach. While there are some limitations to their proposed method, the simplification offers valuable benefits for traditional classification and vision-language tasks.
Strengths
1. The paper is well-written and easy to follow, with clear motivation and well-planned experiments. 2. The proposed method simplifies vision-language pretraining by eliminating the need for a text encoder, using a text tokenizer instead. This approach demonstrates comparable or better performance on both classification and vision-language tasks. It effectively transforms the contrastive pretraining task into an open vocabulary classification task, where text tokens provide supervision. This insight is powerful as it can help the community develop strong vision encoders more efficiently. 3. The experiments and ablations presented are thorough. The authors systematically analyze every component of their method, including data, model scale, and the tokenizer. 4. The paper also highlights that for simple classification tasks or vision question-answering tasks, CLIP-like pretraining is not necessary for learning strong, robust vision encoders. This is a novel insight of the paper.
Weaknesses
1. For classification tasks, the authors only present ImageNet-1K classification accuracy and do not perform experiments on other popular few-shot and zero-shot classification benchmarks. Including these benchmarks would have strengthened the paper. 2. Vision-language models like CLIP are also used in text-to-image generation systems. The authors do not address this aspect or present any experiments showcasing the effects of using their encoder in such systems. 3. This work shares similarities with research that cleans pretraining captions before performing CLIP-style pretraining [1]. However, the authors do not compare their method against such approaches. I believe this comparison is important, as synthetic captions are increasingly used for CLIP-style pretraining. 4. There is a small reference error in line 235. [1] Fan, Lijie, et al. "Improving clip training with language rewrites." Advances in Neural Information Processing Systems 36 (2024).
Questions
1. In Table 7, the authors show that removing stop-words has no effect and state in line 279 that "keeping stopwords could help the vision encoder." This statement is not explained and seems counter-intuitive. Could the authors elaborate on this observation? 2. The authors claim they can use IDF weights in an online manner. Could the authors explain how they achieve this? I don't understand how this process can be done online. 3. The authors demonstrate that the performance of SuperClass is as good as or better than traditional CLIP-style pretraining on ImageNet-1K classification. However, it would be interesting if the authors could specify which classes benefit more from their method compared to traditional CLIP. Such an analysis could help understand where the gains stem from.
Rating
7
Confidence
5
Soundness
4
Presentation
4
Contribution
3
Limitations
1. The authors acknowledge that their approach completely ignores word order, which can significantly impact the encoder's ability to understand tasks requiring spatial reasoning. 2. Additionally, this approach may not be suitable for text-to-image systems, as these models need to understand word order and infer relationships from text to generate accurate images.
Rebuttal by Authors Part 2
> **Q1**. The effect of removing stop-words In Table 7, removing stop-words leads to a decrease in classification accuracy, for example, the linear probing accuracy decreases from 76.0 to 75.7 (-0.3), and the zero-shot accuracy reduces from 61.7 to 61.0 (-0.7). Therefore, we conclude that stopwords could help the vision encoder. This might be explained by the fact that stop-words also carry some useful visual information. For example, 'she', 'her', and 'him' can indicate a person's gender; 'on', 'off', 'above', 'below', 'up', and 'down' can indicate operational status or position; '@' is likely to indicate an email address. Stop words and punctuation are listed below. `['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]` `{'{', '+', '(', '$', '}', '!', '%', '\\', '<', ';', '|', ']', '"', "'", ',', '&', '=', ')', '_', '^', '~', '#', '@', '.', '[', '*', '?', ':', '/', '>', '-'}` > **Q2**. Online IDF **A2**. To implement online IDF, we set up two global variables to track the number of seen samples (N) and the occurrence count of each subword. As training proceeds, these values will gradually approach the true word frequency of the entire dataset. The advantage of online IDF is its ease of use, allowing the code to be directly transferred to new datasets. We also conducted experiments to compare recognition accuracy using online IDF and offline IDF. The zero-shot accuracy with online IDF drops by about 0.3, and the linear probing accuracy drops by about 0.1. > **Q3**. The analysis of classification results **A3**. We evaluated the performance of CLIP and Superclass on the 1000 classes in ImageNet. Among them, Superclass performs better in 502 classes, while CLIP outperforms in 270 classes, and they perform equally well in the remaining classes. We also extracted the top 10 classes with the largest performance differences and displayed them in the table below. We do not find any obvious patterns, but it is possible that Superclass and CLIP can complement each other. This is an interesting direction for future research. | Class name | muzzle | music speaker | yellow garden spider | cardboard box / carton | Carolina anole | black-footed ferret | monitor | missile | cricket insect | garter snake | |------------|--------|---------------|----------------------|------------------------|----------------|---------------------|---------|---------|----------------|--------------| | CLIP | 0.72 | 0.68 | 0.82 | 0.66 | 0.66 | 0.62 | 0.36 | 0.44 | 0.58 | 0.82 | | SuperClass | 0.88 | 0.84 | 0.96 | 0.80 | 0.81 | 0.76 | 0.50 | 0.58 | 0.70 | 0.94 | | Class name | European polecat | oxygen mask | parallel bars | ox | promontory | English Setter | stethoscope | split-rail fence | rotisserie | cassette player | |------------|------------------|-------------|---------------|------|------------|----------------|-------------|------------------|------------|-----------------| | CLIP | 0.64 | 0.78 | 0.82 | 0.68 | 0.62 | 0.92 | 0.94 | 0.88 | 0.98 | 0.48 | | SuperClass | 0.44 | 0.64 | 0.70 | 0.58 | 0.52 | 0.82 | 0.84 | 0.78 | 0.88 | 0.38 |
At its core, this paper introduces a new way to pre-train open-world Vision-Language foundational models, it's simple and elegant and seems to have maintained the zero-shot capabilities that CLIP offers. But I believe that using such a model might not be suited for generation, where one may want a text encoder to infer the relationships between words, i.e. "A dog over a cat" vs "A cat over a dog". Additionally, I believe that such a technique can also be used to clean the pertaining data of VLMs, a problem that LAION has faced in the past [1]. [1] https://cyber.fsi.stanford.edu/news/investigation-finds-ai-image-generation-models-trained-child-abuse Thanks for the comprehensive answers to my questions, I will increase the score for soundness in my review.
Thank you for your insightful feedback and for recognizing the strengths of our paper. We appreciate your thoughtful comments on the potential applications and limitations of our model. Your suggestions provide valuable directions for future work. Thank you again for your comprehensive review and for increasing the score. Your support and recognition are greatly appreciated.
Summary
This paper proposes a simple classification-based vision-language pretraining method. The proposed SuperClass approach directly uses an off-the-shelf subword-level tokenizer to obtain the classification labels from raw text, without requiring any preprocessing. Then, the vision encoder is trained by optimizing the multi-label softmax loss, with Inverse Document Frequency (IDF) as the weight of each label. SuperClass achieves promising performance on classification and various vision-language tasks. Ablation experiments are conducted to validate the impact of different design choices.
Strengths
1. The proposed method in this paper reveals the potential of classification in vision-language pretraining, which provides empirical evidence for further researching. 2. With a simple framework that requires no data preprocessing, SuperClass enables large-scale training on paired image-text data. Compared to previous classification-based pretraining methods, the proposed approach demonstrates greater practical applicability. 3. SuperClass is training-efficient by removing the need for a text encoder, and the extensive experimental results demonstrate the effectiveness and scalability of the proposed approach.
Weaknesses
1. The robustness of the proposed method to different model types remains unclear. All the experiments in this paper use ViT as the vision encoder, and there is no evidence to demonstrate the effectiveness of the SuperClass on other encoder architectures, such as ResNet. 2. The ablation experiment on different classification losses is only conducted on classification tasks, using a ViT-B/16 backbone and 512M seen data samples. It is important to demonstrate the robustness of the softmax loss on a broader range of vision-language tasks, beyond just classification. Furthermore, the impact of the choice of loss function on the scalability of the proposed method is not discussed.
Questions
Please refer to the weaknesses.
Rating
6
Confidence
3
Soundness
2
Presentation
3
Contribution
3
Limitations
The authors discuss some of the limitations of their work in Section 5. But I would like them to consider some of my concerns above.
Summary
This paper explores a new direction to pretrain vision backbones using large scale image-text pairs for learning visual representations which are suitable to various downstream tasks. More specifically, this work proposes a classification based objective function as an effective alternative to CLIP's standard cross-modality similarity based constrastive loss. The proposed model SuperClass uses a image encoder to map image to token probabilities having a head size equal to CLIP's tokenizer vocabulary. The texts are converted into labels using weighted tokenizer. Both CLIP baseline and SuperClass are pretrained on datacomp image-text pairs dataset and evaluation results across various vision and vision-language benchmarks are reported. The proposed method shows greater efficiency due to being text-encoder free and also performs favorably well over previous approaches. Extensive ablation studies are performed to justify the design choices made in the paper.
Strengths
**Strengths:** 1) The idea of pretraining large scale vision models using classification objectives is very motivating, as it provides advantages over image-text contrastive loss such as compute efficiency, disentangling the role of text embeddings etc. 2) The proposed method is simple and effective. 3) The experimental results are favorable for SuperClass against its direct baseline CLIP. 4) The choice of the components in the proposed method such as IDF based weighting, loss function and tokenization has been validated in the paper via ablation studies. 5) Paper is easy to read and understand.
Weaknesses
**Weaknesses** 1) In my understanding, one of the weaknesses of this work is lack of comparisons with related works. For example, the only method with which SuperClass is compared with is CLIP which is only a baseline. I believe there should be comparisons with other related SOTA works. 2) The proposed approach might not be capable of doing multi-label classification or zero-shot classification as compared to other competitors such as RAM, CLIP etc. Also CLIP has out of the box additional features such as prompt ensembling, zero-shot segmentation [1] etc which might be not present in SuperClass. This would question how it can comprehensively show effectiveness over CLIP and other vision-language models. 3) It is not clear how the proposed method learns suitable representations when using the subwords as labels which does not correspond to any visual concept. I believe there is a bit of analysis missing in the manuscript. 4) The results for the zeroshot and linear probing in Tab1 and Tab 2 of the main paper are different for the same model. It is unclear if why the results are different for same model. [1] Extract Free Dense Labels from CLIP (ECCV 2022)
Questions
Please refer to the weaknesses section for the questions. I will highly recommend the authors submit a rebuttal response. I will be happy to reconsider my final scores based on the response from the authors.
Rating
6
Confidence
5
Soundness
2
Presentation
3
Contribution
3
Limitations
Yes the authors have adequately addressed the limitations in the paper.
Summary
This paper introduces a multi-label classification pre-training style for visual image encoder pre-training.
Strengths
The proposed method is straightforward.
Weaknesses
- The zero-shot capacity of such a multi-label pre-trained model is not well demonstrated. - The paper lacks a comprehensive comparision with weakly-supervised or unsupervised visual encoder methods. Only comparing with CLIP is not enough. - Some arguments about previous so-called "bag-of-word classification" pre-trained methods may not be correct.
Questions
- 1. The greatest advantage of CLIP is its zero-shot capacity across various downstream tasks. The zero-shot ability of SuperClass is not well-demonstrated on downstream tasks like zero-shot text-image retrieval, zero-shot text-video retrieval, and zero-shot STR. The reviewer thinks the authors should include an analysis of the zero-shot ability like the original CLIP paper. - 2. The baselines should not be CLIP-style pre-trained vision language models. The proposed method aims to pre-train a vision transformer. The baseline should be other weakly-supervised methods or self-supervised methods like MAE, etc. The authors should compare these methods in terms of training efficiency and performance of transfer learning. These methods also show good transfer learning ability, for example, MAE has good performance of transfer learning on COCO, while it is only pre-trained on ImageNet-1K. - 3. Some arguments about previous so-called "bag-of-word classification" pre-trained methods may not be correct. In the introduction, the authors claim " However, these methods fail to gain popularity from the community, as most of the experiments are conducted on a small scale and there is no evidence showing their scalability to data size and model size in comparison to CLIP ". Nevertheless, CatCLIP[1] already conducts experiments with CLIP-H on DataComp-1.3B. - 3.1. The authors should also compare with these "bag-of-word classification" pre-trained methods. - 4. The authors say "All experiments were carried out on an A100 GPU equipped with 80GB of memory." Could the authors provide training times for the experiments in Table 1? Besides, are experiments in Table 2 also trained on an A100 80G? If so, could the authors provide the training time? [1] https://arxiv.org/pdf/2404.15653
Rating
3
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
Yes.
Rebuttal by Authors Part 2
>**Q4** Compared with these "bag-of-word classification" pre-trained methods **A4.** We have included the comparison with other classification-based methods, like CatLIP in the subsection Word-level tokenizer vs. Subword-level tokenizer. The word-level tokenizer is used in CatLIP [46], which carefully selected approximately 40,000 "gold labels" from the datacomp-1B dataset. Aside from the tokenizer being different, all models are trained under the same settings. The results of Table 4 show that with the increasing size of the model, the subword-level tokenizer gradually outperforms the word-level tokenizer, whether in classification tasks or vision & language tasks. We also provide the results of finetuning on ImageNet-1k in the below Table. The superclass could achieve better performance than CatLIP. | Model | Pretraining | ImageNet-1k Fine-tuning | |---|---|---| | OpenCLIP ViT-L/14 | Datacomp-1B | 87.4 | | CatLIP ViT-L/16* | Datacomp-1B | 86.5 | | Superclass ViT-L/16 | Datacomp-1B | 87.8 | *number from the paper >**Q5** GPU usage **A5.** This is a typo. What we meant to say is all experiments were carried out on 80G A100 GPUs. We will fix it in the revised version.
Rebuttal by Authors Part 3
Thanks for your reviews and feedback. >**Q1:** So far, we only see zero-shot performance on classification tasks. It seems that CLIP training style shows broader zero-shot ability in different downstream tasks. **A1:** We have evaluated zero-shot retrieval on COCO dataset and zero-shot segmentation on Pascal context and COCO stuff (see more details in the response to Reviewer W9bX). The experimental results show that Superclass could achieve competitive performance compared to CLIP. | Case | Model | COCO Image-to-Text | | COCO Text-to-Image | | |:---:|:---:|:---:|:---:|:---:|:---:| | | | R@1 | R@5 | R@1 | R@5 | | openCLIP | ViT-Large | 62.6 | 84.2 | 46.9 | 70.7 | | SuperClass | ViT-Large | 62.1 | 83.3 | 47.1 | 70.7 | | Method | Backbone | #Seen sample | PASCAL Context | COCO Stuff | |---|---|---|---|---| | CLIP | ViT-B/16 | 1.28B | 16.2 | 8.7 | | Superclass | ViT-B/16 | 1.28B | 20.2(+4.0) | 13.2(+4.5) | Besides, we also present results of linear probing and fine-tuning on ImageNet-1K. Compared to CLIP, SuperClass outperforms openCLIP on seven out of ten zero-shot classification datasets. Our method achieves a 1.1% higher accuracy in IN-1K linear probing (85.0 vs 83.9). To further demonstrate the transfer ability of our method, we conducted semantic segmentation on ADE20K and instance segmentation on COCO. Moreover, we provide experimental results on various vision & language tasks after integrating with large language models. The results show that our method outperforms CLIP. We hope that by showcasing our competitive performance in zero-shot retrieval, zero-shot segmentation, linear probing, fine-tuning, semantic segmentation, instance segmentation, and various vision & language tasks, we can address the reviewer’s concern of "only see zero-shot performance on classification tasks". >**Q2:** From the comparison, I do not see the superiority of the proposed pre-trained method compared to self-supervised learning, especially since the proposed method is actually weakly-supervised. **A2:** Compared to the current SOTA self-supervised model DINOv2, our method achieves a 0.5% higher accuracy in IN-1K linear probing (85.0 vs 84.5). Although SuperClass has seen more samples, our method is very simple and straightforward. DINOv2 adopts a dual-tower structure and adds a bunch of bells and whistles as shown in Table 1. Furthermore, we would like to remind Reviewer aefc the very important comparison on vision and language tasks. Our method outperforms DINOv2 in 7 out of 9 tasks, and the overall score is significantly higher than that of DINOv2 (Dinov2 54.66 vs. Ours 58.94). Please note that the Dinov2 used here is distilled from the ViT-giant model. These improvements are significant. We thank the reviewer bBvT for the comment, "At its core, this paper introduces a new way to pre-train open-world Vision-Language foundational models," and Reviewer W9bX for noting, "this is a new pretraining style for vision backbones rather than a variant of CLIP," our proposed new pretraining method is distinctly different from previous pretraining approaches. The differences from other most closely related classification-based pretraining methods are also evident: our method does not require manually curated "golden labels" and directly uses tokenized raw text as supervision. In the paper subsection "Word-level tokenizer vs. Subword-level tokenizer," we demonstrate through experiments that our method has better performance and scalability. As Reviewer 3a7k mentioned, "Compared to previous classification-based pretraining methods, the proposed approach demonstrates greater practical applicability." Overall, we believe we have provided a thorough response to Reviewer aefc's comments and hope that Reviewer aefc will review our feedback in light of the perspectives that align their rating with other reviewers.
Discuss
Dear authors: Thank you for you response. - Q1 *** Could you please tell me where the zero-shot 46.9 COCO Text-to-Image result comes from? I check the paper of CLIP[1], the zero-shot image retrieval result is 37.8. - Q2 *** - CLIP is born in 26 Feb 2021. Many other VLM pre-trained under language supervision emerged, e.g., EVA-CLIP[2] - Self-supervised pre-training and weaky-supervised methods are also hot. According to the provided information, the proposed method uses 10x data (12.8B), and it is close to DINOv2(1.92B). If we talk about "performance and scalability", DINOv2 ViT-g/14@448 achieves 86.7% linear probing results on ImageNet-1K. Overall, the baselines are far from strong. The reviewer does not know why we need such a method when we have many other powerful and versatile pre-trained VLMs/self-supervised large models/weakly supervised large models. I still retain my score at this time. Best, Reviewer aefc [1] https://arxiv.org/pdf/2103.00020 [2] https://arxiv.org/abs/2303.15389
Rebuttal by Authors Part 4
We thank reviewer aefc for their timely response. Regarding Q1, we utilized the open-sourced checkpoint [1] from paper [2], which employs ViT-Large as its backbone and is trained with Datacomp-1B and 12.8 billion seen samples. This serves as a stronger baseline compared to the original OpenAI CLIP. >**Reviewer aefc commented "The reviewer does not know why we need such a method when we have many other ... pre-trained ... large models" (VLMs/self-supervised/weakly supervised).** It seems the reviewer suggests that foundational modeling research might be redundant when the absolute performance are weaker compared to a "large model" pretrained in a systematic way. The reviewer appears to prioritize "higher numbers" even when comparing in an apple-to-orange setting. While we acknowledge that a big model with higher numbers is often advantageous from a downstream users' perspective, we believe that a scientific foundational model researcher would in favor of careful ablations rather than chasing higher numbers with biggest models. For instance, the reviewer argues that SuperClass cannot demonstrate better "performance and scalability" than DINO v2, citing that the DINO v2 *ViT-g 448* model, pretrained on a customized dataset at 448 resolution - superior to our largest *ViT-L 224* model at 224 resolution. We appreciate the reviewer's observation that our *ViT-L 224* resolution model is less powerful compared to a *ViT-g 448* resolution model. It appears that the reviewer has overlooked various systematic differences - unintentionally or for some other unknown reason - comparing scientific papers in an apple-to-orange manner. >**The reviewer aefc further suggested "Overall, the baselines are far from strong. "** The baselines we compared here are CLIP trained with DataComp [2] and DINOv2 [3], which are important milestone papers accepted at NeurIPS 2023 and TMLR 2024, respectively. For some unknown reason, the reviewer believes CLIP, DataComp and DINOv2 are far from strong. We kindly respect the reviewer's false claim. We would like to wish the reviewer the best, and politely disagree with the reviewer's comments and rating. We further thank other reviewers for their professional, scientific, objective reviews. [1] https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K [2] DATACOMP: In search of the next generation of multimodal dataset, NeurIPS 2023. [3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024.
Discussion
Thank the authors for the response. ## 1. "performance and scalability" *** Well, the authors say that your methods have better "performance and scalability" in https://openreview.net/forum?id=Hd2EOwKItm¬eId=KeV1R5CtDE **A2**. So, - The reviewer provides examples that previous methods have better "performance and scalability". - The proposed method use **10x** data, but is close to DINOv2 with the same parameters. - The reviewer does not even mention similar work CatCLIP[1], which also trains CLIP in a classification way. It also demonstrates better "performance and scalability". ## 2. It seems the reviewer suggests that foundational modeling research might be redundant when the absolute performance are weaker *** The authors are trying to mislead readers. - The paper does not propose something novel, CatCLIP[1] already does the classification pre-training job. - The performance/scalability of the paper is no better than recent literature [2,3]. The proposed method does not demonstrate its necessity when there are many other strong self-supervised/weakly-supervised/language supervised (may belong to the weakly-supervised class) methods, so it is reasonable that the reviewer thinks this work may not be necassary for the community. ## 3. Different data from CLIP paper The authors do not answer this question. https://openreview.net/forum?id=Hd2EOwKItm¬eId=vqnfjpW7aT Q1. ``` Could you please tell me where the zero-shot 46.9 COCO Text-to-Image result comes from? I check the paper of CLIP[1], the zero-shot image retrieval result is 37.8. ``` ## 4. Finally, as an independent reviewer, the reviewer thinks he can have his own opinion and rating about the paper. Best, Reviewer aefc [1] https://arxiv.org/pdf/2404.15653 [2] https://arxiv.org/abs/2303.15389 [3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024.
Thank the Reviewer aefc for the feedback. We have addressed all Reviewer aefc's questions in our previous responses, including but not limited to - performance and scalability > Line #212 subsection "Data Scaling results Line" and #221 subsection "Model scaling results" in the paper > The results of Zero-shot classification, VLM downstream tasks ... in https://openreview.net/forum?id=Hd2EOwKItm¬eId=JxjyAuZo5M (Rebuttal by Authors) > Few-shot classification in A1 https://openreview.net/forum?id=Hd2EOwKItm¬eId=1hHvBn6szv (Response to Reviewer bBvT) - comparisons and differences with CatLIP[46] > A3 in https://openreview.net/forum?id=Hd2EOwKItm¬eId=JxjyAuZo5M (Rebuttal by Authors) "CatLIP [46] is a concurrent work. We have already cited and compared it in the paper. " > Line #69 and Line #233 subsection "Word-level tokenizer vs. Subword-level tokenizer" in the paper > Fine-tuning results in A4 https://openreview.net/forum?id=Hd2EOwKItm¬eId=OOhKhswjjl (Rebuttal by Authors Part 2) > https://openreview.net/forum?id=Hd2EOwKItm¬eId=KeV1R5CtDE (Rebuttal by Authors Part 3) "The differences from other...classification-based pretraining methods are also evident...manually curated "golden labels"..." - Performance comparisons with CLIP and Dinov2 > The results of Zero-shot classification, VLM downstream tasks ... in https://openreview.net/forum?id=Hd2EOwKItm¬eId=JxjyAuZo5M (Rebuttal by Authors) > The results of 10-shot classification in A1 of https://openreview.net/forum?id=Hd2EOwKItm¬eId=aVM6gFdSHR (Response to Reviewer bBvT) ... - Different data from CLIP paper > https://openreview.net/forum?id=Hd2EOwKItm¬eId=ogRsx4SgwE (Rebuttal by Authors Part 4) "Regarding Q1, we utilized the open-sourced checkpoint [1] from paper [2]...This serves as a stronger baseline compared to the original OpenAI CLIP." We do not intend to engage in repetitive responses and hope to strengthen and solidify our work based on the reviewers' valuable feedback. We further thank all the reviewers.
Thank you for the response. I will maintain my score at this stage
Thank you for your quick response and for taking the time to review our paper. We appreciate your feedback and are grateful for your recognition of our work.
Thank you for providing the rebuttal response
Dear Authors, Thank you for providing the rebuttal response. While the proposed SuperClass method is not natively zero-shot, it shows other various flexibilities as demonstrated in the rebuttal. More importantly, this is a new pretraining style for vision backbones rather than a variant of CLIP. Honestly, the current pitch of the paper puts too much emphasis on CLIP (instead of advocating for vision backbone pretraining), and that is why many of the concerns from the reviewers are about comparisons with CLIP-like models. In the end, I believe this paper would allow the research community to improve vision backbone pretraining and have good insights. Therefore I will increase my score to weak accept, and hope that the paper is accepted. For the final version, I strongly recommend the authors to include all the rebuttal discussions in the main paper, and also put more emphasis on vision backbone pretraining, so that no ambiguities are left.
Thank you for your valuable reviews and feedback. We appreciate your insights on emphasizing the new pretraining style for vision backbones rather than comparing it predominantly with CLIP. We will ensure that the final version of the paper includes all the rebuttal discussions and places greater emphasis on vision backbone pretraining to eliminate any ambiguities. Thank you again for your constructive comments and for increasing your score to a weak accept. We hope that our paper will contribute positively to the research community.
Discussion
Dear authors: Thanks for your time, efforts, and response. - **Q1** So far, we only see zero-shot performance on classification tasks. It seems that CLIP training style shows broader zero-shot ability in different downstream tasks. - **Q2** From the comparison, I do not see superioty of the proposed pre-trained method compared to self-supervised learning especially the proposed method is actually weakly-supervised. Table 4 in DINOv2 provide a more comprehensive comparison. It seems DINOv2 also have better transfer learning results in its Table 10. Overall, I do not think the proposed methods show enough superiorty compared to CLIP and existing weakly-supervised/self-supervised methods, especially when there are some similar works like CatLIP. At this time, I still hold my original score. [1] https://arxiv.org/pdf/2304.07193
Decision
Accept (poster)