CLIP, one of the pioneering foundation models that connect images and text, has enabled many recent breakthroughs in computer vision. However, its associated training cost is prohibitively high, imposing a significant barrier to its widespread exploration. In this paper, we present a surprising finding that there exists an inverse scaling law for CLIP training, whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. Moreover, we showcase that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law. As a result of this finding, we are able to successfully train CLIP even with limited computational resources. For example, using 8 A100 GPUs, our CLIP models achieve zero-shot top-1 ImageNet-1k accuracies of 63.2% in ~2 days, 67.8% in ~3 days, and 69.3% in ~4 days. Our method also works well when scaling up -- with G/14, we register a new record of 83.0% ImageNet-1k zero-shot accuracy, and meanwhile accelerate the training by ~33x compared to its OpenCLIP counterpart. By reducing the computation barrier associated with CLIP, we hope to inspire more research in this field, particularly from academics. Our code is available at https://github.com/UCSC-VLAA/CLIPA.
Paper
Similar papers
Peer review
Summary
This work explores how to increase the training speed of CLIP in order to make it possible to train on academic resources. They examine how to best compress both the visual and text information given a limited token budget. They do this across multiple CLIP sizes and find that as the size of CLIP increases, it sees less of a drop for fewer tokens which they deem the "Inverse Scaling Law".
Strengths
This is a very useful line of work, as I believe there is still a lot of research to be done in the Language-Vision space and the compute requirements of CLIP is a huge barrier. Allowing CLIP training on an academic resource budget is valuable to the greater resource community. It is presented in an easily understood way. I especially Like figure 7, which summarizes their results nicely.
Weaknesses
My main concern is novelty. Their findings are very similar to that in "RECLIP: Resource-efficient CLIP by Training with Small Images" which also found that images could be sized down and text could be truncated without major accuracy loss, with the goal of increasing training accessibility. This paper essentially takes the main findings of this paper, and provides extensive ablations for a series of design choices (such has the best way to compress the images/text), but ultimately comes to a very similar conclusion. However, as RECLIP, was only on ArXiv 2 months ago, I believe it should be considered as concurrent work. I would however ask that the authors include a discussion of this work in their next version.
Questions
- How do you see your work in relation to the concurrent RECLIP (https://arxiv.org/abs/2304.06028)? - What do you see as the main limitations of this work? - [minor note] typo in line 208 (double ampersands)
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
In future versions I would recommend a section for limitations.
Summary
Different methods to reduce the number of tokens entering the image or text encoders when training a CLIP model are compared. In doing this the authors discover that larger models are able to learn faster from fewer tokens per example than smaller models; this is the inverse scaling law in the title. Models are compared keeping the following variables constant: * Training set: LAION-400M * Total pre-training examples: 6.4 epochs * Tokens/example is **not** constant so tokens/epoch will be vary accordingly * Total fine-tuning examples: 0.36 epochs * Learning rate: 8e-6 pre-training and 4e-7 fine-tuning * However, this is not kept constant for S/16 and B/16 models in scaling experiments The authors investigate alternative strategies to reduce tokens/example based on FLIP, designing them based on the attributes of images (grid masking: "preserving one patch in each 2x2 grid window") or attributes of text (syntax masking preferentially retains nouns). They then investigate the effect of scaling while reducing tokens/example. Using the discovery that larger models learn faster from fewer tokens the authors are able to train a CLIP Academic (CLIPA) (I can't find where this is defined in the paper but I think this is what it is?) on LAION-2B using 1/15 the compute budget of OpenCLIP at the H/14 scale and achieve comparable performance.
Strengths
The experiments in this paper are comprehensive and follow a reasonable thread of investigation. The authors define clearly what they aim to investigate, then show complete results of that investigation, then apply those results to a pressing problem in the field. The comparison of different methods to reduce tokens/example is valuable and the methods chosen for comparison seem like good choices. The results are presented in full in Figures 5 and 6 and Table 1. The training results are striking and of obvious importance to the community. CLIP was a major step in the field of deep learning so speeding up training is valuable to increase access and to speed up experimental iteration on these architectures. The speedup of 16x is more than an order of magnitude. This paper builds on the results of FLIP by making a useful observation on the effect of model scaling on token/example reduction strategies.
Weaknesses
Figure 1 is difficult to parse because the number of tokens in the top row could indicate entries in the contrastive matrix or it could indicate the number of tokens at the input to the image/text encoders. From reading the paper, I think it is the latter but it would be helpful if this was immediately obvious. If this mirrored the design of Figure 1 in the original CLIP paper I think it would be very clear. The results of this paper are quite different if we view epochs as being a measure of tokens or of steps. I think that it is a measure of steps because the alternative would be more difficult to code but I think it would be worth making clear so that's what I wrote in the summary above. I think to make this extremely clear and better state the scaling law there should be a figure comparing log total training tokens to loss or accuracy. From Figure 4 I think this would show increasing accuracy with the number of tokens processed by the network during pre-training. It is not clear in Section 5 exactly which methods are applied in CLIPA training to reduce tokens/example, from the methods proposed in previous sections. It would be useful for this to be made clear, even if it is stated in prior sections or appendices, because the results hinge on this. The speedup quoted in Section 6 and in line 51 compares to OpenCLIP but Table 4 shows similar comparison to the work this is most directly based on, FLIP, was trained with only 5x the compute. This is a more fair comparison and deserves to be stated instead of the alternative unless you qualify that it is 16x cheaper than training without doing anything to reduce tokens/example. Smaller issues: * In the abstract the bold results at the end are impressive only if one knows the training time of OpenCLIP but it might be more reasonable to assume that most readers won't * Lines 33-36 describe the idea of a scaling law in parameters but neglect scaling laws in tokens or compute etc, it might be valuable to be more precise * The text in Figure 5 x-axes is very small, could it be made more legible somehow? * I think Figure 7 shows the effect of the tokens/example reduction strategies on GFLOPS required for training but the caption does not explain what is happening in this Figure. Is GLOPS the number of flops per example, so the x-axis would show the total training cost?
Questions
What is the formal definition of a scaling law this paper is applying, such that this demonstrates an inverse scaling law? I'm not sure if the field has one, it seems to be "a straight line predicting performance on a y-axis and training cost on a logarithmic x-axis". Have I misunderstood the role of the masking methods to reduce tokens/example? I don't think I have but my entire review depends on it.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
Comparing this to a [Chinchilla][] scaling law framework it may be that CLIP models are typically trained with more tokens than necessary, in which case we would expect to observe larger models perform better than smaller models regardless of the reduction in tokens. This could be investigated in a way analogous to the Chinchilla paper, by varying the total number of training tokens seen be the model for different model scales and aiming to see the IsoFLOP curves shown in Figure 3 of the Chinchilla paper. However, this would be a computationally expensive experiment to run. Perhaps the authors can come up with a cheaper way? [chinchilla]: https://arxiv.org/abs/2203.15556
Summary
This paper is about increasing the training efficiency of vision-language models such as CLIP. These models can serve as a foundation for many vision applications and are therefore very useful. However, they are typically (pre-)trained on large datasets for many steps, which makes research involving training such models costly. This motivates efforts to reduce the training cost. The paper focuses on the sequence length of both the image encoder and the text encoder in CLIP-style VLMs, which is a key determinant of model compute and memory requirements. The paper evaluates random token dropping, dropping tokens in grids or blocks, and reducing the image resolution. The paper finds that reducing image resolution performs best based on zero-shot ImageNet classification performance. For the text encoder, the paper finds that dropping text tokens such that nouns are preferentially kept and other words are dropped yields the best performance, again on zero-shot ImageNet classification. The paper also sweeps across different model sizes and finds that larger models can tolerate larger reductions in sequence length on both the image and text encoder. With the proposed sequence length reductions, it is possible to train models matching or exceeding the performance of OpenCLIP (but not original OpenAI CLIP) on image classification and image/text retrieval in 10-20x less training time.
Strengths
1. The work is well motivated. Reducing the cost of pretraining foundation models has broad impact. 2. The reduction in training time is significant and could make research on training VLMs easier (although it comes with caveats, see below). 3. The presented experiments are clear and and well-structured.
Weaknesses
1. **Incomplete evaluation:** The paper relies on zero-shot classification and retrieval to evaluate the proposed model against the original CLIP and OpenCLIP. This covers only a small part of the tasks that CLIP is typically used for. The original CLIP paper also evaluates representation quality with linear probing and finetuning, on a much wider range of tasks and image types. These evaluations may be more informative about general representation quality than zero-shot image classification. In particular, approaches such as reducing the input resolution and dropping non-noun words may work for classification, where there is a single big object in the image and the text usually consists of a single noun, but might harm more structured or fine-grained downstream applications. This should at least be discussed and ideally tested, e.g. with linear probing and finetuning, and a wider range of tasks, ideally including structured tasks such as segmentation or detection. 2. **Inaccurate and unnecessarily polemic presentation:** The paper makes several inaccurate statements with the apparent motivation of drawing attention. These inaccuracies distract from the content: 1. **Inverse scaling law:** Despite the title, the paper does not propose a quantitative scaling law, and there is nothing “inverse” about the presented scaling results. Figure 7 shows that as the compute budget increases, the optimal model size grows. This is the same result first presented in Figure 2 of https://arxiv.org/pdf/2001.08361.pdf and confirmed in many works since. The observation that larger models tolerate larger reductions in sequence length neither makes the scaling “inverse” nor is it a surprising result, since bigger models are known to be more sample efficient (https://arxiv.org/pdf/2001.08361.pdf, https://arxiv.org/pdf/2106.04560.pdf), i.e. they learn faster from less data than smaller models. I suggest rephrasing the title to avoid confusion. 2. **CLIP restricts accessibility:** The abstract and introduction suggest that foundation models such as CLIP are “restricting accessibility to a small group of researchers and technology companies”. This seriously misrepresents the impact these models had on low-resource research. In fact, foundation models like CLIP made it possible to train a model once and then apply it zero-shot, or with cheap fine-tuning, to many different tasks. Sharing pre-trained foundation models therefore has been a huge gift to low-resource research, since complex research questions can now be explored with a pretrained CLIP model, often without having to do any model training. Portraying such models as hindering low-resource research is inaccurate and unfair towards the researchers who develop and release them. I suggest the authors revise these statements. 3. **Academic resources:** Another questionable premise is that of “academic” resources (e.g. in the title of Section 5 and the name of the proposed method). There are academic labs with large resources and industry labs with low resources. In addition, research labs in industry that publish their work can reasonably be considered to be part of academia (for reference, Wiktionary defines “academia” as “The scientific and cultural community engaged in higher education and research, taken as a whole.”). Equating “academic” with “low resource” is therefore inaccurate and reinforces stereotypes. Please simply talk about “few/low resources” instead of “academic resources.” In summary, additional evaluation and major revision of the presentation are necessary before publication at NeurIPS.
Questions
Further suggestions: * Please provide Table 1 data in graphical form like Figure 4. * Table 3 should include inference FLOPs to be able to compare models. Typos: * L170: For instance, For instance * L208: Double ampersand
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The paper does not address the concern that the limited evaluation of the proposed model may hide model deficiencies compared to the original CLIP. This should be addressed in the rebuttal.
Summary
This work studies the impact of token length (image and text) on the quality of image-language pretraining in CLIP-style models. Crucially, the authors find that as the model size (both image and language encoders) are scaled up, the drop in accuracy owing to a smaller input token length keeps on decreasing. This is pitted as the inverse scaling law. A smaller input token size enables training significantly faster, achieving nearly 15x speed up over the conventional training protocols, and also improving over the gains observed in prior work, FLIP.
Strengths
1. The work provides a comprehensive analysis of various token reduction strategies for both image and text and also proposes new ones. 2. Analysis of different scales of models, across different masking strategies is really helpful to convert a previously under-explored finding into a consistent phenomenon that can be useful for future research. 3. The work makes large model training accessible by suggesting an alternate training paradigm of pre-train and then fine-tune that enables CLIPa training at a much lower cost than before (15x speedup).
Weaknesses
1. I find Figure 1 confusing. What is the relation between the encoder input token length and the encoder output "image/text embedding" dimension. This should be discussed in more detail in the paper. Also, since the whole work is about token length reduction, I think it is important for the authors to discuss how images are converted into patches and then into tokens and how this calculation is done. 2. I do not feel very confident about the results in the paper because the parameter choices appear to be heavily tuned in order to get good performance for the chosen strategy of resizing. For instance, the authors use a base learning rate of 8e-6 in the main training stage and 4e-7 for fine-tuning---these are non standard parameters and a discussion on parameter tuning is important here. 3. Similarly, in line 147 the authors write that "one exception that a larger learning rate of 8e-7 is utilized during fine-tuning for S/16 and B/16." Such opportunities were not given to baselines such as FLIP. Further, special tricks and augmentations were used for H-14 making it unclear how the results will translate further. 4. Scaling Law: Since this paper advertises a scaling law, the graphs should have model size on x axis and the various fractions of token reduction as individual lines. Moreover, there should be more points on the "new" x axis to claim a scaling law. 5. Why are there no graph for results corresponding to Table 1. 6. Evaluations: Model performs significantly worse on text retrieval tasks. I think the authors should do a more comprehensive evaluation of CLIPA on various benchmarks such as VTAB (check datacomp paper for current best practices). 7. Table 3 should compare with FLIP and other masking strategies
Questions
1. What is the performance of CLIPA models without fine tuning 2. How was the fine tuning duration and learning rate chosen? 3. How were parameters for pretraining chosen? 4. What are the embedding dimensions of the models when the token lengths are changes (see Figure 1)
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Discussed in Weaknesses
Summary
The key finding in this paper is that, the larger the model is, the shorter the token sequence is required to maintain a certain level of performance. Based on this finding, this paper proposes to train larger model with shorter sequences, which is more computationally favorable than training smaller models with sequences of full length. As a result, one can achieve 69.3% top-1 zero shot image accuracy with just eight A100 GPUs in ~4 days.
Strengths
- The finding about the inverse scaling law for CLIP is interesting. Given that CLIP is such a significant way of training vision language models, the inverse scaling law may yield high impact in this direction as well. - The writing is overall good except some minor points (see in the next section). The visualization in Figure 2 and 3 well demonstrate the shortening strategies used, making it easier to understand for readers who can only afford a quick scan. The results presented in Figure 4, 5, amd 6 are clear, and easy to follow. - The experimental results are strong. As shown in Table 3, one can train the same arch without much drop (e.g., 2-4% drop) but saving 10x-20x GPU hours. This is can be very important for academic labs where resources are not huge. - The figure 7 shows that, when given a fixed number of computation budget, you can achieve better accuracy with larger models with this inverse scaling laws applied.
Weaknesses
I did not see major flaws of this paper, but with some minor comments: - firstly, I think this paper misses a subsection or paragraph in Related works, which can talk about similar findings in language models. To my best knowledge, it has been well known that, larger models can be trained with less steps/epochs/durations compared with smaller models. One example is the paper "Scaling Laws for Neural Language Models". So I think it would be nice to give readers a full spectrum of knowledge by including a series of such related work in the large language domains, and I would not think including them will make this paper less innovative, rather this paper becomes more complete. - this paper claims multiple times that CLIP is the first foundation model that connects images and text. I am less comfortable about this claim, since there are a bunch of image-text relevant work before CLIP, e.g. such as image captioning model. Having not trained with modern large models does not mean that they are not foundational or they could not have been foundational models if they were trained with large models. This is a minor point, but would love to see more rigorous and nice claim. - Why there is performance improvement for block mask in Figure 4, and similarly why this is happening fro Figure 5 too. And it seems there is in consistency between different tasks in terms of which strategy can improve or not? - Why it improves a lot for ImageNet-A in Figure 6, while not the case for other datasets/tasks? - in the paragraph of line 204, it says syntax masking even enhances the performance for CLIP. I wonder if this is mainly because it keeps nouns, which are more informative for tasks relevant to object recognition. What about some other tasks that focuses on predicting/modeling relationships between objects? Would syntax masking still effective, or would still text masking useful at all?
Questions
see above
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
I would not say this is the limitation, but it pushes me to wonder how general this inverse scaling law is? e.g., would it be easily applied to other tasks?
Thanks for clarifying
From your rebuttal I think I understood the paper in my review and I think it is a worthwhile submission for the conference. I will update the confidence of my review.
Thanks for your support! We deeply appreciate your acknowledgment of the significance and potential of this work.
Thank you for the response
Thank you authors for the rebuttal. One idea about the Q5. Maybe it's not too hard to try the "Attribution, Relation, and Order (ARO) benchmark", given their colab has been released for evaluation. I have read other reviewers' comments, I agree that the current evaluation, i.e., zero-shot, is limited, though VITA is included in the rebuttal. After a more comprehensive understanding, I think the current score is a proper reflection of how I view this work, so no change.
Thank you for these additional valuable suggestions, which are important for furthering the quality of our work. ### Q1: ARO Benchmark We genuinely appreciate your suggestion regarding the ARO benchmark [1]. Our preliminary results are shown in the table below. We can observe that, while OpenCLIP B/16 slightly outperforms CLIPA B/16, the absolute performance of both on this benchmark remains somewhat limited. This aligns with the observations from [1], emphasizing that **CLIP models generally exhibit limitations in capturing relationships, attributes, and order information**. To mitigate this relational understanding issue, a composition-aware hard negative mining strategy (NegCLIP) is introduced in [1]. Note that this strategy is extremely lightweight, and can be seamlessly integrated as an additional fine-tuning stage in enhancing CLIP’s text understanding ability. Our results below also corroborate the efficacy of NegCLIP, e.g., both OpenCLIP and CLIPA nearly double their performance on benchmarks like COCO-Order and Flickr30k-Order. We will add the discussion above in the next version. We hope that these additions, coupled with the insights from [1], can help alleviate reviewers’ concerns about the capabilities of CLIP’s text encoders. | Model | NegCLIP | VG-Relation | VG-Attribute | COCO-Order | Flickr30k-Order | |:-------------:|:----------------:|:------------:|:------------:|:------------:|:---------------:| | OpenCLIP-B/16 | | 44.7 | 59.9 | 41.8 | 45.3 | | OpenCLIP-B/16 | ✓ | 78.6 (+33.9) | 69.5 (+9.6) | 87.6 (+45.8) | 89.1 (+43.8) | | CLIPA-B/16 | | 43.8 | 57.1 | 37.8 | 39.1 | | CLIPA-B/16 | ✓ | 78.5 (+34.7) | 68.0 (+10.9) | 86.1 (+48.3) | 87.9 (+48.8) | ### Q2: Zero-shot Evaluation We recognize and understand the concerns regarding the evaluation breadth. But it is essential to emphasize that our primary focus on zero-shot performance is not arbitrary. One of the most distinguishing facets of CLIP, setting it apart from other learning paradigms, is its strong capability in zero-shot recognition. This unique strength is consistently underscored in both the official CLIP repository and the widely-recognized OpenCLIP repository. Our research approach aligns with this established practice, and extensively validates the inverse scaling law by measuring models’ zero-shot abilities across various datasets. While our current study offers insights into one (important) aspect of CLIP's capabilities, we do acknowledge that understanding in a broader context, especially in different downstream tasks, is also crucial. We are interested in and committed to keep exploring these insights in our future endeavors. **References** [1] Yuksekgonul M, Bianchi F, Kalluri P, et al. When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?[C]//The Eleventh International Conference on Learning Representations. 2022.
Thank you for your efforts on the ARO Benchmark benchmark. It makes sense to see that CLIPA underperforms CLIP, because of language drop out. I encourage the authors to include these results in the final version.
Yes, these discussions will be included in the final version. Thanks again for your invaluable suggestion about the ARO benchmark and NegCLIP training algorithm.
Thanks for your response. Q1: Thanks for providing the VTAB results. They do not address my concern that the short token sequences may bias CLIPA so that it does well on classification, but perhaps less well on tasks requiring more complex language understanding. VTAB is a classification benchmark, so it adds little additional information here. In fact, as Reviewer UF9M pointed out, CLIPA performs worse on retrieval, which may be due to the shortened sequences during (pre-)training. Q2. The authors write "it's crucial to distinguish our findings from the scaling law in [1]". One way to distinguish their work from [1] would be to not appropriate the term "scaling law". What the paper presents is simply not a scaling law in the sense commonly used in the literature (as other reviewers have also pointed out). Regarding the term "inverse": A simple way to re-state your findings would be to say that as models get bigger, performance goes up. When performance goes up, you can train on less data (or shorter sequences) to reach the same performance. So there's nothing inverse -- it's just a re-statement of the well-known fact that bigger models reach higher performance. It is therefore an exaggeration to talk about a new "discovery" here, since the method is an application of well-known facts about neural networks. My original review continues to apply.
Thank you for the additional comments. ### Q1: Language Understanding Thanks for raising this concern. In light of your suggestion and as recommended by Reviewer Mhr1, we delved into the ARO benchmark to further our understanding. In short, our results 1) echo the finding in [1] that **CLIP models generally exhibit limitations in capturing relationships, attributes, and order information** and 2) confirm that NegCLIP, developed in [1], can effectively serve as a lightweight and drop-in solution to enhance the text understanding ability of both OpenCLIP and CLIPA. For a comprehensive breakdown of our observations on this front, please refer to our response to Reviewer Mhr1’s Q1. In addition, we would like to clarify the concerns about the drop in text retrieval performance (presented in Table 3). In short, our CLIPA models in Table 3 are intentionally trained with limited resources (e.g. on a 8-GPU server). Considering that other baselines use **20-60x** more compute, the fact that our CLIPA achieves competitive results is noteworthy, and should be considered as one of our key contributions. In addition, a strong evidence is that, when we scale to the size of H/14, our CLIPA model exhibits competitive outcomes in both classification and retrieval tasks compared to OpenCLIP (as presented in Table 4). For a more nuanced understanding of this aspect, please refer to our response to Reviewer UF9M’s Q2. ### Q2: Inverse Scaling Law We’d like to first reiterate that our findings are **novel** and should be distinguished from the scaling law in [2]. The differences can be summarized as: 1) the sample efficiency in [2] is realized by using few training samples, but our token efficiency is realized by using fewer tokens in each training sample. 2) our inserve scaling law focuses on **performance drop**, which really is a comparison between the **same** model trained with reduced-length or full-length input. One illustrative example is our CLIPA-H/14 (I36,T8), which is trained with **~87%** less tokens per sample during pre-training, and yet shows *barely any performance drop* compared to OpenCLIP-H/14 trained with full-length input. This phenomenon is not revealed in prior scaling law works. Additionally, we stress that our results indeed present an inverse relationship between the model size and the length of image/text tokens in CLIP training — that the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. We will make these points clear in the next version. **References** [1] Yuksekgonul M, Bianchi F, Kalluri P, et al. When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?[C]//The Eleventh International Conference on Learning Representations. 2022. [2] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).
I thank the authors for their efforts in responding to the questions. 1. First, the results on VTAB also show that the performance of CLIPA models when compared to OpenCLIP models is quite random, performing worse by quite a few points in many datasets, and also similarly performing better in a few others. Added to this is the significant drop in retrieval performance, which is one of the most important use case of image-language models. 2. I am not satisfied with the answer on hyperparameter tuning. In particular, the authors still do not answer how they selected these parameters. 3. I do not like the framing of this paper trying to pitch the story as an inverse scaling law when results are only discussed at 3 points on the x-axis, and when going to H-14 models special tricks and augmentations need to be used, making it unclear how the results will translate further. I resonate with the opinions of Reviewer KzVG in this regard. I congratulate the authors on their new efforts that have led to a new state of art open-source model, and recognize the efforts that this might have taken. I do think that the claim that "vision embeddings" of CLIP models do not need "long sequence length in text embedding" is the correct framing of this paper and this would still be valuable for publication at NeurIPS. But in its current form, the claims are an overstatement.
Thank you for these additional valuable suggestions, which are important for furthering the quality of our work. ### Q1: VTAB result When examining the VTAB benchmarks, our CLIPA-H/14 model outperforms on 6 datasets, the OpenCLIP-H/14 excels in 4, and FLIP leads in 6 datasets. This competitive performance of CLIPA, especially when considered alongside its efficiency benefits, substantiates our rebuttal claim that “our CLIPA still achieves comparable or even superior performances” ### Q2: Drop in text retrieval performance We recognize the concern raised about the drop in text retrieval performance (presented in Table 3). However, it is essential to contextualize our work's primary objective, which is uncovering the inverse scaling law in CLIP training — i.e., The performance drop caused by shorter token length will diminish for larger models. This observation is substantiated by our results presented in Table 4. For instance, when we scale to the size of H/14, our CLIPA model exhibits competitive outcomes in both classification and retrieval tasks Moreover, the intent behind Table 3 is not merely to showcase the absolute performance of CLIPA, but to demonstrate its potential in constrained resource settings. We intentionally trained CLIPA with strict computational bounds, limiting the training duration to 2-4 days on an 8-GPU A100 server. When considering these constraints, the fact that CLIPA can achieve competitive results is noteworthy. While other baselines outperform due to their **20-60x** higher computational budget, the efficiency and competitive performance of CLIPA in its restricted setting are pivotal aspects of our contribution. ### Q3: Hyper-parameter tuning Compared to FLIP, the only hyperparameter change in training is learning rate. The main reason is that our total training epoch is only ~6.4 epochs (which is relatively short, making models under-trained), therefore a slightly larger learning rate is applied to help convergence. It is crucial to note that our primary conclusion remains unaffected by this hyperparameter change. For clarity, 1) when applying the short 6.4 training epochs, all our baselines (including random masking/FLIP) are trained under the same hyper-parameter setup, therefore ensuring fair comparisons; and 2) for public baselines (like OpenCLIP) which are trained with sufficient epochs (i.e., 32 epochs), we verified that this learning rate change only lead to a negligible performance variation (e.g., within 0.2% in zero-shot ImageNet classification). ### Q4: Scaling law We stress that our observed scaling law in CLIP training is different from the typical scaling law observed in language models. More specifically, what we are trying to clarify is that larger models can apply shorter token length in CLIP training. We extensively examine this scaling law with 8 different strategies, each at varying masking ratios. By ranging the model size from S/16 to L/16, we obtain a total of **105** data points in well supporting the conclusion of inverse scaling law in CLIP training.
We are glad to see all your concerns have been addressed! Thanks for increasing the score!
Decision
Accept (poster)