No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

Summary

This paper investigates the "zero-shot" performance of multimodal models like CLIP and Stable-Diffusion. By analyzing 34 models and 5 pretraining datasets, the study finds that these models require exponentially increasing amounts of data to achieve linear improvements in performance. Additionally, the study suggests that the key to effective "zero-shot" generalization with large-scale training data and compute paradigms remains elusive.

Strengths

- The authors conducted a comparative analysis involving two main factors: (1) the performance of models across various downstream tasks, and (2) the frequency of test concepts within their pretraining datasets. - This paper is easy to read. - There is ample work involved in this study.

Weaknesses

- Please unify the description of the table, above or below the table. - This work does not point out correlations or differences with prior work, preventing me from assessing the novelty of this work. - Qualitative visualizations and relevant theoretical analysis are lacking, although the perspective of the work is easy to understand.

Questions

- Does this work conflict with research into improving the generalization of multimodal models? I think the author should discuss it. - Explanation of Figures 24-27 is insufficient. - What is the motivation behind this work? What implications does it have for future research? - To validate the points made in this paper, should other tasks beyond classification and retrieval be evaluated?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

- See Weaknesses and Questions.

Summary

The authors evaluate CLIP's zero-shot performance and its correlation to the pretraining data. Their experiments, based on the open LAION datasets, show that the limitations of Vision-Language Models (VLMs) in downstream tasks such as image generation and zero-shot recognition are linked to the frequency of concepts in the data. VLMs like CLIP are pivotal for downstream multimodal applications, and understanding their biases is crucial for developing more robust systems. While the paper demonstrates the limitations of these foundational models, it currently focuses more on breadth rather than presenting in-depth insights. I would be willing to increase my score if the authors could address my questions and fix some of the references. [1] Parashar, Shubham, et al. "The Neglected Tails in Vision-Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.

Strengths

1. The paper is well-written and motivated, highlighting the importance of understanding the limitations of Vision-Language models like CLIP. 2. The paper uniquely demonstrates that the generation capabilities of T2I models are correlated with the frequency of a given concept, which is novel and insightful. 3. The authors also introduce a new dataset that aggregates tailed concepts from Vision-Language Model pretraining datasets.

Weaknesses

1. I feel the authors are somewhat aggressive in their claim of a 'constant linear relationship' as stated in Fig 2's caption. There are noticeable dips in accuracy that are not well explained. Given that the authors can identify false positives using their frequency measurement pipeline and are confident about the frequency of a given concept, a true linear correlation should not show these dips. This observation is not adequately addressed by the authors. 2. The authors' claim that their work is the first to establish that pretraining datasets of VLMs are long-tailed seems overzealous. Prior work has demonstrated this fact [1], and the correlation between zero-shot accuracy and pretraining frequency has been established. Additionally, [1] analyzes both LAION-400M and LAION-2B, a superset of LAION-400M [2], whereas the biggest dataset evaluated in this work is LAION-400M. Furthermore, [1] has been incorrectly cited in the paper. 3. The authors present a challenging new benchmark but provide very limited information on how these concepts were collected. There could be multiple ways to sample 290 classes, which could lead to worse performance for CLIP compared to popular benchmarks like ImageNet. 4. Additionally, the authors could have provided some high-level information about which concepts are relatively less present, such as snakes, birds, airplanes, etc. This information could offer insights toward improving CLIP.

Questions

1. Can the authors provide insights about the classes used to create the dataset? Specifically, what was (a) the most frequent concept, (b) the least frequent concept, and (c) the average frequency of a concept? This would help understand the structure of the dataset, and how tailed the dataset is. 2. Can the authors present the standard deviation of each frequency bin? It would be interesting to see the variance as the frequency increases, as this may help explain the dips. CLIP might show less variance for relatively frequent concepts.

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

4

Limitations

1. While the authors highlight a significant problem, they do not propose any solutions to mitigate it. Offering potential solutions could have strengthened the paper. 2. The paper covers a broad range of topics, but it would benefit from a stronger focus on the generation aspect and dataset. Previous literature [1] has indicated that zero-shot recognition of CLIP models depends on the distribution of concepts in the pretraining data. The value of this work lies in highlighting that generative systems also face this issue. Emphasizing this aspect would make the paper easier to follow and enhance its impact. 3. Lastly, while analyzing the smaller YFCC or CC datasets is commendable, I feel that modern VLMs are pretrained on a scale of billions. Therefore, presenting this analysis on these smaller datasets has limited practical impact compared to insights derived from larger datasets like LAION or DataCOMP. Additionally, YFCC and CC datasets have experienced data loss over time, further diminishing their relevance.

Summary

This paper examines the relationship between the frequency of concepts in pretraining data and the performance of downstream tasks associated with those concepts. Extensive experimental results reveal a log-linear relationship, suggesting that exponential increases in data are necessary to improve zero-shot model performance. Additionally, the authors reconfirm the long-tailed distribution of concepts in well-known pretraining datasets, highlighting the challenge of handling rare concepts in foundation models. Based on this analysis, the authors provide a benchmark to evaluate model performance on tail concepts.

Strengths

- This paper significantly advances our understanding of data efficiency in multimodal foundation models, revealing a log-linear relationship between concept frequency in pretraining datasets and downstream task performance. - Extensive experiments are conducted, covering various pretraining datasets and downstream tasks. - The additional analysis provides valuable insights into the long-tailed distribution problem of foundation models. - The paper introduces a new benchmark called “Let It Wag,” crucial for evaluating the performance of multimodal foundation models on long-tail distributions. - The paper is clearly written and easy to follow, with most relevant details included.

Weaknesses

Nothing I can think of.

Questions

- I am curious about the phenomenon where the average accuracy drops after the concept frequency of 10^4 in Figure 2. Could the authors provide some examples and hypotheses about why this happens? - How much variance is there in each point of Figure 2? I am curious about how much the tendency varies depending on the difficulty of the concept, its size, or other aspects. - Does zero-shot performance predict few-shot or linear probing performance? I wonder if zero-shot models still learn important features for rare concepts, but these features are dominated by class imbalance in zero-shot tasks. - Typically, how long does it take to train a model on each dataset? (GPU hours)

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

The authors adequately addressed the limitations with future directions in Appendix N.

Summary

This work explores the extent to which zero-shot generalisation really occurs in large-scale in model that were trained on web-scale datasets. The approach taken relies on identifying concepts that are present in train and test data and evaluating concept frequencies and per-concept performance. From extensive experiments, the authors find that test performance of the models correlates strongly with the frequency of concepts seen during training.

Strengths

* The paper tackles an interesting and important topic - something that is often mentioned / discussed, but prior to this paper did not get proper analytic treatment. * Good and mostly clear presentation, clear methodology, extensive experimental evidence

Weaknesses

* My general feel after reading the paper is that the issue of zero-shot performance of model trained on web-scale data is not as bad as the paper (abstract) makes it out to be. For example, from Fig 6 we see that ImageNet and long-tail (e.g. Let It Wag!) benchmarks actually tend to agree as model performance improves. Similarly, we see from Fig 2 that larger models tend to have better performance even for concepts with low train dataset frequencies. * Presentation of the results is not always clear. For example, what are the different pannels in Fig 3 or Fig 5? * Results presented in the paper do not really study zero-shot performance. Specifically, in Fig 2 or 3 there appears to be no evaluated points with freq = 0. As such, the paper really studies few-shot performance as a function of the number of training examples. * Methodologically, I have an issue with way concept frequency in images was estimated - it relies on a pre-trained model (RAM++) to tag/classify images. This model itself likely suffers from training datasets biases and thus could miss or over- or under-estimate image concepts. This limitation also puts into questions the image-caption misalignment results presented in Tab 3.

Questions

See weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

See weaknesses, specifically whether zero-shot performance is really being evaluated in the paper, and how concept frequencies were determined for images.