CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning

Summary

This paper introduces a novel metric, negCLIPLoss, for selecting high-quality data. Additionally, the paper proposes a norm-based metric, Normsim, which offers an improved measure of data quality and is compatible with existing methods. Both negCLIPLoss and NormSim demonstrate significant performance improvements, outperforming state-of-the-art methods, while maintain low preprocessing time. Theoretical interpretations are provided for NormSim within the framework of a linear model.

Strengths

1. The paper is intuitive, well-motivated and well-written. 2. The proposed methods are simple and effective. 3. The experiments are sufficient.

Weaknesses

1. In Line 86, the authors assert that NormSim does not explicitly consider diversity but provide no further explanation. Since diversity is often linked to the generalization performance of models, it is unclear how the proposed methods implicitly connect with diversity.

Questions

1. The algorithm 1 requires the knowledge of the batch size B and the parameter $\tau$ from the teacher model. If the teacher model is private and both B and $\tau$ are not accessible (for example, only api is provided), are the proposed methods still workable? How critical is the batch size B to model performance? Can the parameter $\tau$ be estimated?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

NA

Summary

This work proposes two new approaches for data selection for vision-language pre-training. The first approach, negCLIPLoss, adds the contrastive loss as a normalization term on top of the existing CLIPScore metric. The second approach, NormSim, further improves the performance if examples from target task distribution are available. Together, the methods achieve state-of-the-art results on ImageNet-1K and 38 downstream tasks with DataComp-medium without any external data or model. Several important theoretical justifications and interpretations are provided for the methods.

Strengths

Quality: the quality is overall high. Without external resources (on which previous methods rely), the proposed approaches improve evaluation performances by 5.3% on ImageNet and 2.8% on average of 38 downstream tasks. Further, there are many theoretical results that focus on the guarantees of NormSim (though with strong assumptions). Clarity: this paper is very well written. It is well-motivated, the distinctions of previous approaches are succinctly laid out, the methods are well presented, and the results have a clear structure. Significance: this paper will bring significant impacts. The data selection problem has been increasingly vital for training higher-quality vision-language models. The proposed approaches, which focus on metrics instead of models or data, are compatible with different techniques that can be combined with advanced models in the future. The approaches also provide significant efficiency improvements (e.g., from 70 L40 hours to 5 L40 hours). The theoretical analyses can provide useful tools for future research as well.

Weaknesses

Quality: this is a minor complaint, but in Lines 229 - 230 the authors state that "the results of baselines on the leaderboard do not apply to our datasets, and we reproduce all the top baselines on the leaderboard with their public UIDs of the selected data" because some URLs of images become invalid. The leaderboard scores of baselines seem higher than the reproduced results in the submission. Could the authors also include the DataComp leaderboard results in the Appendix for fair comparison? There are also some minor questions below.

Questions

1. In Lines 135 - 136, the inaccessible batch division $B^*$ from teacher CLIP models is different from the actual batch $B_k$ in this work, in terms of both the actual image-text pairs and the batch size. Are there any potential theoretical guarantees or approximations to show that such a difference is reasonably negligible? 2. Could the authors further show the derivations of the discussions on the two important NormSim instances? 1) Lines 179 - 180 ($p=2$, equivalent to selecting a subset that aligns with the principal components), and 2) Lines 181-182 ($p=\infty$, a sample will be selected if it has high similarity to any target)? These may help other readers to understand NormSim better.

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors discussed the limitations of this work.

Summary

Data selection is crucial in the pretraining stage to clean the web-crawled, large, and noisy pretraining dataset. Typically, existing methods use embeddings to compute CLIPscore in order to assess the data sample alignment quality. This paper introduces two new methods to enhance this measurement: 1. negCLIPLoss: a better adjustment to reduce bias within a given batch. 2. NormSim: provides additional information when downstream tasks are known, allowing the selection of samples that are close to the target downstream tasks. Empirical results demonstrate that these proposed methods can be easily combined with existing filtering approaches. The authors also illustrate that their approach yields state-of-the-art results on the DataComp leaderboard.

Strengths

* Originality: Most of the work in data curation relies heavily on the original CLIP score. It's a new idea to adapt the CLIP score and elevate this measurement for better use. * Quality: The resulting performance is solid and achieves the top position on the leaderboard (medium-scale). * Clarity: The motivation behind the two approaches is clear, but some areas need further clarification. Questions are listed below. * Significance: Data selection in the pretraining dataset is important to the field, and they have demonstrated that their approaches are effective in achieving state-of-the-art results.

Weaknesses

1. I think we need more clarification on how to interpret the Top X% in three different metrics in Figure 1. Can the authors provide a more detailed description? Also, how is the R score derived from the batched data? How to find the proper batched data to use? 2. It seems that the negCLIPLoss is not incorporated into the training loss. We use it as a measurement when CLIP embeddings are provided. In this scenario, how do we determine the batch data, B, for subtracting the regularization term? Would the size of the batched data affect the measurement? The sampling method to find batched data is unclear to me. 3. I am unclear about the process for greedily selecting samples using NormSim, especially when the raw data pool is massive, and how to define the size of S. 4. I would suggest moving algorithm steps from the Appendix into the main body, or showing some steps in the main body. They are good at understanding filtering steps.

Questions

1. In Figure 1, R scores on the left side are in the top 100%, while on the right side they are in the top 10%. How should these be interpreted and categorized as underestimates or overestimates of quality? 2. When the downstream targets are not accessible, we may use the current filtered dataset as a reference, but how do we find the first-round reference dataset as a proxy to compute NormSim? 3. I would like to list several papers that I found and read for data selection. https://arxiv.org/abs/2405.15613, https://arxiv.org/abs/2401.12225, https://arxiv.org/abs/2302.03169, https://arxiv.org/abs/2401.04578, https://arxiv.org/abs/2404.07177

Rating

6

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

I didn't see any potential negative societal impact of their work.