Multilingual Diversity Improves Vision-Language Representations

Summary

The paper conducts a systematic study to explore the performance benefits of using translated non-English data for English vision tasks. By translating multilingual image-text pairs from a raw web crawl to English and re-filtering them, the authors show that continual pre-training on this data increases the performance of the English model on a diverse set of tasks with a single model.

Strengths

* The authors show that by incorporating translated multilingual and multicultural VL dataset can improve English model's results empirically on English only tasks. * The paper provides interesting analysis showing that using diverse training data improves benchmark results that require geographical or cultural diversity.

Weaknesses

- The improvement from using translated data diminished as the training converged, which is not surprising. Similarly, using more training data to improve the evaluation results on downstream tasks is also expected. Hence, the novelty of insights is limited in this paper. - The paper only evaluated on one LLM, which more evaluations are required to validate the generality of the claim of this paper.

Questions

Possible experiments to make the paper better: - Add more multicultural evaluation datasets - Add more evaluations to show the benefits of using diverse training data could benefits multiple models sizes and types - Perform similar training and evaluation in other languages, to show all languages would benefits from diverse training images and captions

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

While this research may be interesting for those working primarily on monolingual research (i.e., English), its framing should be more sensitive to the many efforts related to multilingual and multicultural NLP. Personally, I think the benefits of using diverse images and captions can be achieved by collecting diverse images and paraphrasing captions or through better sampling of the training data, rather than just using multilingual or multicultural datasets and translating them. As a community, we already lack people working on multicultural and multilingual models, yet this work uses data and research from other cultures to improve **English-only** models (although with good intentions). Perhaps there could be some discussions on this.

Summary

This paper points out an important issue in current training of CLIP models---the push for including more english-centric / english-only data in the pretraining dataset. The paper points out that this is mainly driven by the downstream evaluation test-beds primarily being english-focused and hence the need to include multi-lingual data has not been found so far. The paper conducts a thorough study of using multilingual data into the pretraining datasets of CLIP models by using caption translation models. The paper evaluates and shows improvements on several downstream datasets from the 38-Datacomp benchmark evaluation suite, and provides analysis on the key differences between multilingual and english-only pretraining datasets.

Strengths

- The paper is very well written and easy to follow - All the claims in the paper are verified with substantial empirical evidence - The paper's main message is adequately represented throughout the paper without any distracting claims, hence all the experiments done in the paper flow very smoothly - The paper points out a very important issue in current pretraining datasets of CLIP models, and showcases a simple intuitive fix for future models and pretraining datasets

Weaknesses

- The paper could benefit from some additional analysis. I note down some analayses that I think might strengthen the paper below. - For a particular concept is it possible to showcase the diversity afforded by looking at only English data and multilingual data? For instance, take the concept of “stove”. You could perhaps manually take 10-30K images of a stove (assuming that many exist), by filtering the texts for this particular concept, and do the same for both English only and multilingual data. Then, quantify the diversity of images, by perhaps taking a strong Dino-v2 pretrained encoder and then clustering them, and perhaps looking at some intra- and inter-cluster distance metrics. I think this would nicely validate the qualitative visualisation in figure 1 with some empirical evidence that multilingual data indeed boosts the diversity of visual concepts in pretraining datasets. It would be great to see this analysis on perhaps one or two concepts. Why is this an important analysis? My reasoning would be that while it is totally plausible that multilingual concepts add diversity, it is unclear to me how much of this is actually true? Since it is plausible that English webpages also contain such diverse visual concepts, I think it would be nice to see if there are “new visual concepts” being included in the pretraining corpus, or is it mostly that these “culturally-diverse” visual concepts were present in the english-only pretraining datasets, but we just boost up their frequency in the pretraining dataset? I think either of the two cases is a valid justification for the claim that multilingual data increases diversity, but it would be nice to have a precise answer for what exactly out of the two is the case. - The main claims of the paper that multilingual data improves VL representations is sufficiently backed up, but benchmarking on culturally diverse / multilingual-sourced data is still limited. The paper only considers GeoDE and DollarStreet for evaluation. I would recommend to also benchmark and report the performance on some more culturally and geospatially diverse datasets like [1,2,3,4]. Some of these datasets might need to be ported into a classification / retrieval format, but in general I think this is an important experiment to do to further validate the important of multilingual-sourced pretraining data. - By filtering on top of the translated image-text pairs, it seems likely that the training data diversity increases, and potentially train-test similarity [5] also increases. This could be an added confounder in the takeaways of the paper. Could the authors comment/discuss this a bit more in the paper? - The analysis in the paper would further be improved by checking how the distribution of CLIPScores look before and after the translation. Potentially there would be a shift to the right in the similarity distribution, but it would be a good analysis to include for better intuitions on what might be good filtering thresholds if such a model were to be used in the future for vector arithmetic / data filtering itself. [1] Kalluri et al, GeoNet: Benchmarking Unsupervised Adaptation Across Geographies [2] Weyand et al, Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval [3] Yin et al, Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning [4] Romero et al, CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark [5] Mayilvahanan et al, Does CLIP’s generalization performance mainly stem from high train-test similarity?

Questions

I have a few additional questions/comments which I include below: - How do you estimate the english part of the web-crawl? You mention it is in 1/3rd in the introduction. - Is there an explanation for why filtered translated captions might improve english-centric performance over the filtered raw captions? This seems counter-intuitive given as you say in the paper, that ImageNet is mostly western-centric. This also seems to contradicts the results of the No-Filter paper [1]. - Could the model trained on this filtered raw subset itself be used as a DFN, and potentially further improve the performance when training on the raw multilingual pool? The idea being that you might have to do less tuning on the filtering threshold since the model is more robust to noise from the translated captions. - In fig 3, are the results from the "filtered raw captions" and "filtered translated captions" methods from tab 1? Doesn’t seem so since the performance improvements on the 38 tasks mentioned in the caption of fig 3 is 1.5% whereas the performance improvements from tab 1 is only 0.9%. - Why does performance on Food101 go down? Is it because all the food classes included in Food101 are primarily English-centric? I am certain that is not the case. I would be very interested in a more detailed analysis of why the performance on Food101 goes down when we include more multilingual data since that is one aspect that I would have thought improves quite a lot. [1] Pouget et al, No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors are explicit about their limitations. Any further limitations I think are mentioned in the weaknesses section above.

Summary

The authors investigate whether *multilingual* vision-language data improves the *English-only* performance on a model in vision-language tasks. They translate captions from DataComp from English to other languages and train a CLIP model on these multilingual captions. They find that this action boosts performance on both English-only tasks like imagenet matching and geographically diverse tasks like GeoDE.

Strengths

This is a super efficient way to gain a performance boost on DataComp (for revs who might not be aware, this is a cool benchmark for assessing efficient training of vision-language representations evaluated over a bunch of downstream datasets within a fixed compute budget). Any method that boosts DataComp performance across so many tasks for free (just translation instead of collecting new captions) is awesome in my opinion. Performance boost on GeoDE (geographically sorted img classification) is improved considerably in all regions, justifying the purpose of the intervention.

Weaknesses

Unclear how the proposed automated translation pipeline ties in to the original motivation (culturally specific items not being captured in English data) Could benefit from a bit deeper description of what the GeoDE task is for unfamiliar readers to improve reach.

Questions

Figure 1: where are these examples from? Are they actually in the datacomp datasset? Doesn't translating datacomp captions into other languages keep the learned representations in the "English concept space" of sorts?

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes, limitations are sufficiently covered.

Summary

In this paper, the authors conduct a thorough exploration of how multilingual image-text pairs benefit English vision tasks. They first present how to effectively utilize translated data to improve performance on standard vision tasks and derive valuable conclusions through detailed ablation studies. Second, they illustrate the differences in image and text distributions between English and non-English image-text pairs. These findings highlight the potential of leveraging multilingual datasets to enhance the robustness and accuracy of vision models.

Strengths

1. The paper presents effective strategies for utilizing translated data to enhance performance on standard vision tasks 2. Detailed ablation studies are carried out to derive valuable conclusions, which help the community understand the impact of various factors. 3. The paper is well-written and easy to follow.

Weaknesses

1. No quantitative assessment for the quality of translation. 2. Only evaluation on representation task. Does such data strategy also work for the training of Multimodal LLM?

Questions

No

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Response to author rebuttal

I thank the authors for their thorough analysis and further experiments. - The clustering experiment now adds a very strong demonstration that the non-English data improves the diversity of visual samples even for concepts that are already present in the English pool. This is a great result! - GLD experiments look great and provide additional validation for the significance of including non-english data. - The points discussing the differences between your and the No-Filter paper are quite valid and important. I would encourage the authors to add this as an explicit discussion section either in the related works or the appendix. After reading all the other reviewer comments and the author's responses to them, I am inclined to strong accept this paper as it makes a significant contribution to the data-centric VL community and provides relevant, practical insights. I am increasing my score to 8.

Rebuttal follow-up

We thank reviewers again for their time. As the discussion period is drawing to a close, we would like to ensure we have addressed all of the concerns. Let us know if you you have any further questions/ thoughts after our rebuttal, we would be happy to engage in further discussion. -- Paper authors

Re

Thank you very much for the additional clarifications. I have revised the score to reflect my current assessment of the paper.