T-MARS: Improving Visual Representations by Circumventing Text Feature Learning

We thank you for your time, and valuable feedback, acknowledging “very strong results” and “promising scaling trends”. We are glad to see your recognition of our pilot study as a “significant step towards understanding multimodal datasets”. Below we provide clarifications to the questions raised. ### **Why is exactly 50% of the data filtered** While designing the approach and experimenting on the LAION-400M dataset, we tried three choices for the amount of data to be filtered— 40%, 50%, and 75%, with 50% working out to be the best. Note that these experiments further filtered an already filtered LAION dataset. One can definitely vary this and use other filtering metrics, say based on a threshold CLIP score after text masking. For example, on DataComp (filtering on the common crawl), we use a CLIP-similarity threshold of 0.281, which is similar to that used in prior work, such as LAION filtering. Fine-tuning the threshold may bring further gains, but running such parameter searches is beyond our compute scope. We have clarified this in the updated manuscript as well. ### **Downstream Datasets** We apologize for the oversight on our part and have updated Table 2 (Datacomp results) with an evaluation on all 38 downstream tasks. We have fixed the typo on the number of datasets in Table 1 as well. Our experiments in Table 1 were on quite a small scale initially, due to which we did not evaluate on some tasks like WILDS in Datacomp on which all models get random performance. ### **Some missing baseline from DataComp paper** The DataComp paper also considered an ‘image-based’ filtering approach that uses information from the Imagenet dataset (ImageNet $\cap$ CLIP Score). T-MARS outperforms this baseline as well, but we do not list it because we wanted to only consider methods that do not use any information of the downstream evaluation dataset to provide for a fair evaluation between all baselines. --- We again thank you for your time and hope that we have addressed the concerns. We are happy to provide any further clarifications if requested.

Thank you for your time and valuable feedback. We are happy to see that you appreciated the simplicity of our approach and the promising empirical results on a variety of tasks. We address all of your concerns below: ### **Why text-only images hurt, but mislabeled data does not?** First, we want to clarify that the CLIP similarity metric is already good (not perfect) at capturing mislabeled data. In fact, most image-caption pairs below a CLIP score of 0.2 are already mislabeled or have random captions. They continue to have a low CLIP score even post-masking and are hence still removed by T-MARS. The natural CLIP score does not, however, help remove text-dominant image-caption pairs. As seen in our synthetic experiment in Section 6, mislabeled examples and text-dominant image-caption pairs hurt visual representations almost equally which supports your insight. However, we also highlight that in Section 4.2, we proposed two competitive data filtering baselines, C-RHO and C-SSFT, that filter out the mislabeled and ambiguous examples. These baselines have been proposed by drawing insights from the long line of work around mislabeled and hard example mining in the supervised classification regime. Further, as shown in Table 1 and Table 2 (see T-MARS $\cap$ C-SSFT row), taking an intersection of data filtered by T-MARS (text-dominated images) and the proposed baselines (say C-SSFT, which removes additional mislabeled samples that CLIP score didn’t capture) gives additive gains. This highlights the importance of filtering out both the text-dominated and mislabeled samples, as you rightly suggested. ### **Larger pilot study beyond 500 images** We agree with the reviewer that a larger scale extension of our pilot study will definitely be helpful for the community. In Appendix C.2, we talk about statistical estimates of error rates for the estimated proportions. Assuming that the sampling is unbiased, 500 samples are enough to estimate the proportions with a 5% margin of error (95% confidence). However, to verify whether the sampling itself is representative of the population (the unbiased assumption above), we performed a similar pilot study over 500 additional examples (once again randomly taken from the LAION dataset). The table below gives the estimated and the actual count of data points from the various categories. | | Category 1 | Category 2 | Category 3 | Category 4 | Category 5 | | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | | Estimated (based on previous study) | 25 | 225 | 50 | 100 | 100 | | Actual (new 500 examples study) | 24 | 232 | 53 | 96 | 95 | ### **Will there be gains on CC12M** CC12M is a small and already well-curated dataset. Hence we do not think that there will be significant gains using T-MARS or any other baselines for data filtering here. However, note that such small datasets cannot be used for training large-scale state-of-the-art CLIP models, which use noisy webscale image-caption datasets like Common Crawl. Our work focuses on building scalable and efficient curation approaches for these webscale datasets. ### **Training on masked images** This is indeed an interesting question. We chose not to train on masked images to avoid distribution shifts and loss in performance potentially due to the aberrations introduced by masking (although they are indeed quite negligible, see Appendix E). Moreover, there is no reason to train on masked images from Category 5 (text only) because they will be data points where the text and the (now masked) input are uncorrelated. As far as Category 3 is concerned, the uncorrelated texts are already ignored by the model (as also seen in Section 6). Finally, let us come to the most important category in question (Category 4 with both text and visual features). Currently, we retain the unmasked input in such categories (that have both text and visual features). One proposal would be to only retain the visual features in these (as you suggest). We believe that keeping the text features can be helpful for retaining an understanding of certain visual features as well, such as brands and trademark logos (such as the Superman logo) which can be in-part textual, but also have a distinct visual pattern. Our masking algorithm will mask these patterns and potentially hinder the model’s knowledge. ### **Section 6** We acknowledge your feedback about the difficulty of parsing Section 6. **We have rewritten the same** by focussing on the insights and results in the main paper and moving details to the Appendix that were previously hindering clarity in the interest of brevity. We hope the updated version reads better. --- We again thank you for raising these interesting questions and hope that the clarification helps you assess the work more positively. We are happy to provide any further clarifications if requested.

Dear reviewer, We worked on addressing all of your concerns in the response above. We hope to hear back from you. As a quick summary, in our rebuttal: - we highlighted that T-MARS already removes most of the mislabeled images and the paper also proposed 2 baselines focused on filtering out mislabeled images. Interestingly, T-MARS + proposed baselines give additive gains, highlighting the importance of removing mislabeled samples, as you also guessed. - we conducted an additional pilot study, to show that our estimates for different data type proportions are indeed within the error range estimated statistically (i.e. 2-3%). - we have rewritten Section 6 (on estimating the utility of each data type), focusing on the insights and key takeaways, as requested. We believe that these points address all the concerns raised and please let us know if there is anything else that you are still concerned about.

Dear reviewer, Since today is the last day of the author-reviewer discussion period, we kindly request you to please take a look at the rebuttal above, which we believe addresses all the concerns raised by you.

We thank the reviewer for going through the rebuttal and raising the score. We will definitely work on incorporating the above suggestion about mislabeled examples in the manuscript. Authors

Thank you for your time and valuable inputs. We provide clarifications for the concerns raised in your review: ### **Directly remove all the text in the images** The first step of TMARS, i.e. text-detection can indeed be seen as trying to identify and remove all the texts in the images. However, note that TMARS then filters out the images with low CLIP score after masking. This is because, as outlined in our pilot study (Section 3), a significant number of images primarily contain text as the sole feature correlating with the caption. It would be suboptimal (waste of compute) to retain these images and spend training compute on them, as after masking, they do not have any visual features (correlated with caption) that could possibly help the model to learn better visual representations. ### **Experiments on more datasets** We point to Table 2, where we also show results on the DataComp, a latest benchmark for multimodal data filtering. For example, on the medium scale of datacomp, TMARS observes around $6.5\%$ gains on ImageNet zeroshot accuracy compared to CLIP filtering and $4.4\%$ compared to TextMatch[1]. The goal of our work is to improve training on noisy, web-crawled datasets, and in that regard, we have shown effectiveness on both, filtering the common crawl, and pre-filtered LAION datasets. The common crawl is pretty much all the data on the web, and captures any subset of data that may be created out of it. We hope this helps acknowledge the significance of our results. ### **Writing** **We have re-written** parts of the paper that we felt might have been hard to follow. This especially included a re-write of Section 6. If you have any particular comments on any other parts of the paper, we would be happy to make amends. Thanks! --- [1] Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. arXiv:2301.02280, 2023.

Thank you for your time and valuable feedback. We are happy to note that you appreciated the simplicity of our method, supported by promising results. We provide clarifications to your questions below: ### **Accuracy as the only metric** Sorry for not being precise about this, but we follow the standard evaluation metrics as used in the DataComp benchmark (https://github.com/mlfoundations/datacomp/blob/main/tasklist.yml). For example, on retrieval tasks (MSCOCO and Flickr) the metric is mean_recall@1 and on the WinoGAViL dataset it is jaccard_score. Further, we have updated Table 2 in the paper with an additional column of average performance across all the datasets in DataComp, which involves F1 score as the metric for some of the class imbalanced datasets like wilds-iWILDCam and worst region accuracy as the metric for wilds-FMoW. ### **Distribution of scores before and after masking** Thank you for the suggestion. Recall our 500-example pilot study where we hand-labeled examples into various categories, like those with only visual features or those with only text features. We calculated the CLIP similarity score of these image-caption pairs, before and after masking the text in the respective images. **A histogram of this distribution of scores (before and after masking) is now uploaded in Appendix F**. Our observations indicate that after masking, the CLIP score of images with just text features (OCR) drops significantly (indicating they will be filtered out by T-MARS). At the same time, CLIP scores of images with only visual features stay nearly the same. This demonstrates the efficacy of the filtering method for removing samples with only text features. ### **Data augmentation after filtering** We do not perform any additional data augmentation after filtering out the bad data to ensure a fair comparison with the baselines. We follow the standard and widely used CLIP training implementation given here[https://github.com/mlfoundations/open_clip]. Indeed, data augmentation may provide additional improvements in visual representation, but our work aims to improve representations by intervening on the data, not the training process. ### **Does inpainting alter original data?** If the text overlays a patch with significant visual features, inpainting might create some aberrations. However, in general, we observe that the text detection algorithm (FAST[3]) gives very tight bounding boxes around the text, leading to no major aberrations of visual features when inpainting the text area, serving our main purpose in this work well. In Appendix E, we have added some pairs of original and masked images to highlight the same. Additionally, the histogram of CLIP score distribution in Appendix F (which you suggested) provides evidence that the method works well at masking out text features, thereby, lowering CLIP scores of only the images with just text features. However, we agree with you that exploring more nuanced ways to remove the text from the image is an interesting direction for future work. ### **Impact of percent of Masking** To decide the threshold of image-caption pairs to be removed post-masking, we use a CLIP-similarity threshold of 0.281, which is the same as that used in prior work, such as LAION filtering. Fine-tuning the threshold may indeed bring further gains, but running such parameter searches is beyond our compute scope. ### **New Discussion: Appendix G (Other ways to resolve text overlap with caption)** We have added a discussion section on other ways to circumvent text feature learning in Appendix G (Page 26 and Page 27), as requested by you. We give a brief summary here: - One can remove all inputs with any text features matching the caption. However, this is suboptimal as it removes datapoints with both visual and text features [1]. - Alternatively, one can consider a much more expensive process of modifying the input captions of the dataset. One way for the same would be to remove identified text OCR from the input captions. Another way would be to generate new captions altogether, as recently explored in [2]. - Finally, one could also consider modifying the input images to in-paint the OCR region. Note here that in T-MARS, we chose to use the masked images just to calculate new CLIP scores for filtering. A more elaborate discussion is provided in the appendix. --- We again thank you for raising these interesting questions and hope that the clarification helps you assess the work more positively. We are happy to provide any further clarifications if requested. [1] https://arxiv.org/abs/2301.02280. [2] https://arxiv.org/abs/2307.10350. [3] https://github.com/czczup/FAST.