TIPS: Text-Image Pretraining with Spatial awareness

Rebuttal comment to wggG

We thank the reviewer for the detailed comments. We are encouraged by the many positive remarks and will try our very best to address the reviewer’s concerns. In the following, we provide detailed responses to the weaknesses and questions from the reviewer: **W1**) *Potentially limited novelty.* First of all, we are glad to see the reviewer acknowledge solid results, despite the concern. While it is true that we are inspired by previous work’s explorations on synthetic captions and contrastive/self-supervised learning, we would like to highlight a few points: - It was previously not known that synthetic captions could benefit spatial understanding tasks, and the improvements are very significant (Tab 1). To the best of our knowledge, ours is the first paper to demonstrate this. - We propose a novel dual image-text embedding learning technique, which shows strong results by combining synthetic and alt-text captions in the right way. - We are the first to combine contrastive image-text learning with self-distillation and masked image modeling at the same time, showing that they can provide complementary strengths with positive synergy, and lead to outstanding experimental results in a broad range of tasks. **W2**) *Incremental gains over CLIP and DINOv2.* First, we would like to highlight that our main goal is to design a general-purpose method achieving strong performance across both spatial understanding and image-text tasks. CLIP and DINOv2 are disjoint models that lack capabilities on spatial understanding and image-text, respectively, while ours is the first model shown to provide strong results in both of these tasks. Second, we highlight some strong performance improvements of our method compared to recent work: - TIPS outperforms DINOv2 in fine-grained retrieval (UnED) by 9.2% absolute (Tab 2). Note that TIPS has close or better performance on other evals compared to DINOv2. - TIPS outperforms the same-size ViT-g EVA-CLIP in absolute terms by 5.8% on COCO I→T, 9.1% on COCO T→I, 1.4% on Flickr I→T, 5.6% on Flickr T→I, as per Tab 4 (*relabeled to Tab 3 in latest version*). These are substantial improvements upon recent work. **W3**) *Ablation study on impact of spatial coherence from the captions.* Thank you for the suggestion. We would like to point out that this ablation study is already provided in Tab 1, showing that spatial tasks (segmentation and depth) improve significantly when using synthetic captions. Simply replacing the web captions by PaliGemma-generated ones improves segmentation by 10.1 percentage points and reduces depth RMSE by 0.076, which are big positive gains (compare Tab 1 (A) vs Tab 1 (B) “PaliGemma captions”). Additionally, we provide more synthetic caption ablations in Table 5 (appendix), which help understand which components in the synthetic captions help spatial tasks – e.g., segmentation benefits significantly from listing the different objects in the image, while depth obtains a substantial boost when spatial relationships of scene content are described in the caption. We hope that these results help alleviate the reviewer’s concern but are happy to continue the discussion in case there is any additional feedback. **Q1**) *Ablations isolating the impact of synthetic captions on spatial tasks.* This is the same as **W3**, see answer above. **Q2**) *Alternative ways of introducing spatial awareness besides synthetic captions and masking.* Yes, additional ways could include the use of dense annotations such as boxes and masks. Since these are generally expensive to collect, one possibility would be to use high-confidence boxes and masks produced by state-of-the-art off-the-shelf models as pseudo ground-truth annotations. We can also leverage synthetic captions of boxed regions to obtain richer grounded supervision, e.g. “a yellow city bike” instead of simply “bike” (which is usually what one would obtain with standard class names). In our paper, though, we aimed to keep our method as simple as possible without requiring any additional expensive annotation, which could complicate the model design, but this could be a fruitful research direction. Additionally, we plan to explore modifications to our training strategy to enhance spatially-grounded multimodal learning. For example, one can leverage the text embedding to find the most text-aware patches in the image, and preferably mask them to incentivize the model to learn better visual representations.

Rebuttal comment to BGwY

We thank the reviewer for the detailed comments. We are encouraged by the positive evaluation of our work. Some of the weaknesses and questions raised in the review suggest the need for additional experimental studies. We are currently working hard on experiments which could provide results to help alleviate the concerns, and will send an update in this comment thread once the results are available. In the following, we provide detailed responses to the weaknesses and questions from the reviewer: **W1**) *Potentially limited technical contributions.* Firstly, we are glad to see the reviewer acknowledge that the proposed model effectively and adequately solves spatial awareness limitations. Our work does leverage learnings from previous work in representation learning, however we would like to highlight a few points: - It was previously not known that synthetic captions could benefit spatial understanding tasks, and the improvements are very significant (Tab 1). To the best of our knowledge, ours is the first paper to demonstrate this. - As the reviewer points out, we propose a novel dual image-text embedding learning technique. To emphasize, our experiments show that combining synthetic and alt-text captions in the right way helps with a variety of downstream applications, for both dense or image-level prediction. - Our method is the first to combine image-text contrastive learning with masked image modeling and self-distillation at the same time, showing that they can provide complementary strengths with positive synergy, resulting in outstanding experimental results in a broad range of tasks. **W2**) *Potential issues with claim in Line 300: missing FLIP/MaskCLIP citations?* We thank the reviewer for the detailed comments on this point, which will help us improve our paper and better position our work against previous methods. Let us discuss in detail: - The reference “Scaling Language-Image Pre-Training via Masking” pointed out by the reviewer corresponds to the FLIP paper, which is already discussed in the submission (see reference Li et al., 2023). FLIP has a very different goal from ours, since they proposed to combine contrastive learning with masking without any reconstruction loss, aiming only at efficient language-image training (no spatial awareness goal). We believe that the current version of our manuscript provides sufficient discussion regarding FLIP, but we are open to any additional suggestions from the reviewer on this point. - Regarding the “MaskCLIP” reference: indeed, we have missed it in the submitted version of the paper, and we will fix this. We will upload a new PDF version of the paper in the next few days with this reference included, including relevant discussion. In terms of differences from our TIPS method compared to MaskCLIP, we would like to emphasize: i) our approach goes beyond masked image modeling to also include self-distillation losses, which we show important via ablations; ii) we show the power of synthetic captions for spatial understanding, which is not an aspect studied in their work; iii) we aim for off-the-shelf usage for many vision tasks, which is different from their goal (their dense prediction results are mainly in the setup of full model fine-tuning, covering only a small number of dense tasks). Additionally, we can directly compare some experimental results of our method against MaskCLIP: while our ViT-B model achieves 89.2% on Flickr I→T retrieval (Tab 1), MaskCLIP’s ViT-B achieves only 70.1% (Tab 5 in their paper). And on Flickr T→I, we achieve 77.3%, compared to MaskCLIP’s 45.6%. This shows that our full training recipe for TIPS tends to achieve significantly better results than MaskCLIP. **W3**) *Experiments on fine-grained spatial awareness under multimodal settings.* We are working on this experiment currently and will report results in this comment thread once they are available. **Q1**) *How SSL design choices affect image-text representation learning.* We are working on this ablation experiment currently and will report results in this comment thread once they are available. **Q2**) *Visualization of the attention maps of the two different [CLS] to see their focus areas.* We are working on this experiment currently and will report results in this comment thread once they are available.

Rebuttal comment to FXGp

We thank the reviewer for the detailed comments. We are encouraged by the positive evaluation of our work. Here we respond to the weakness and the question raised by the reviewer: **W1**) *Formatting.* We agree that there is room for improvement on the paper formatting, in particular around Figures 1 and 2, as pointed out by the reviewer. We will make changes accordingly and upload a new PDF version of the paper in the next few days. **Q1**) *Model/data release.* We are planning to release the pretrained model together with the final version of the paper. We are currently following the model release process required by our organization and expect that all approvals will be obtained in time. Regarding the dataset: our curated dataset is part of a much larger corpus of images that has not yet been publicly released. Therefore, unfortunately, our organization prohibits its release.

Rebuttal comment to 6Wzj

We thank the reviewer for the detailed comments. We are encouraged by the positive evaluation of our work. The two weaknesses raised in the review suggest the need for additional experimental studies. We are currently working hard on experiments which could provide results to help alleviate the two concerns, and will send an update in this comment thread once the results are available. In detail, the two weaknesses were: **W1**) *Comparing against the EVA method of combining contrastive image-text learning and masked image modeling.* The reviewer mentions EVA-CLIP, and by that we understand the suggestion to compare against the sequential method of i) contrastive, then ii) MIM reconstruction, which was originally presented in the first EVA paper (reference “Fang et al., 2023” in the paper). We are working on experiments to compare this approach against our method. The reviewer mentions concerns of potential unstable training with our approach, but we do not observe this in practice. **W2**) *Improvements potentially coming from better data (use of WebLi).* We would like to point out experimental results in the submitted version of the paper which already suggest that the gains are mainly coming from a better training method, rather than better data. Table 6 (in the appendix) ablates dataset versions according to our curation pipeline, showing that our curated dataset leads to moderate gains in NYUv2 (from 0.698 to 0.62 RMSE), when using a standard CLIP method. However, Table 1 indicates a much larger gain by changing from CLIP to our method, from 0.62 to 0.478 RMSE, which is roughly 2X the gain from data curation. Additionally, we are working hard on providing experimental results with a public dataset, as suggested by the reviewer. This is a significant engineering task, which consumes a very large amount of resources, not only for training, but also for downloading, curating and re-captioning the large-scale datasets. Given that WebLi and the other public datasets are collected in similar ways, we believe that other datasets of the same size will yield similar results. Nevertheless, we are doing our best to provide results on this as soon as possible.