Rebuttal
**1. It is not clear why it happens w.r.t "it is hard to align the image and text modalities."**
We explain why we think it is hard to align image and text modalities in the following sentences in the text:
While every image is metaphorically valued at a thousand words, it is often paired with a short, sometimes noisy, text that neither exclusively nor comprehensively describes it. For example, current vision-language models are good at associating images of cars with generic concepts such as “car”, “mechanics” or “road trip”, because these are common words paired with car images, but less at finegrained, instance-level, associations such as the specific brand, series or year of that car...
**2. Abstract mentions that "fine-grained entities which are rare". Having concepts rare seems like a different reason from "being hard to align image and text modalities". Which reason is more reasonable?**
Both are valid reasons. It is hard to align image and text, and it is especially harder for model to align concepts that are rare.
**3. "One caveat that we identify in this approach is that initial captions are augmented within their modality only, hence limiting the potential added-value brought by the retrieved items." Can authors clarify?**
Thanks for the comment. We will make sure to make this point clearer. In our paper, we show that it is more beneficial to combine text and image modalities. In K-Lite, the captions are only augmented with additional text and they are not combined with image modality to create cross-modal embeddings.
**4. The sentence is unclear -- "However, when crossing modalities, these representations are less successful in identifying suitable matches, such as finding the text with the closest representation to a query image representation."**
Thanks for the comment. We will make sure to make this point clearer. Crossing modalities means computing the similarity between images and text, i.e. two different modalities. We show that off-the-shelf CLIP models struggle with this. But it can do a much better job when retrieval is performed within modality (images retrieved given query image, or text retrieved given query text). See cross-modal search in Figure 4 for some examples. In this work, we address this issue by augmenting the input with retrieved images.
**5. The sentence is unclear -- "Through this process, we successfully transform the image and text representations into multi-modal versions, which significantly simplifies their alignment"**
When the query is an image, we augment its embedding with text embeddings corresponding to the similar images. On the other hand, for the text corresponding to the class names, we augment them with the image embeddings corresponding to the similar captions. We perform a similar process when the query is text also. Hence, we end up with multi-modal embeddings. The resulting multi-modal embeddings are easier to align.
**6. The paper does not discuss computation cost, complexity, etc.**
We do discuss the complexity and computation cost in Section 4.3:
Another limitation is that the performance gains of RECO come at the cost of increased inference time. In practice, we use a highly-optimized approximate k-NN algorithm (Guo et al., 2020). It takes about 14ms to query Webli (956M examples) with a single 512-d ViT-B/32 CLIP embedding. Using retrieval at inference time incurs an overhead of 25% compared to not using any retrieval at inference time (e.g. baseline CLIP model), but improves the accuracy by up to 10.9.
**7. The paper uses a dataset called WebLI and explains that it is a private dataset. However, how to access the private dataset? Does it mean that authors own the private dataset (indicating a leakage of author identities)? How to fairly compare methods if authors use a private dataset?**
WebLI is used as the dataset in a series of previous works, including PALI (Chen et al., 2023), PALM-E, CLIPPO etc. Moreover, we show in Table 6 of the Appendix that our method is not specific to WebLI, and we still have large gains when using publicly available LAION dataset.
**8. When discussing "Is the performance boost merely due to additional training?", the paper uses ResNet backbone and learns a transformer. Given that transformer might be better than convnets, it is questionable to claim using a transformer is a novel technique. Can authors discuss results if using a transformer backbone in CLIP along with transformer head for fine-grained recognition?**
The reviewer mentions the section "Is the performance boost merely due to additional training?". We do not use ResNet for results in that section. Instead, we use the ViT-B/32 transformer backbone of CLIP. This is clearly stated in the beginning of Section 4.3: “We use ViT-CLIP-B/32 throughout this section.”
We always make sure that our model uses the same backbone as the baselines we compare against. We will make sure that this becomes more clear in the paper.