Summary
The proposed method leverages language priors for better downstream adaptation and generalization of CLIP [37], which is similar to CuPL [36] that utilizes prompts generated by a LLM (e.g., GPT-3) for zero-shot image classification using CLIP. Unlike CuPL, the proposed method aggregates multiple LLM-generated or human-generated prompts to construct an aggregated text feature. Using the aggregated text feature as a guidance, the authors adaptively transform CLIP image features to tackle few-shot image classification, image-to-text retrieval, image captioning, or VQA. A notable difference from CuPL is that the proposed method does not run an inference of LLM at test time.
Strengths
This paper has the following strengths:
(i) The proposed method is timely, and seems to be effective and efficient. By aggregating text features produced from LLM-generated or human-generated prompts and then using the aggregated feature as a guidance, the proposed method could effectively leverage language priors for better downstream adaptation and generalization of CLIP. After the distillation of rich language knowledge during the training time, the proposed method does not need to run an inference of LLM at test time.
(ii) The authors validate the effectiveness of the proposed distillation method for downstream tasks (few-shot image classification, image-to-text retrieval, image captioning, VQA). It demonstrates that the learned adaptive transformation guided by language priors actually improves the adaptation and generalization of CLIP for downstream tasks.
Weaknesses
This paper has the following weaknesses:
(i) This paper lacks the justification of how the adaptive transformation of CLIP image features, guided by an aggregated text feature, overcomes the image-text modality gap which exists in CLIP vision-language space [A]. Despite the modality gap, the proposed method simply encourages transformed CLIP image features to be located nearby an aggregated text feature (using L2 distillation loss).
[A] Liang et al., “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning”, NeurIPS, 2022.
(ii) The proposed method might not be able to improve the model robustness to distribution shifts which are difficult to be explained by natural language. For example, the Terra Incognita dataset [B] contains distribution shifts caused by different camera locations in wild environments. The widely used benchmark of in-the-wild distribution shifts, WILDS [C], also contains such distribution shifts. Since the proposed method leverages only language priors, it might not be good at handling those distribution shifts.
[B] Beery et al., “Recognition in Terra Incognita”, ECCV, 2018.
[C] Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts”, ICML, 2021.
(iii) This paper lacks experimental results obtained with different distillation loss coefficient values, although the distillation loss is the main contribution of this paper. According to L191-196, it simply states that the proposed model is not sensitive to different coefficient values. Without relevant experimental results, it is difficult to evaluate this statement.
Questions
(i) Despite the image-text modality gap in CLIP vision-language space [A], the proposed method simply transforms CLIP image features using an aggregated text feature. Does “h(x)” overcome the image-text modality gap? How do the authors deal with the modality gap? If the proposed method fails to overcome the modality gap, the authors should justify why the proposed method leads to better downstream adaptation and generalization of CLIP.
[A] Liang et al., “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning”, NeurIPS, 2022.
(ii) According to L176-180, the authors claim that learning h(x) is much more parameter-efficient than learning a sequence of token embeddings. Is it true? For example, suppose that we have 16 learnable word tokens, where the dimension size of each word embedding is 512. In this case, there are 16 * 512 learnable parameters. In contrast, as described in L169-170, “h($\cdot$)” is a network which consists of two fully connected layers. Given that the dimension size of CLIP features is 768 or 1024, the network seems to have 2 * 768 * 768 or 2 * 1024 * 1024 learnable parameters. What does it mean that learning h(x) is much more parameter efficient than learning a sequence of token embeddings?
(iii) To validate the statement in L191-196 (“Performance is found to be not very sensitive to the $\lambda$ value in a wide range.”), the authors need to provide experimental results obtained with different $\lambda$ values.
Limitations
Yes, in page 9 and 16.