Summary
This paper presents Dual Risk Minimization (DRM), a novel approach that combines empirical risk minimization (ERM) and worst-case risk minimization (WRM) for fine-tuning the CLIP model while maintaining its out-of-distribution robustness. The idea is to create a classifier that utilizes concept descriptions of each class generated by large language models (LLMs). This classifier helps “pull out” core features from the image embeddings, serving as a regularizer to enhance the robustness of CLIP fine-tuning. Moreover, the paper introduces a min-max normalization technique to address a caveat in the regularization. Experimental results demonstrate the effectiveness of the proposed DRM, yielding promising in-distribution and out-of-distribution accuracy.
Strengths
- The paper is well-written and generally easy to follow.
- The paper brings an interesting viewpoint for CLIP fine-tuning that separates the image features into core (content) and non-core (style) features.
- The experimental results look promising, especially on the two specialized datasets, iWILDCam and FMoW.
Weaknesses
- At Line 199, the paper claims that the LLM-generated concept descriptions can “pull out” the core features from the image embeddings. However, this claim is only demonstrated by the few examples shown in Figure 2, which is insufficient. Moreover, the stochastic nature of LLM generation can further complicate the matter. The paper lacks a thorough analysis of the reliability and robustness of the core features obtained through this proposed method.
- At Line 217, the paper mentions that the affinity between $x$ and any other class $y'$ should ideally be zero, thereby motivating the modification of the second term in Eq. 10. However, this might ignore the fact that the ground-truth class $y$ exhibits varying degrees of similarity with different $y’$. It remains unclear how the proposed method can achieve better performance, as shown in Table 3, while disregarding the learning of such inter-class similarities.
- The second term in Eq. 10. is similar to knowledge distillation, as acknowledged by the paper at Line 207. However, the paper neglects to discuss or compare this term with other robust fine-tuning methods, such as [1], which also incorporate knowledge distillation.
[1] DELTA: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks, ICLR 2019.
Questions
Besides the weakness shown in the above section, please also see the following questions:
- Given that the paper suggests that the classifiers built with LLM-generated concept descriptions have the ability to extract core features, could we simply fine-tune the CLIP model utilizing these concept descriptions instead of using the one with default descriptions? If we do so, could standard fine-tuning with ERM already maintain its robustness?
- What would be the in-distribution and out-of-distribution accuracy for the concept description classifiers? Given that these classifiers aim to extract more core features, would they be more robust to distribution changes, even without fine-tuning?
- How does the min-max normalization affect the training on long-tailed datasets, like iWILDCam? Specifically, since the min-max normalization computes $x$ with all images in a class $y$, could tailed classes result in less stable normalization because they have much fewer samples?
Limitations
The paper acknowledges that a potential limitation of the proposed method lies in the potential limited domain knowledge of LLMs in specific domains. To better illustrate this limitation, it would be beneficial for the paper to provide concrete examples, such as failure cases, that show how the method behaves when it reaches its limits.