Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective

Summary

This paper investigates the two different misalignment issues between CLIP and downstream tasks, i.e., task misalignment and data misalignment. The author designed several experiments that demonstrated that over-fitting occurs when tuning with the learnable prompt. They propose the Causality-Guided Semantic Decoupling and Classification(CDC) method to mitigate the impact of task-irrelevant generative factors on downstream tasks. The extended experiments demonstrate that the proposed CDC method is effective.

Strengths

This paper investigates the difficulty of adapting CLIP to downstream tasks via two-level misalignment, which is a bright idea to help the community understand the working mechanism. Then the author provides a comprehensive experiment that reveals how the overfitting occurs and impacts the recognition of new classes. The author uses the perspective of causal inference to alleviate the data misalignment and proposes CDC with front-door adjustment for implementation, which predicts with explicit evidence.

Weaknesses

1. There is no obvious evidence to show that the CDC will improve the prediction in certain cases, which category previous wrong and correct with CDC. It would be better if several compared failure cases to demonstrate that the CDC can solve the question at which level and which case still needs more advanced methods. 2. It is interesting how the misalignment between CLIP and downstream pattern will change as the model’s capabilities increase, such as ViG-Large, and whether the more powerful model can solve the misalignment problem. Hence, it will be better if some comparisons of models can be added. 3. As we all know, MLLMs already sweep through the multimodal community, it will be better if expand some discussion about the misalignment in this paradigm and current method whether easily general to that. 4. How many parameters are tuned during downstream adapting? Compared to the pre-trained model, what is the ratio of tuned parameters?

Questions

see weaknesses

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

This paper addresses the two-level misalignment (task and data) issue in adapting CLIP to specific tasks. The authors develop a structural causal model to analyze CLIP's pre-training and adaptation processes, revealing how task-irrelevant knowledge interferes with predictions. To mitigate this, they propose Causality-Guided Semantic Decoupling and Classification (CDC), which implements front-door adjustment. CDC includes Visual-Language Dual Semantic Decoupling (VSD) to represent different semantics through multiple prompt templates, and Decoupled Semantic Trusted Classification (DSTC) to perform classification based on each decoupled semantic while estimating uncertainties. Experiments demonstrate CDC's effectiveness in enhancing CLIP's performance across various settings and tasks, addressing the challenge of data misalignment in vision-language model adaptation.

Strengths

* CDC is well motivated from a causal perspective and has significant technical novelty. * Clear writing and well organized. * Experiment results show the effectiveness of CDC.

Weaknesses

* Figure 1(a) appears to illustrate task misalignment. Consider enhancing the caption of Figure 1 with more detailed explanations to clarify this concept. * Regarding data misalignment, it would be beneficial to provide a more precise definition. Does it specifically refer to discrepancies in classes between training and testing processes? It's important to clarify that data misalignment encompasses both label misalignment and distribution misalignment. A brief explanation of each type would improve understanding. * In Figure 3, the term "fuse" is used. It would be helpful to clarify the meaning and context of this term within the figure. * How about accuracy if we directly use zero-shot test for CDC?

Questions

See Weaknesses

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See Weaknesses

Summary

This paper investigates the task and data misalignment issues in pre-trained vision-language models such as CLIP. It discovers that the task-irrelevant information significantly affects the prediction of CLIP and soft prompt tuning cannot mitigate the data misalignment issue. The authors propose a novel Causality-Guided Semantic Decoupling and Classification method to mitigate the interference of task-irrelevant information. The experimental results show that the proposed method effectively mitigates the data misalignment and improves the generalization of CLIP.

Strengths

1. The paper is well-organized. The introduction of the method and the figures are clear and easy to understand. The description of the experiment setting is detailed, which makes the paper reproducible. 2. The proposed methods to mitigate the task and data misalignment of CLIP are highly-motivated and intuitive. 3. The authors design and conduct exhaustive experiments to demonstrate the effectiveness of the propose method. The proposed methods provide significant improvements on the generalization of CLIP.

Weaknesses

1. In the experiments section, the method is currently adapted solely to the CLIP model. This limitation may not fully demonstrate the model's universality. The authors can adapt the method to various vision-language models with different architectures to showcase broader applicability. 2. The experiments are exclusively conducted on image classification tasks. The authors can explore adapting vision-language models (VLMs) to a wider range of tasks, such as object detection, image captioning, or visual question answering, to further validate the model's versatility and performance across diverse applications.

Questions

Is it feasible to adapt the proposed method to various tasks such as object detection, image captioning, or visual question answering.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have discussed the limitations in the paper.