Library

RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models

Fine-tuned vision-language models (VLMs) often capture spurious correlations between image features and textual attributes, resulting in degraded zero-shot performance at test time. Existing approaches for addressing spurious correlations (i) primarily operate at the global image-level rather than intervening directly on fine-grained image features and (ii) are predominantly designed for unimodal settings. In this work, we present RaVL, which takes a fine-grained perspective on VLM robustness by discovering and mitigating spurious correlations using local image features rather than operating at the global image level. Given a fine-tuned VLM, RaVL first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, RaVL mitigates the identified spurious correlation with a novel region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. We evaluate RaVL on 654 VLMs with various model architectures, data domains, and learned spurious correlations. Our results show that RaVL accurately discovers (191% improvement over the closest baseline) and mitigates (8.2% improvement on worst-group image classification accuracy) spurious correlations. Qualitative evaluations on general-domain and medical-domain VLMs confirm our findings.

cs.CVcs.AIComputer Science
Source

Paper

References (57)

01Merlin: A Computed Tomography Vision-Language Foundation Model and DatasetL. Blankemeier, J. Cohen, Ashwin Kumar et al.2024 · Nature · 141 citations In Library
02A whole-slide foundation model for digital pathology from real-world dataHanwen Xu, N. Usuyama, Jaspreet Bagga et al.2024 · Nature · 965 citations
03Zero-Shot Robustification of Zero-Shot Models With Foundation ModelsDyah Adila, Changho Shin, Lin Cai et al.2023 · International Conference on Learning Representations · 34 citations In Library
04ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World DataM. Varma, Jean-Benoit Delbrouck, Sarah Hooper et al.2023 · IEEE International Conference on Computer Vision · 15 citations In Library
05A visual–language foundation model for pathology image analysis using medical TwitterZhi Huang, Federico Bianchi, Mert Yuksekgonul et al.2023 · Nature Medicine · 840 citations
06Quilt-1M: One Million Image-Text Pairs for HistopathologyW. Ikezogwo, M. S. Seyfioglu, Fatemeh Ghezloo et al.2023 · Neural Information Processing Systems · 259 citations In Library
07Where Does My Model Underperform? A Human Evaluation of Slice Discovery AlgorithmsNari Johnson, Ángel Alexander Cabrera, Gregory Plumb et al.2023 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · 18 citations In Library
08Discover and Cure: Concept-aware Mitigation of Spurious CorrelationShirley Wu, Mert Yuksekgonul, Linjun Zhang et al.2023 · International Conference on Machine Learning · 105 citations In Library
09Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningYu Yang, Besmira Nushi, Hamid Palangi et al.2023 · International Conference on Machine Learning · 68 citations In Library
10Spuriosity Rankings: Sorting Data to Measure and Mitigate BiasesMazda Moayeri, Wenxiao Wang, Sahil Singla et al.2022 · Neural Information Processing Systems · 19 citations In Library
11On Feature Learning in the Presence of Spurious CorrelationsPavel Izmailov, P. Kirichenko, N. Gruver et al.2022 · Neural Information Processing Systems · 198 citations In Library
12Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learningE. Tiu, Ellie Talius, Pujan Patel et al.2022 · Nature Biomedical Engineering · 536 citations

Scroll for more · 38 remaining

Similar papers

RoVLM: Region-Aware Optimal Vision-Language Alignment for Zero-Shot RecognitionFeng Guo, Zhongshu Chen, Yunqian Yu et al.2026 In Library
Stabilizing Modality Gap & Lowering Gradient Norms Improve Zero-Shot Adversarial Robustness of VLMsJunhao Dong, Piotr Koniusz, Xinghua Qu et al.2025 · Knowledge Discovery and Data Mining · 29 citations In Library
Debiased Prompt Tuning for Vision-Language Models without AnnotationsChaoquan Jiang, Yunfan Yang, Rui Hu et al.2025 · IEEE International Joint Conference on Neural Network · 1 citation In Library
Debiased Prompt Tuning in Vision-Language Model without AnnotationsChaoquan Jiang, Yunfan Yang, Rui Hu et al.2025 · arXiv.org · 1 citation In Library
Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine UnlearningYiwei Chen, Yuguang Yao, Yihua Zhang et al.2025 · arXiv.org · 11 citations In Library
RoRA-VLM: Robust Retrieval-Augmented Vision Language ModelsJingyuan Qi, Zhiyang Xu, Rulin Shao et al.2024 · arXiv.org · 19 citations In Library
VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative TrainingJipeng Zhang, Kehao Miao, Renjie Pi et al.2025 · arXiv.org · 3 citations In Library
Zero-shot Vision-Language Reranking for Cross-View GeolocalizationY. Erzurumlu, John Anderson, William J. Shuart et al.2026 · ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences · 0 citations In Library
Zero-shot Vision-Language Reranking for Cross-View GeolocalizationYunus Talha Erzurumlu, John E. Anderson, William Shuart et al.2026 · ISPRS annals of the photogrammetry, remote sensing and spatial information sciences In Library
Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningYang Yu, Besmira Nushi, Hamid Palangi et al.2023 · arXiv (Cornell University) · 68 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer onxK5/10 · confidence 4/52024-07-12

Summary

This paper introduces RAVL, a method designed to identify and address spurious correlations in Vision-Language Models (VLMs). The authors emphasize two key areas: (a) prioritizing local image features over global image-level features, and (b) concentrating on the fine-tuning phase rather than the pre-training phase. They implement the first idea through the following steps: 1. Utilizing the VLM embedding space to extract candidate image features from the validation set. 2. Filtering these candidate features to pinpoint those that directly contribute to classification errors. 3. Ranking the remaining image features based on the extent of their learned spurious correlations. To mitigate spurious correlations, the authors regularize the dissimilarity between the spurious regions and correlated labels while encouraging high embedding similarity.

Strengths

1. The paper is well-structured and easy to follow. 2. The proposed framework is sensible and, to the best of my knowledge, novel. I believe its contribution is slightly above the acceptance borderline. 3. The enhancement of performance quantitatively demonstrates the proposed method. It is notable.

Weaknesses

1. This framework is limited to mitigating geometric spurious correlations by focusing on regional candidates. While some studies such as [1] highlight texture and background information as significant contributors to spurious correlations, it is more desirable to address all potential sources of spurious correlations.ss all the potential spurious correlations. 2. Although they used RoIAlign, there are no qualitative visualization results to show which regions affect which classes. I believe this would be important for understanding the model's behavior from a human vision perspective. [1] Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." arXiv preprint arXiv:1811.12231 (2018).

Questions

1. What if this framework, instead of using RoIAlign, considers feature maps as candidates and filters them channel-wise? If this approach is effective, it could provide a more general framework from the perspective of Weakness 1, as feature maps are activated differently, representing various features [1]. 2. Why did the authors use K-Medoids clustering? Additionally, I am curious about the authors' choice to use softmax-normalized embeddings. Feature maps might offer more diverse information for comparison during clustering. 3. Is this framework robust on batch size? [1] Jeon, Myeongho, Myungjoo Kang, and Joonseok Lee. "A Unified Framework for Robustness on Diverse Sampling Errors." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed the limitations and potential negative societal impacts of their work in the Appendix. I fully agree with their assessment.

Reviewer aCX16/10 · confidence 3/52024-07-12

Summary

Vision-language models (VLMs) tend to exhibit poor zero-shot performance when compared to task-specific models. However, fine-tuned VLMs may capture spurious correlations in domain-specific datasets which may be small in size. The paper proposes an automated spurious correlation detection and mitigation method for fine-tuned VLMs. The method first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, it mitigates the identified spurious correlation with a region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. Experiments demonstrate the effectiveness of this method on general domain and medical-domain VLMs.

Strengths

- The paper is well-written. Key assumptions are clearly stated and the proposed method is easy to follow. - The paper proposes a useful method called RAVL to discover and mitigate fine-grained spurious correlations that can be easily interpreted by humans. - Experiments in the controlled and uncontrolled settings demonstrate that RAVL effectively discovers and mitigates spurious correlations between image features and textual attributes.

Weaknesses

There is a lack of computational complexity analysis. To detect fine-grained spurious correlations, RAVL needs to segment images into multiple regions, cluster all the regions per class, and calculate the contributions of each cluster to mispredictions. Therefore, the computational complexity seems high. It would be better to analyze the time cost of the proposed method.

Questions

1. How to obtain region-level labels when constructing the evaluation dataset (Line 220)? 2. How to determine the hyperparameters used in the spurious correlation discovery and mitigation stages?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Reviewer LC5i6/10 · confidence 4/52024-07-12

Summary

This paper tackles spurious correlations between image features and textual attributes in fine-tuned VLMs. It proposes an approach to discover and mitigate spurious correlations using local image features (image regions rather than a whole image). Experiments are done in both controlled settings and realistic settings.

Strengths

Discovering and mitigating spurious correlations is an interesting problem. The approach of looking into image regions is sound. The paper is generally well written.

Weaknesses

1) Looking into the regional features may make the approach computationally expensive. Computational complexity analysis is missing in the paper. 2) A lot of experiments were done under controlled settings. More evaluations in the wild would make the results stronger. 3) A small error in writing, line 259 says “are designed for unimodal settings” - ref [50] is designed for multi-modal settings.

Questions

Have you tried the approach on other settings, other than fine-tuned VLM? For example, spurious correlations can happen in other vision classification models as well.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

There are discussions on limitations in the supplementary material.

Reviewer LC5i2024-08-13

Thanks again for the rebuttal. This paper is based on the assumption that, “a model M that has learned a spurious correlation between an image feature e_a and a textual attribute y will demonstrate low zero-shot performance on (i) images in D_V with label y without the feature e_a and (ii) images in D_V with other labels Y \ {y} with the feature e_a. ” (e.g., see lines 124 - 127 in the paper) I have one more question on the paper. For some “true” image features, when the following are true, they may also appear to have the property (e.g., see lines 124 - 127 in the paper) mentioned above. 1) The “true” image features only appear in a subset of the class y. This can happen because not all images in the same class have the same features. 2) The training data and test data are sampled differently: the “true” features only appear (or appear more often) in the training data with class y, and appear less often in the test data with class y. How do you distinguish between spurious image features and the “true” image features with different appearing frequencies in training and test data?

Authorsrebuttal2024-08-14

Response to Reviewer LC5i

Below, we respond to the points raised by Reviewer LC5i during the discussion period. **Definition of spurious correlations. “For some “true” image features, when the following are true, they may also appear to have the property (e.g., see lines 124 - 127 in the paper) mentioned above. (1) The “true” image features only appear in a subset of the class y. This can happen because not all images in the same class have the same features. (2) The training data and test data are sampled differently: the “true” features only appear (or appear more often) in the training data with class y, and appear less often in the test data with class y. How do you distinguish between spurious image features and the “true” image features with different appearing frequencies in training and test data?”** Our definition of spurious correlations is consistent with prior work (e.g. [1,2]). In reference to the specific cases listed by the reviewer above, it is standard practice to consider a "true" image feature as one that appears consistently within the class. This means that “true” features will (1) appear in all images rather than just a subset and (2) will be consistently associated with the class label $y$ at both training and test time. For example, when considering images from the class “bird”, it is reasonable to assume that all images will include the true features of feathers, wings, and a beak, which are defining characteristics of birds. This will be true for all birds, independent of whether they are in a training or testing dataset. We refer the reviewer to Singla et al. [2], which formally defines true features as those that are "always a part of the" class definition; meanwhile, spurious features are defined as those that "are likely to co-occur" with the true features but are not a part of the class definition. We follow these definitions in our work. We demonstrate with 654 evaluations in both synthetic and real-world settings that our approach can differentiate between spurious features and true features, as shown in Figure 2 and Table 1; we corroborate this further with in-the-wild experiments. We hope that this clarifies the definitions of true and spurious features. [1] Eyuboglu et al. “Domino: Discovering Systematic Errors with Cross-Modal Embeddings” ICLR 2022. [2] Singla, S. et al. Salient ImageNet: How to discover spurious features in Deep Learning? ICLR, 2022.

Reviewer LC5i2024-08-14

Thanks for the response. My concerns have been well addressed.

Reviewer onxK2024-08-10

Response to Author Feedback

Thank you for your feedback. However, after further consideration, I have become less favorable toward this submission and have decided to downgrade my initial score for the following reasons: - In response to Q1, the authors suggest that the empirical results are a key justification. However, with only a brief experiment provided in Q5, I am not convinced that RaVL outperforms other baselines across various batch sizes. The impact of batch size is not a trivial factor. - Thank you for including the additional qualitative experiment. However, I find the statement "The new VLM more precisely relies on core features" unclear. Could you please specify which core features you are referring to? - (i) Is it not possible to mitigate shortcut correlations (SCs) in an automated manner using feature maps? (ii) This makes sense. (iii) Why do you believe that feature maps represent global features? Given the receptive field, feature maps typically capture local features within the feature space (e.g., the first channel corresponds to the top-left, while the last channel corresponds to the bottom-right). - The justification provided for the use of K-medoids clustering seems appropriate to me.

Reviewer LC5i2024-08-12

Thanks for the detailed rebuttal.

Authorsrebuttal2024-08-14

Response to Reviewer ONXK (1/2)

Below, we respond to the points raised by Reviewer ONXK during the discussion period. **Batch Size. “With only a brief experiment provided in Q5, I am not convinced that RaVL outperforms other baselines across various batch sizes. The impact of batch size is not a trivial factor.”** In our paper, we used a batch size of 128 across all mitigation experiments (Section 4). We selected a batch size of 128 for the following reasons. First, a batch size of 128 is the largest size that will fit on a single A100 GPU; we specifically used a single A100 GPU for all experiments in order to demonstrate the ease of use and cost-effectiveness of RaVL. Second, a batch size of 128 is either larger or equivalent to the batch sizes used by prior works that fine-tune VLMs for real-world applications (e.g. medical settings [1,2]) demonstrating that our choice of the batch size hyperparameter is consistent with real-world practice. Third, the Spurious-Aware Mitigation baseline, which is the only other previously-developed approach for the fine-tuned VLM setting (Appendix A), is designed to operate with a batch size of 128; consequently, using a batch size of 128 across evaluations enables a fair comparison with this baseline. As shown in Table 3 in the paper, RaVL Stage 2 outperforms all mitigation baselines. Beyond the batch size justification shared above, to be further responsive to the provided reviewer comment, we extend the batch size ablations provided with our original rebuttal by evaluating RaVL across three batch sizes (32, 128, and 256) and six baselines. We evaluate multiple COCO evaluation settings for each ablation. Across a total of 72 trained VLMs $M_{new}$ evaluated in the tables below, **we show that RaVL outperforms all baselines across all evaluated batch sizes**. We thank the reviewer for this suggestion, and we will be sure to update the manuscript with this ablation. | Batch Size | Method | Img. Overall | Img. Worst Group | Reg. Overall | Reg. Worst Group| |----------------------|------------------|------------------|------------------|------------------|------------------| | 32 | Standard FT | 59.1 | 25.2 | 69.3 |44.1 | | 32 | Upsampled FT | 60.3 | 28.0 | 65.2 | 34.3 | | 32 | VL-ERM |58.4 |21.6 | 67.7 | 37.2 | | 32 | VL-GDRO | 59.6 |24.3 | 69.3 | 35.2 | | 32 | Spurious-Aware |54.2 |12.2 | 66.9 | 36.2 | | 32 | RaVL (Ours) | **64.3**| **39.7**| **75.5** |**44.7**| | Batch Size | Method | Img. Overall | Img. Worst Group | Reg. Overall | Reg. Worst Group| |----------------------|------------------|------------------|------------------|------------------|------------------| | 128 | Standard FT | 67.6 | 33.6 | 77.3 | 46.0 | | 128 | Upsampled FT | 69.5 | 37.1 | 79.6 | 52.9 | | 128 | VL-ERM | 68.2 | 29.0 | 78.1 | 46.3 | | 128 | VL-GDRO | 69.7 | 31.5 | 77.5 | 44.6 | | 128 | Spurious-Aware | 69.4 | 31.8 | 78.6 | 48.1 | | 128 | RaVL (Ours) | **71.0** | **42.9** | **82.1** | **57.4**| | Batch Size | Method | Img. Overall | Img. Worst Group | Reg. Overall | Reg. Worst Group| |----------------------|------------------|------------------|------------------|------------------|------------------| | 256 | Standard FT | 64.3 | 41.4 | 73.1 | 53.9 | | 256 | Upsampled FT | 66.8 | 48.8 | 74.0 | 58.5 | | 256 | VL-ERM | 62.5 | 24.9 | 73.2 | 44.9 | | 256 | VL-GDRO | 63.5 | 29.8 | 75.2 | 45.1 | | 256 | Spurious-Aware | 66.7 | 34.3 | 74.6 | 44.5 | | 256 | RaVL (Ours) | **68.1** | **53.9** | **79.0** | **66.7**| **Visualizations. “Thank you for including the additional qualitative experiment. However, I find the statement "The new VLM more precisely relies on core features" unclear. Could you please specify which core features you are referring to?”** The images in the figure are obtained from the test set of a COCO evaluation setting in which the fine-tuned VLM $\mathcal{M}$ learned a strong spurious correlation between the class label "chairs" and the spurious feature "person". Visualizations show that before mitigation, the VLM is unable to precisely localize the core features associated with the class labels, often relying on the presence of people or various background details. After the RaVL mitigation procedure, the VLM $\mathcal{M}_{new}$ more precisely relies on core features (namely, the chairs).

Authorsrebuttal2024-08-14

Response to Reviewer ONXK (2/2)

**Feature Maps. “(i) Is it not possible to mitigate shortcut correlations (SCs) in an automated manner using feature maps? (ii) This makes sense. (iii) Why do you believe that feature maps represent global features? Given the receptive field, feature maps typically capture local features within the feature space (e.g., the first channel corresponds to the top-left, while the last channel corresponds to the bottom-right).”** Our approach uses RoIAlign to extract relevant regions of the feature maps, which we then compress into a low-dimensional embedding. These embeddings (1) can then be accurately and efficiently processed in an automated fashion and (2) are explicitly designed to characterize local, region-level features. We emphasize that using RoIAlign to embed regions of interest is a standard technique grounded in prior literature (e.g. [3]). (i) Performing automated identification of region-level spurious correlations using raw feature maps would be computationally challenging due to the presence of large numbers of feature maps; for instance, a ResNet-50 model produces 2048 distinct 2D feature maps per image at the last layer. For a small validation dataset with only 1K images, this would result in over 2 million 2D feature maps. Strategies for reducing the number of feature maps (e.g. via pruning) can result in the loss of important information. Using ROIAlign and region-level embeddings enables us to reduce dimensionality of feature maps while preserving semantic meaning, enabling efficient automated analysis. (iii) We first clarify that a feature map produced by a model has three dimensions: height $H$, width $W$, and channels/depth $C$. For a ResNet-50 with an input image size of 224x224, the feature map dimensions at the last layer are $H=7$, $W=7$, and $C=2048$. The channels themselves do not explicitly represent image geometry and locations (i.e. the top left or bottom right); rather, each channel corresponds to a 2D feature map of size $7 \times 7$ that characterizes a feature within the input image. These 2D feature maps exhibit significant variations in feature granularity; the features characterized in each map can range in granularity from global (e.g. color, texture) to local (e.g. individual leaves on a tree). This variability limits the use of raw feature maps for our task. On the other hand, ROIAlign is an effective strategy for embedding specific regions of interest, focusing exclusively on local, fine-grained features. We hope that the above responses address your concerns. Thank you for your time. [1] Tiu, E., et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat. Biomed. Eng (2022). [2] Huang, Z et al. A visual–language foundation model for pathology image analysis using medical Twitter. Nat. Medicine (2023). [3] Zhong et al. RegionCLIP: Region-based Language-Image Pretraining. CVPR 2022.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC