Visual Question Answering (VQA) systems have seen remarkable progress with the incorporation of multimodal data. However, their performance is still hampered by biases ingrained in language and vision modalities, frequently resulting in subpar generalization. In this study, we introduce a novel counterfactual causal framework (CC-VQA). This framework utilizes Counterfactual Sample Synthesis (CSS) and causal inference to tackle cross-modality biases. Our approach innovatively employs a strategy based on causal graphs, which effectively disentangles spurious correlations in multimodal data. This ensures a balanced and precise multimodal reasoning process, enabling the model to make more accurate and unbiased decisions. Moreover, we propose a contrastive loss mechanism. By contrasting the embeddings of positive and negative samples, this mechanism significantly enhances the robustness of VQA models. Additionally, we develop a robust training strategy that improves both the visual-explainable and question-sensitive capabilities of these models. Our experimental evaluations on benchmark datasets, such as VQA-CP v2 and VQA v2, demonstrate substantial improvements in bias mitigation and overall accuracy. The proposed CC-VQA framework outperforms state-of-the-art methods, highlighting its effectiveness in enhancing the performance of VQA systems.
Paper
Full text
Counterfactual causal inference for robust visual question answering
Semantic Scholar · Medicine · 2025
Abstract
Visual Question Answering (VQA) systems have seen remarkable progress with the incorporation of multimodal data. However, their performance is still hampered by biases ingrained in language and vision modalities, frequently resulting in subpar generalization. In this study, we introduce a novel counterfactual causal framework (CC-VQA). This framework utilizes Counterfactual Sample Synthesis (CSS) and causal inference to tackle cross-modality biases. Our approach innovatively employs a strategy based on causal graphs, which effectively disentangles spurious correlations in multimodal data. This ensures a balanced and precise multimodal reasoning process, enabling the model to make more accurate and unbiased decisions. Moreover, we propose a contrastive loss mechanism. By contrasting the embeddings of positive and negative samples, this mechanism significantly enhances the robustness of VQA models. Additionally, we develop a robust training strategy that improves both the visual-explainable and question-sensitive capabilities of these models. Our experimental evaluations on benchmark datasets, such as VQA-CP v2 and VQA v2, demonstrate substantial improvements in bias mitigation and overall accuracy. The proposed CC-VQA framework outperforms state-of-the-art methods, highlighting its effectiveness in enhancing the performance of VQA systems.