Summary
Prevalent Audio-Visual Question Answering (AVQA) methods often learn biases from the dataset, which can lead to reduced robustness. Additionally, current datasets may not accurately assess these methods. To address these issues, the authors introduce MUSIC-AVQA-R, a dataset created in two steps. First, they rephrase questions in the test split of a public dataset (MUSIC-AVQA) and then introduce distribution shifts to the split questions. Rephrasing the questions allows for a diverse test space, while the distribution shifts enable a comprehensive evaluation of robustness for rare, frequent, and overall questions. The authors also propose a robust architecture that uses a multifaceted cycle collaborative debiasing strategy to overcome biased learning.
Weaknesses
- First, I have a concern about the quality of the new dataset, it is not verified enough whether the dataset is properly collected. Similar to the case of the VQA-CP dataset, the authors should claim that the proposed dataset is a useful testbed to measure the distribution shift out of distribution robustness by showing the distribution difference between the train and test datasets. However, the authors only show the distribution of the rephrased dataset without showing the distribution difference between the train and test datasets. In short, there is no way to see whether the collected data after paraphrasing and distribution shifting is a valid one or not. In other words, how does the resulting distribution differ from the previous distribution?
The proposed solution also lacks novelty. Extensive work has been done to mitigate modality bias in the VQA-CP dataset, such as [1,2,3]. Therefore, in order to study the modality bias problem in AVQA, the authors should also devise a novel solution specialized for the AVQA problem by comparing it with the existing debiasing methods for VQA.
[1] Counterfactual Samples Synthesizing for Robust Visual Question Answering. CVPR 2020.
[2] Greedy Gradient Ensemble for Robust Visual Question Answering. ICCV 2021.
[3] Generative Bias for Robust Visual Question Answering. CVPR 2023.
- In addition, the authors should discuss why the proposed method is better than existing solutions theoretically. However, there is no explanation for why simply increasing the logit distance among the modalities in Equation 1 reduces the bias. Is there any reference from which the authors adopted the ideas? What I especially don't understand is whether there is any theoretical evidence as to why it is necessary to add the inverse of distance. How about minus? More theoretical discussions are required.
- In addition, the proposed solution itself also doesn’t make sense. Equation 3, the KL divergence between logits, and equation 2, the Euclidean distance, are the same in that they measure the distance between the logits. In theory, when the temperature term in the softmax function is large enough, the kl divergence loss becomes equivalent to the Euclidean distance. It is unclear what is the meaning of putting these two similar losses in opposite directions? The authors should further explain the theoretical motivation for using those loss functions.
- The performance in Tables 1 and 2 is also questionable. If other existing debiasing methods, such as [1,2,3], were applied to the same baseline, could the proposed method still outperform them? The proposed solution will be meaningful only if the proposed method sufficiently outperforms the existing methods.
I don't understand how a simple method in this paper (ours in Table 1) achieves a favorable performance in MUSIC-AVQA with such a simple baseline architecture. What is the baseline model performance without adding the proposed loss functions? In Table 3, how does the model without any proposed loss function already show a favorable performance that outperforms the state-of-the-art methods?