Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering

Summary

This paper studies two problems in knowledge-based VQA: 1) the image input is often transformed into the text; 2) the relevance score between query and document is estimated based on one-dimensional dot product. To address these problems, the authors leverage several image features, including ROIs features from ViT and text descriptions. The experiments are conducted on the OK-VQA benchmark dataset. After being equipped with the recent strong BLIP v2 model, the proposed method achieves very promising results.

Strengths

- The paper is well-written and easy to follow for most parts. - The two studied problems are very important and practical for knowledge-based VQA models. - The proposed method achieves improved model performance after combining with BLIP v2.

Weaknesses

- Is there also ground-truth knowledge for the questions in the test set? - With the first question, did the authors use ground-truth knowledge for testing? - In fact, one dataset is not fairly convincing and the A-OKVQA should also be selected as a benchmark dataset. The reason that the authors provided does not make much sense. - The ablation study in Table 1 is very confusing. For example, there are no detailed experiments for T5-Large. Combining line 26 and 27, it seems that ROI&VE poses a negative effect on model performance. - For Eqn.3 and Eqn.5, why use max rather than mean or sum? - The ```late interaction``` in this paper means? - Line 141 confuses me a lot. It seems that the authors first map each ROI image to one feature, and then transform it to $N_{vt}$ tokens. Why don't we just use the $N_{vt}$ token features from ViT?

Questions

The proposed method achieves very promising results on the OK-VQA benchmark. However, several concerns refrain me from holding a positive view of this paper. I would like to see the authors' rebuttal to these concerns.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

The paper proposed a fine-grained Late-interaction multimodal knowledge retriever for knowledge-based VQA. The core idea to the to both visual tokens and question tokens as token-level queries to retrieve document in late interaction fashion. The paper suggests that (1) squeezing query representation and document representation into one vectors hinder the retrieval performance. (2) fine-grained visual representation helps retrieval. The paper also show that the framework can be easily extended to retrieve multimodal documents

Strengths

(1) The paper provide a recipe to take advantage of late Interaction and multi-dimensional representations to capture fine-grained, cross-modality relevance to improve retrieval performance that further boost the (A)OK-VQA tasks (2) The paper perform very thorough analysis on the retrieval results using different corpus choices for OKVQA tasks (Google search and wikipedia) (3) The paper is well-written and easy to follow (4) The paper provide 2 baseline VQA model (T5 and BLIP2) as the answer generator and the results looks very promising (5) The paper also report the additional inference time cost in the appendix indicating that the inference time over head is not un acceptable

Weaknesses

Overall, I think it is a solid paper, here is some minor comments (1) While the inference time cost has been mentioned, it is also good to mention storage cost as you need to storage the features for each tokens. Hope to know if it requires very good SSD or RAM though it is cheaper than the GPUs (2) in the bottom block in table 1, I hope to understand the performance difference when the number of knowledge sentences grows. It is good to plot the performance curve with x axis denoting the number of sentences and the y shows the vqa score. This can help us understand (a) performance-knowledge saturation speed compared to DPR, showing we need to use the FLMR if we have enough bucket to hold more knowledge as inputs. (3) [1] also uses fine-grained visual information as multiple queries to retrieve knowledge and it is good to compare to that as well ------------------------------------ post rebuttal: I read the author's rebuttal and still willing to support the paper as my initial rating References: [1] Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering

Questions

Please comment on the weakness part

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations

Summary

This paper proposes a multi-vector (late interaction) multi-modal retrieval-based approach for Knowledge-based Visual Question Answering, inspired by the ColBERT multi-vector representations for text. The retriever improves the recall substantially over prior multi-modal methods that use single-vector encoding. The approach obtains token-level embeddings for both textual input and visual input. They also align the vision and text modalities, by training a mapping network. The paper achieves very strong SoTA scores on OK-VQA.

Strengths

1. The paper's experiments are intuitive and extensive. They experiment test two types of visual representations (text-based and feature-based), aligning vision and text modalities, multi-modal late interaction, etc. in a series of very instructive ablations. 2. The results are very strong and convincing, aligned with the strong intuitions offered in the writing. 3. The paper offers nice, though minimalistic, qualitative analysis of their (nicely interpretable) retrieval.

Weaknesses

1. What are the connections to FILIP (Yao et al., 2021), which is also a late interaction retriever for some multi-modal settings. FILIP seems to tackle zero-shot image classification and image-text retrieval, which are much simpler tasks perhaps, but the connection must be explored in the paper. 2. The authors do report quite a few ablations for the retriever and the full method. That's great. But it's not necessarily great to report so many results on the test set! It might be better in the future to report such ablations on a validation set.

Questions

n/a

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

n/a

Summary

This paper introduces a new method, Fine-grained Late-interaction Multi-modal Retrieval, to improve knowledge retrieval in RA-VQA. The method addresses two major limitations in RA-VQA’s retriever: incomplete image understanding and lossy compression of visual scenes and questions.

Strengths

1) The paper proposes to capture fine-grained, cross-modality relevance that significantly improves retrieval performance. 2) The paper introduces a simple yet effective alignment procedure that can complement image representations obtained via image-to-text transforms, leading to more complete image understanding, better knowledge retrieval, and higher VQA accuracy. 3) The method improves approximately 8% in knowledge PRRecall@5 over other state-of-the-art retrievers in the OK-VQA dataset.

Weaknesses

1) Method. This work used the VinVL approach, which leverages regions of interest for visual features. I could not pinpoint the main contributions of this work, which DPR does not make. For a better understanding of the differences between the two methods, perhaps the authors could present them side-by-side in a figure. 2) FLMR and DPR. Why FLMR preserves richer information better than DPR representations? Because it also uses regions of the image with object class labels for predictions? I understand that FLMR allows full interaction between every query embedding vector and every document embedding vector. Is this the only difference? 3) In-batch negative sampling. Has DPR also used in-batch negative sampling? If not, did you try to compare when DPR uses in-batch negative sampling? How significant is this to the proposed method? 4) Results. Regarding Table 1, I am surprised to see that the comparison against BLIPv2 is not that large, while the proposed method uses the BLIPv2 caption generator. If PALI would use BLIPv2 text decoder, would it be better? I am unsure whether the main contribution is related to the BLIPv2/T5 text decoder or to better modeling. Overall, I appreciate the paper as it presents good results, but the contributions are largely unclear. I am open to the authors' feedback and other reviewers' opinions.

Questions

I wrote above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I wrote above.

Summary

This paper proposed an updated version of RA-VQA by incorporating a vision model aligned with text-based retriever in a late interaction way. Extensive experiments show the effectiveness of the proposed method. Thorough ablation studies and analysis were conducted.

Strengths

1. The paper is well-written and easy to read. 2. Extensive experiments and analysis are provided to show the effectiveness of the proposed method. 3. Experimental results demonstrate the superiority of the proposed method.

Weaknesses

1. Though the performance shows the superiority of the proposed method, the method of adding visual features is somewhat incremental compared to RA-VQA. 2. Why is the performance of adding ROI&VE in w/o late-interaction setting (row 26 in Table 1) worse than w/o them (row 27)? This result is quite strange. 3. It would be more intuitive to add the model size of each model in addition to the model names in Table 1.

Questions

See the above weakness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

none

Summary

In this paper, the authors focus on RA-VQA framework to solve the KB-VQA task. Specifically, this paper proposed Fine-grained Late-interaction Multi-modal Retrieval (FLMR) to address the limitation of inaccurate image representations and one-dimensional embeddings. FLMR leveraged late interaction and multi-dimensional representations to capture fine-grained relevance to retrieval performance. The proposed methods achieve sota performance in the OK-VQA dataset compared with systems with similar parameter sizes.

Strengths

The paper mainly focus on improving multimodal retrieval for the KR-VQA system, in detail, the proposed method uses an additional vision model to capture fine-grained features and align with text features instead of only using image-to-text features. The experiments show that the introduction of image representations from large vision models can lead to better retrieval and VQA performance which could benefit further research. The results on OK-VQA dataset and ablation study show the effectiveness of the proposed method.

Weaknesses

This paper only reports results on a single dataset KR-VQA and thus didn't clearly show if the conclusion could be generalized to other related benchmarks.

Questions

Have the authors performed experiments on other KB-VQA datasets or is that planned? If not, do authors have any particular reason for the missing of multiple datasets?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Summary

This paper focuses on retrieval augmented visual question answering and addresses two major limitations in previous RA-VQA. Specifically, they propose FLMR and overcome the limitations by obtaining image representations using a vision model. FLMR also encodes images and questions using multi-dimensional embeddings to capture finer-grained relevance between queries and documents. To verify the effectiveness, FLMR conducts extensive experiments on the OK-VQA dataset.

Strengths

1. This paper solves the two limitations of previous RA-VQA. It is important for improving the effectiveness of RA-VQA. 2. This paper is well-organized and easy to follow. 3. The proposed model is effective with much less parameters than large models.

Weaknesses

Actually, the novelty of this paper is somewhat trivial. The multi-modal late interaction has been proposed by ColBERT. Besides, using multi dimensional embeddings to improve retrieval is not very novel.

Questions

1. The authors should further clarify the technical novelty of this paper. 2. In line 163-164, please explain the improvement and details of pre-training the mapping network.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Response

I would like to express my gratitude to the reviewers for their helpful responses. Based on that, I raise increased my score to 5.

Response

Thanks for the authors' rebuttal. I will keep my original score.

Response

Thanks for the response! My concern is basically addressed. I will keep my score.