SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge

Response to Reviewer qhJo

> **Weakness4: Differences between UDK-VQA and existing datasets. Definition of up-to-date knowledge and timestamp threshold.** **Response:** The difference between our UDK-VQA dataset and existing datasets lies in the timeliness of the test data. Existing RAG datasets rely on external knowledge that is usually commonsense knowledge, which is not time-sensitive and does not update frequently. Such knowledge can be learned by LVLMs during pre-training and supervised fine-tuning. In contrast, the knowledge our dataset relies on is time-sensitive and belongs to the up-to-date knowledge, which refers to the most current and relevant information, data, and understanding available at a given point in time. Specifically, this refers to information that emerged after the LVLMs completed their training, such as news generated after a certain date. We set the timestamps for data collection to be after the latest release date of all LVLMs we tested, ensuring that the knowledge required by the test set was not accessible to LVLMs during their training. This work is a long-term project, and we will continue to update the test set automatically using our framework to ensure the data is new for LVLMs. > **Weakness5: Broader evaluation on other benchmarks requiring external knowledge.** **Response:** Thanks for your suggestions. We have evaluated different LVLMs on a broader benchmark, including GQA [1], INFOSEEK [2], A-OKVQA [3] and the proposed UDKAG. The experimental results are shown in the table below, where GQA does Not Rely on external Knowledge **(NRK)**, INFOSEEK and A-OKVQA Rely on Commonsense Knowledge **(RCK)**, and UDKAG Relies on the Up-to-Date Knowledge **(RUDK)**. From the table, we can observe that our framework improves the performance of different LVLMs across various datasets. The improvements on these three datasets are not as significant as on our UDKAG dataset for the following reasons: (1) GQA does not rely on external knowledge and is used to evaluate the reasoning ability of LVLMs, which is beyond the scope of our framework. (2) Our framework focuses on retrieving the up-to-date knowledge, whereas INFOSEEK and A-OKVQA rely on commonsense knowledge, much of which has already been used in the training data of LVLMs. | Baseline | Variant | Local Data (clear GT) | Internet Data (unclear GT) | GQA [1] (NRK) | INFOSEEK [2] (RCK) | A-OKVQA [3] (RCK) | UDKAG (RUDK) | |---------------|-----------|-----------------------|----------------------------|---------------|--------------------|-------------------|--------------| | CFR [5] | - | - | - | 72.10 | - | - | - | | Oracle→FID [2]| - | ✓ | - | - | 45.60 | - | - | | Omni-SMoLA [4]| - | ✓ | - | - | - | 84.10 | - | | LLaVA-1.6 | Raw | - | - | 61.66 | 37.86 | 75.53 | 31.8 | | | **Ours** | **-** | **✓** | **62.33** | **41.25** | **76.22** | **90.2** | | InternVL-1.5 | Raw | - | - | 74.03 | 51.13 | 84.53 | 42.6 | | | **Ours** | **-** | **✓** | **74.41** | **53.10** | **84.59** | **92.9** | --- > **Weakness6: Performance dependency on the hierarchical filtering model.** **Response:** Our experimental results have proven that: (1) The hierarchical filtering model has transferability and can be applied to different LVLMs and datasets without fine-tuning. (2) The quality of the hierarchical filtering model is not significantly related to the backbone it uses. As shown in Table 1 and Figure 4 of the main manuscript, we tested our hierarchical filtering model on 13 different LVLMs, all of which showed significant improvement. As described in our response to weakness 5, our hierarchical filtering model can also bring some improvement when directly transferred to other datasets, whether they are based on additional knowledge or not. Furthermore, as shown in Table 2 of the main manuscript, using LLaVA and Qwen as the backbone for the hierarchical filtering model can achieve significant performance improvements.

Response to Reviewer qhJo

> **Limitation1: Detailed impact of separate training.** **Response:** Separate training is necessary as joint training results in performance degradation. As shown in the table below, whether using Qwen-VL or LLaVA-1.5 as the baseline model, separate training brings more significant performance improvement. The main reasons are: (1) Our training data uses pseudo-labeling instead of high-quality human annotation. Training based on such data may cause LVLMs to lose their original semantic understanding capabilities. (2) Our training and testing sets are generated from news from different time periods, involving different entities and having different distributions. Training LVLMs on our training set easily leads to overfitting, resulting in lower generalization on the test set. | Baseline | Variant | Training Strategy | UDKAG | |-----------|--------------|--------------------|-------| | QWen-VL | Raw | - | 35.2 | | | Ours | Joint Training | 68.5 | | | **Ours** | **Separate Training** | **84.8** | | LLaVA 1.5 | Raw | - | 41.2 | | | Ours | Joint Training | 68.0 | | | **Ours** | **Separate Training** | **88.9** | --- > **Limitation2: Addressing incomplete snippets.** **Response:** Thanks for your suggestions. The completeness of snippets has little impact on accuracy because their purpose is to allow the website filter to perform an initial screening of web pages, reducing the input the content filter needs to process, thereby improving the efficiency of our framework. Directly using incomplete snippets with existing search engines is already widely practiced [7][8]. There are two intuitive strategies to enhance the completeness of snippets: (1) Using large language models (LLMs) to continue writing the snippets. (2) Crawling the complete content of websites to fill in the snippets. The first method may result in incorrect snippets because LLMs lack the up-to-date knowledge and cannot accurately continue the writing. The second method will incur additional time consumption during inference because it requires crawling the content of all websites, regardless of whether the content filter needs to process that content. This contradicts our original intention of improving efficiency with the website filter. We are implementing the second method to verify whether complete snippets will bring performance improvements, and the experimental results will be provided in the discussion phase. --- [1] Hudson et al. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR 2019. [2] Chen et al. Can pre-trained vision and language models answer visual information-seeking questions? In EMNLP 2023. [3] Schwenk et al. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV 2022. [4] Wu et al. Omni-SMoLA: Boosting generalist multimodal models with soft mixture of low-rank experts. In CVPR 2024. [5] Nguyen et al. Coarse-to-fine reasoning for visual question answering. In CVPRW 2022. [6] Yu et al. Generate rather than retrieve: Large language models are strong context generators. In ICLR 2023. [7] Lu et al. Chameleon: Plug-and-play compositional reasoning with large language models. In NeurIPS 2023. [8] Yang et al. Mm-react: Prompting chatgpt for multimodal reasoning and action. In arXiv 2023.

Message to Reviewer qhJo

Dear Reviewer qhJo, We hope this message finds you well. We sincerely appreciate the time and effort you are dedicating to evaluating our submission. As the conference timeline approaches important deadlines, we would be grateful for any updates you could provide regarding the review status of our paper. If there is anything further we can provide to assist with the review process, please don’t hesitate to let us know. Additionally, we present the **experimental results of snippet completeness** on performance here, where $\theta$ represents the percentage as mentioned in Section 5.6 of the main manuscript. For each website snippet, we attempted to locate the full sentence corresponding to the snippet by crawling the website's content. However, the content of many websites could not be crawled. For such websites, we experimented with two strategies: (1) Discarding these websites during training and testing. (2) Using the incomplete snippets. The experimental results are shown in the table below, where "Raw" represents all snippets without completion, "Discard" represents strategy (1), and "Mixture" represents strategy (2). The experimental results show that directly discarding the websites leads to a significant performance loss, as discarding reduces the number of usable websites by approximately half, thereby limiting the performance of the website filter. As $\theta$ increases, the performance of strategy (2) becomes increasingly close to that of all snippets without completion, which validates our rebuttal argument that the completeness of the snippets has little impact on accuracy. We greatly appreciate your attention to this matter and look forward to any feedback you may have. Best regards, Authors --- | Baseline | Variant | $\theta$=10\% | $\theta$=25\% | $\theta$=40\% | $\theta$=55\% | $\theta$=70\% | $\theta$=85\% | $\theta$=100\% | |-----------|----------|-------------|-------------|-------------|-------------|-------------|-------------|--------------| | LLaVA-1.6 | Raw | 80.8 | 86.1 | 88.4 | 89.4 | 89.7 | 90.0 | 90.2 | | | Discard | 76.6 | 80.2 | 80.7 | 82.1 | 81.9 | 81.3 | 81.6 | | | Mixture | 83.6 | 87.9 | 89.4 | 89.6 | 89.8 | 90.4 | 90.2 | | InternVL-1.5 | Raw | 84.6 | 89.2 | 91.1 | 92.9 | 92.4 | 92.7 | 92.9 | | | Discard | 79.7 | 82.2 | 82.9 | 83.8 | 84.0 | 82.8 | 83.2 | | | Mixture | 86.0 | 89.5 | 92.2 | 92.3 | 92.2 | 92.5 | 92.9 |

Looking Forward to Your Reply.

Dear Reviewer qhJo, I hope this message finds you well. We are truly grateful for the effort you have put into reviewing our manuscript and for the insightful feedback you have provided. We understand that you may have a busy schedule, but we kindly ask if you could review our responses at your earliest convenience. Your feedback is essential for us to move forward, and we would greatly value any additional insights you may have. Thank you once again for your time and consideration. Best regards, Authors

Thank you for your review!

Thank you for your prompt reply and valuable feedback on our manuscript. In the revised version, we will include experimental results on more datasets to better demonstrate the generalizability of our framework.