Summary
This paper introduces a new framework called Vision Search Assistant, which aims to enhance the capabilities of mutimodal large language models and web agents through collaboration, targeting multimodal search tasks involving unfamiliar images and novel concepts. The authors identify three key aspects of visual search: what to search, how to search, and what to conclude. To address these, the proposed framework utilizes three steps: Visual Content Formulation, Chain of Search, and Collaborative Generation. The Vision Search Assistant is evaluated on the LLaVA-Wild dataset, and further assessed through open-set evaluations by human experts. Results indicate that Vision Search Assistant achieved the best outcomes across multiple metrics.
Strengths
1. Open-domain visual search is valuable in practice, and combining multimodal large language models with search represents an intriguing research direction.
2. The paper identifies three critical questions in visual search — what to search, how to search, and what to conclude, which are interesting research points.
Weaknesses
1. There is a lack of efficiency analysis, especially concerning the average latency per query, number of LMM calls, or token consumption. While efficiency is often an issue with large models, an essential analysis should not be overlooked.
2. The evaluation has some limitations. The paper only tests on the LLaVA-Wild dataset, which contains only 60 queries. In practice, many VQA datasets involving external knowledge, such as A-OKVQA, could serve as more comprehensive evaluation benchmarks.
3. There is insufficient information regarding reproducibility. Sampling parameters, such as temperature and top-p during large model inference, are not provided. Additionally, prompts used with large models (e.g., in Eq. 4 and Eq. 7) are not specified.
Questions
1. What is the motivation for transforming visual regions into text descriptions for search, instead of directly using images? In my view, some critical information may be lost in the descriptive process, and many models struggle with entity recognition (such as the athlete’s name in Fig. 3), potentially affecting search accuracy. How do the authors think of this issue?
2. In Table 1, what does "LLava-1.6-7B (Baseline)" represent? Does it indicate zero-shot LLava? Why does the performance on Conversation decline when image search is introduced (naive search)? The way Google Image Search is utilized requires further explanation.
3. There is a lack of detailed qualitative analysis. The authors could provide a case study showing descriptions of each region, the results of each search chain step, and the final model-generated response to illustrate the benefits of the proposed method.
4. In Fig. 6, the authors conduct human evaluations of open-domain QA. What criteria are used to assess Factuality, Relevance, and Supportiveness? If these definitions follow prior work, citations should be provided; otherwise, detailed explanations are needed. Furthermore, are the human evaluation results statistically significant and consistent?