Responses to Reviewer kFPK (Part III)
**(2.2) GPT-4 Evaluation on MinGPT-4, LLaVA, MIMIC-IT Datasets.** We would like to respectfully clarify that the Mini-GPT4, LLaVA-Instruct-150K, and MIMIC-IT datasets are generated by LLMs and are primarily used for instruction tuning. These datasets do not have corresponding human-annotated test sets that are typically used for evaluation purposes. This unique nature of the datasets presents a challenge in terms of conventional evaluation protocols. Therefore, we leverage GPT-4 (text-only) to evaluate the quality of our model’s generated responses on these datasets.
Specifically, we randomly sample 200 examples from each of these three datasets. The sampled instruction-response pairs, along with additional object bounding boxes and captions for the images, are input into GPT-4 as references. Following recent studies [*Liu et al., 2023; Yin et al, 2023*], we leverage GPT-4 to evaluate the helpfulness, relevance, accuracy, and level of details of the responses from MLLMs. GPT-4 provides an overall score on a scale from 1 to 10 for each response, where a higher score indicates better performance.
As shown in the following table, we compare our VPG-C with InstructBLIP, MiniGPT-4, and LLaVA. While LLaVA and MiniGPT-4 have encountered some of the testing data during training, our VPG-C still achieves competitive performance. Particularly, our VPG-C demonstrates significant superiority on the challenging MIMIC-IT dataset, of which each instruction is accompanied by complex multi-modal in-context information.
| | MiniGPT-4 Dataset | LLaVA-150K | MIMIC-IT |
| :----------- | :---------------- | :--------- | :------- |
| LLaVA | 7.2 | **8.5+** | 5.2 |
| MiniGPT-4 | **8.8+** | 7.4 | 5.4 |
| InstructBLIP | 7.3 | 7.5 | 6.5 |
| **VPG-C** | **8.1** | **7.8** | **7.6** |
*(+ indicates that the testing samples have been used during the training stage.)*
**(2.3) GPT-4V Evaluation on MinGPT-4, LLaVA, MIMIC-IT Datasets.** To further illustrate the effectiveness of our method, we use recently opened GPT-4V, which can directly take images as input. Since the API interface is not available and there are strict limits on the number of uses, we randomly sample 30 examples from each dataset and manually prompt GPT-4V via web interface. We use similar prompt templates as a recent study [*Yin et al., 2023*]. The following table verifies the consistent superiority of our VPG-C on these complicated instruction data.
| | MiniGPT-4 Dataset | LLaVA-150K | MIMIC-IT |
| :----------- | :---------------- | :--------- | :------- |
| LLaVA | 7.0 | **8.7+** | 4.9 |
| MiniGPT-4 | **8.4+** | 7.3 | 5.0 |
| InstructBLIP | 7.1 | 7.1 | 6.4 |
| **VPG-C** | **8.2** | **7.6** | **7.5** |
*(+ indicates that the testing samples have been used during the training stage.)*
Thank you once again for your constructive suggestions, which have been instrumental in enhancing the quality of our research! We have now incorporated these experiments into our revised manuscript (Appendix H, I).
**References**
[*Xu et al., 2023*] LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models. Arxiv 2306.09265.
[*Liu et al., 2023*] Improved Baselines with Visual Instruction Tuning. Arxiv 2310.03744.
[*Yin et al., 2023*] Woodpecker: Hallucination Correction for Multimodal Large Language Models. Arxiv 2310.16045.