Response to follow-up questions
Thank you for reviewing the rebuttal. We would like to address your follow-up concerns:
----
**Benchmark**
**Q1**: There is a large discrepancy between the qualitative and quantitative evaluation. The qualitative results provided in Figure 1 are quite different from the distribution of the academic benchmark datasets in Table 1.
**A1**: We agree that the distributions of quantitative and qualitative samples are different. We have made every effort to conduct a systematic and comprehensive evaluation using public benchmarks. Additionally, we have addressed out-of-domain cases in our qualitative results, as shown in both Fig. 1 and Appendix B of the supplementary material, to demonstrate our model's capabilities. However, we believe that a qualitative comparison between models is not exhaustive, as it encompasses only a limited number of cases and can be subjective.
----
**Q2**: About LVLM-eHub.
**A2**: We believe the Arena in the LVLM-eHub is not a reliable evaluation due to the following reasons:
1. The paper does not describe how many samples were collected in the Arena leaderboard.
2. The June 29 version of the Arena Ranking, which can be found on the LVLM Hub website (as links are not allowed in the response), differs significantly from the June 13 version (Fig.1c in their paper), even though no new models were added. We believe that the current Arena is not stable enough to accurately demonstrate the true capabilities of various models. One potential reason could be the limited number of samples.
3. It is a bit hard to distinguish between *in-domain* and *out-of-domain* in this evaulation. In the quantitative results from LVLM-eHub, many datasets they utilized were not included in InstructBLIP's training and, thus, cannot be classified as *in-domain*. Despite this, InstructBLIP still demonstrates strong performance on these datasets. Furthermore, for the Arena evaluation, it cannot be directly treated as *out-of-domain*, since we are unaware of the data provided by the users.
4. InstructBLIP significantly outperforms other models in avoiding the issue of object hallucination, making it more reliable and trustworthy. One potential reason is that InstructBLIP can adaptively adjust the length of its responses, as illustrated in Appendix B. In contrast, many other models consistently produce lengthy paragraphs, a pattern they learned during training. While this tendency might result in more hallucination, users might favor longer responses because they appear more detailed.
----
**Q3**: About VisDial.
**A3**: We apologize for the ambiguity. VisDial is indeed not truly held-out, we will clarify this in the next version of our paper. However, it is nearly the only high-quality visual dialog dataset available for quantitative evaluation, so we still incorporated it in our evaluation. Additionally, for the other held-out datasets, there is no overlap with the held-in ones.
----
**Q4**: About ablation studies in Table 2.
**A4**: The experiments in Table 1 and Table 2 are entirely independent, they do not influence each other. For Table 2, we conducted separate experiments to demonstrate the impact of using instruction-aware visual features and the balanced data sampling strategy. We've presented the results for only 5 datasets to maintain clarity in our paper's format, and these are sufficient to support our insights.
----
**Architecture**
**Q1**: About multi-turn conversation.
**A1**: As the model focuses on the question or utterance of the current turn, we only need to use 32 latent tokens for the visual features. Previous turns simply serve as the dialog context.
----
**Q2**: Is there any evidence that Qformer can handle long and complex instructions?
**A2**: The capability of InstructBLIP depends on the data it has been trained with. Therefore, its capability of handling complex instructions is influenced by the amount of such data we utilize.
----
**Q3**: If "Q-former can simply ignore the instruction and extract general visual features", is 32 tokens enough?
**A3**: When there is no useful information in the instruction, it typically represents a generic task, and 32 tokens are usually sufficient in such cases. Moreover, based on our preliminary experiments, increasing the number of query embeddings beyond 32, for instance to 48, does not yield any further improvement. We will include this discussion in the next version of our paper.
----
Thank you once again. We hope our reply adequately addresses your follow-up concerns.