Thank you for the additional comment. We address them in below.
**Limited Number of Examples**
Our intention was to highlight specific instances that reveal the models’ deficiencies in handling complex visual deductive reasoning tasks. Table 4 only serves as an illustrative case study, presenting three qualitative examples for demonstrating the blindspot. We agree that a broader set of examples would provide a more robust analysis, we have incorporated new results from the Raven subset (60 examples) to expand our figure 3, which shows the same conclusion that additional segmentation dramatically improve the perception of the VLMs:
| | Raven | Manual-easy | Manual-medium | Manual-hard |
|-----------|-------|-------------|---------------|-------------|
| Before Segmentation | 7.4 | 50 | 20 | 20 |
| After Segmentation | 21.2 | 80 | 60 | 30 |
We also tested many examples in the Mensa and IT datasets, which shows the same finding as well. We will add the results and additional analysis into our paper.
**Motivation for Abstract Visual Puzzles**
Your point about the decoupling of image distribution shift and deductive reasoning abilities is well taken. As we mentioned in our last comment, we would investigate these pattern-based reasoning tasks using naturalistic images. This will help us assess their performance with realistic objects which does not have data distribution shift. We will also investigate if instruction fine-tuning can help close the data distribution gap and enhance the VLM performance. These studies will help us gain more insights in whether data distribution shift is the culprit. However, they require more effort, making them beyond the scope of this paper, which focuses on identifying existing issues rather than meticulously tracing their origins.
Our goal was to simulate a more holistic challenge that mirrors scenarios where generalist VLMs must navigate through even unseen and unfamiliar scenarios. You questioned whether it is appropriate to use RPMs for VLM evaluation. Although the PRMs were designed for humans as what you pointed out, we think that RPM problems represent a fundamental type of visual-spatial reasoning that artificial intelligence systems, particularly those aimed at achieving general intelligence, should also be able to perform well on.
In fact, using RPMs to assess VLM is not rare. For instance, many general VLM benchmarks like MATHVISTA [1] and MMMU [2] uses RPMs problems to probe models’ reasoning and understanding ability of complex patterns. When GPT-4v came out, technical evaluation work like [3] also probed its performance on abstract patterns, especially the RPMs. Different from them, our work dives deep into scrutinizing the blindspot, and the underlying issues when VLMs deals with these abstract patterns, showing some potential directions for future improvement.
```
[1] Lu, Pan, et al. “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.” ICLR 2024.
[2] Yue, Xiang, et al. “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.” CVPR, 2024.
[3] Yang, Zhengyuan, et al. “The dawn of lmms: Preliminary explorations with gpt-4v (ision).” arXiv preprint arXiv:2309.17421 9.1 (2023): 1.
```