Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Summary

This work presents 3 new synthetically generated VQA evaluation benchmarks and conducts a comprehensive evaluation of the limitations of current VLMs in spatial reasoning tasks. These tasks include spatial relationships, navigation, position understanding, and counting. The authors demonstrate using their introduced benchmarks that VLMs have limited performance on tasks requiring detailed visual understanding and reasoning of images.

Strengths

Evaluating the spatial reasoning capabilities of SoTA VLMs is highly impactful and very relevant. I find this work novel, intriguing, and highly useful as it points to very important limitations in VLMs. The experimental analysis is very comprehensive, incorporating both several open-source VLMs and proprietary models. Additionally, the paper is well-written and structured, contributing to its overall readability and clarity.

Weaknesses

Spatial Understanding and Evaluation: The use of synthetic data in the evaluation, while valuable, may introduce confounding factors unrelated to the task of spatial understanding/reasoning. * For instance, a vision-only model with limited OCR capabilities may perform poorly on the spatial-map task, regardless of its actual spatial reasoning ability. It is challenging to disentangle the contributions of OCR performance from spatial understanding in such benchmark. * The synthetic maze-navigation and spatial-grid images might be out-of-distribution for some models, especially open-source ones. This could also explain why a noise image improves the accuracy of Llava-1.6 on maze-navigation tasks, while a similarly out-of-distribution maze image does not necessarily harm or consistently improve the performance. It is important to consider the potential impact of out-of-distribution data on open-source models. By looking at fig 12 it looks the performance of the latest gpt4o on vision-only is similar to vision-and-text. That said, I still believe the work remains valuable and important. Human Performance Comparison: * The claim on line 172 that "the performance of these models still lags significantly behind human levels" would benefit from concrete numbers to substantiate it. Providing specific human performance metrics on these tasks or citations here could would make the claim more robust. Alternatively, softening the statement to reflect the need for further comparison could be considered. Sampling Strategies and Prompting Techniques: * Exploring different sampling strategies or reasoning/prompting techniques could yield valuable insights into the model's performance. It would be beneficial to include discussions on how these variations impact the results. To my understanding only a simple prompting technique is utilized "step-by-step explanation". Was there any particular reason to append the selected prompt vs others. In line 157, the paper also mentioned "For each model, we adopt the default configurations and decoding strategies, e.g., argmax for deterministic decoding and top-p for non-deterministic decoding. " It would be very useful to be more specific, e.g. what top-p and temperature was used. How "deterministic" was the deterministic decoding in API based models? VQA Evaluation Benchmark Details: * The paper would benefit from clearly specifying the size of each VQA evaluation benchmark in terms of the number of samples or data points. This information is crucial for understanding the scope and scale of the evaluations conducted. I might have missed it but I couldn't find it in the paper. I would encourage the authors to particular address this in rebuttals.

Questions

please see above

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. It is discussed and properly addressed.

Summary

The authors make a benchmark to test the spatial reasoning of multimodal language models (MLMs) using synthetic data consisting mostly of diagrams, mazes, and charts. They benchmark a lot of existing MLMs. Their key findings are: 1) Spatial reasoning is limited in most MLMs. 2) MLMs rely heavily on textual information and not the visual, and 3) interestingly, the VLMs are better at utilizing spatial text information than LLMs suggesting multimodal training helps understand the visual natural of language even though VLMs tend to rely less on visual information during test time.

Strengths

The idea of stress testing MLMs with spatial and figure understanding has been a recent trend and is very useful to push these models to do real-world tasks. The analysis done is interesting and provides some useful insights into how these models use visual or textual information.

Weaknesses

- Some of the claims may be too broad to make based on results of just this dataset. For instance, the models using more textual information than visual. This could simply be because the visual inputs in this benchmark are of a starkly different domain than what the models are trained on (real images instead of synthetic graphs and mazes and charts), whereas the domain gap in language is much lesser since the language tokens remain the same. Hence, we can only make the statement that the model is using more textual cues for such kinds of data. It is unclear if that is also the case on real image data. - There are many ways to test spatial reasoning using real VQA style datasets like BLINK, MMBench etc. So , it is unclear why they came up with these complicated tasks. Is there any real-world relevance for such tasks? - The authors mention they opt for synthetic data due to controllability, but all the analysis they do can be simply done on real image QA datasets like GQA. For instance, for checking vision vs textual reliance, just feed in scene graphs vs the image to an MLM from GQA. Or, for mismatched image and text, randomly pair image and QA. At least a discussion on why such a benchmark is needed is missing. What special analysis in the paper is enabled by this benchmark that couldn't be enabled by an already existing benchmark like Visual Genome, GQA, BLINK etc. - Some settings could be explained more clearly in the paper. For instance, for the vision-only input - I am guessing the authors mean that they do not describe the image in text, but the question is still in text, correct? - A nit pick, but some grammar could be improved. e.g., intro, the first line (line 18):" had a transformation effect on research" - a transformational effect not affect. Other grammatical errors are also occasionally present in the paper. - Another nit pick, but this paper seems more fitting for the datasets and benchmarks track.

Questions

See above concerns. Overall I like the direction of the paper. But some analysis justifying the need of such type of synthetic data (mazes and custom diagrams) instead of real images or actual graphs (useful for scientific figures) would be nice.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors discuss some limitations of models, but not the benchmark. What can this benchmark power, and what are some analyses that the benchmark currently cannot power that would be good to have?

Summary

The paper proposes a set of synthetic tests to compare the spatial understanding in VLMs and LLMs. The tests include Spatial-Map, Maze-Nav, and Spatial-Grid, which all include an image, paired with text that describe the image, and a question. Using the synthetic data, the VLMs and LLMs are studied using different modalities, including VQA, TQA, and VTQA, where the models are asked to answer questions based on only the image, only the text, and both image and text. Findings include that the current VLMs are strongly biased towards texts (blind); VLMs outperform LLMs in TQA. Both open models and proprietary models have been tested.

Strengths

1. The TQA, VQA, VTQA test is a smart way to reveal the modality bias in VLMs and LLMs, clearly showing the blindness of VLMs. 2. Intensive results and analysis are provided.

Weaknesses

1. All datasets contain only 2D synthetic images, which is substantially different from real images. At least the images can be made more realistic using computer graphics, like [4,5]. 2. The synthetic nature of the tasks introduces artifacts. Good spatial reasoning ability on this task may not generalize to real ones. The results may be highly dependent on whether the model has been trained on a similar task. For example, the performance of GPT-4O almost doubles that of Gemini in this paper, while the difference of these models on real images is not this significant. 3. The observation that VL models are highly biased towards text, and are “blind” visually, has been studied long in the VQA community, even before LLMs [1,2,3, etc.]. The related literature should be discussed. [1] Eyes wide shut? exploring the visual shortcomings of multimodal llms [2] Explicit Bias Discovery in Visual Question Answering Models [3] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering [4] Clevr: A diagnostic dataset for compositional language and elementary visual reasoning [5] Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning

Questions

1. How does the findings in this paper differ from previous literatures like [1,2,3]. Are the findings different, or are they discovered in a new way as proposed in this paper? 2. Any discussions about the synthetic-real gap? 3. In Spatial-Map, without text descriptions, it looks like the VLMs may describe spatial using “left/right/up/down” instead of “west/east/north/south” - will this happen, and how is it addressed?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper develops a novel benchmark to understand spatial reasoning ability of LLM and VLM. Using such a benchmark, authors conduct experiments to evaluate models' performance, and reveal several results.

Strengths

1. This paper is well-written and well-structured. 2. This paper conducts a series of experiments and include very recent LLMs and VLMs.

Weaknesses

1. The design rationales of the three benchmarks are not included or discussed. "Spatial-Map" and "Maze-Nav" look reasonable but "Spatial-grid" seems ill-posed. Would we really meet any similar application scenario in the real world? 2. In the experiments of "The impact of input modality", LLMs and VLMs take very different inputs in the benchmarks, with no part overlapping. In this case, it is unsure whether such a comparison is fair or even valid. 3. It is unclear why the results of "Spatial Map" are not provided in main paper or appendix. In addition, the performance of some VLMs are also missing. Given that it can be already observed that there exist some outliers, it is important to show full results of the study to ensure the validity of observations. 4. LLaVA-1.6-34B is the most performant model in this model family. However, in Figure 7-9, it consistently under-performs compared to other LLaVA models on Spatial grid. I would like to see authors' comments on these finding. What might be the cause to the weakness of LLaVA-1.6-34B? 5. This paper overall provides quantitative results of current VLMs without deeper analysis. It remains unclear what on earth causes the weakness of these VLMs. What are insights or takeaway to improve VLMs?

Questions

See weakness.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

No negative societal impacts of the study.

Thanks for the response

I thank the authors for the very comprehensive rebuttal and new experiments added to the papers. I have also read and appreciate other reviewers comments and the authors response. I think this is an important paper, technically solid with potential moderate-to-high impact. I keep my score of 6, and recommend acceptance.

Thanks for the response

I thank the authors for providing the rebuttal. The rebuttal discusses the differences with prior works. I also appreciate the effort for experimenting with real dataset - it is interesting to see that the modality gaps becomes even larger on real image. I am still concerned about the artificially created toy tasks in the paper. I think more experiments with real data, or semi-real data like rendered images using graphics (e.g. CLEVR) that are 3D, will make the paper much stronger. I will keep my score as 5.