Summary
This paper proposes a new benchmark for evaluating the depth and height understanding capability of the vision language foundation models (VLMs). The benchmark has two subsets, one in 2D and another in 3D, both consisting of rendered images with synthetic objects placed on random real backgrounds. The models are asked about the depth order of the objects, or the height of a stack of rectangles. The questions are formatted into either multiple-choice or true-or-false. The authors tested the performance of 18 VLMs, including both open-source and closed-source ones, as well as human performance. All VLMs perform poorly on the benchmark.
Strengths
1. The presentation of this paper is clear and easy to follow.
2. The coverage of 18 VLMs is comprehensive. Moreover, human performance is also reported, which helps understand the difficulty of the proposed questions.
Weaknesses
1. Limited novelty and contribition for the depth task. The proposed 3D benchmark is highly based on existing the CLEVR dataset. CLEVR already contains questions requires reasoning about spatial relationships among objects, partially covering the depth task proposed by the authors. The CLEVR dataset has even higher complexity (up to 10 objects, compared to 5 used by the authors) and is more photo-realistic than the images shown in the paper. The authors should discuss more about the connection with CLEVR and what new aspects can the proposed dataset bring.
2. The height understanding task is novel to me, but importance of this task need justification. Depth is important for 3D understanding in human vision, but height is less straightforward, especially under the toyish setting of counting the height of a stack of building blocks. The authors should give some concrete example about how solving the proposed height task could benefit real-world applications of VLMs. Also the black platform makes the task a bit confusing to me, and the failure of VLMs may simply because they don't understand the setting.
3. The scope of the the task is limited (only 2, depth and height), and only synthetic images are used. This is a problem as more diverse and realistic benchmarks already exist. For example, a recent work BLINK[1] covers more fundamental vision tasks (including depth) with real-world images. The authors should discuss the connection with it in related work. Also since the authors use only synthetic images, it would be helpful to add discussions about the advantages or disadvantages of using synthetic images vs real-world images for benchmarking.
Overall, I feel the paper is trying to analyze an important and interesting question, i.e., the spatial relationship perception ability of VLMs. However, I feel that the current version of this paper doesn't meed the ICLR standard due to the limited scope, novelty and diversity of the proposed dataset. I would recommend the authors to cover a wider range of vision tasks in a more realistic setting. For example, covering more 3D vision tasks such as visual correspondence, object pose estimation, camera pose estimation, or using real-world images to make the benchmark more realistic.
Minor 1: L59 mentions 2 cats in Fig 1, but I see no cats in the figure. Is this a typo/mis-inserted figure?
Minor 2: From my understanding, Fig 4 and Tab 2 are basically displaying the same result, with Tab 2 simply merging the numbers for depth and height. This is kind of redundant and confusing, and I would recommend only keeping the table.
[1] Blink: Multimodal large language models can see but not perceive. ECCV 2024.
Ethics concerns
1. The authors use real-world background images to render their proposed dataset but don't mention the detailed source of those images. The license and copyright of those images needs to be checked.
2. The authors use human subjects to evaluate the human performance on their dataset but didn't mention the details of the human data collection process. For example, whether a proper IRB approval is obtained.