Summary
The paper proposes a new prompting method, "Visualization of Thought" (VoT) prompting, to enable LLMs to perform better on spatial reasoning tasks. In VoT, the LLM is prompted with instructions on performing a multi-step spatial reasoning task, followed by the text "Visualize the state after each reasoning step". The authors evaluate this prompting method on three types of spatial reasoning tasks, and show that it substantially improves performance on these tasks as compared with Chain-of-Thought prompting, as well as with other variations on prompting.
Strengths
VoT prompting is an interesting and (I think) novel approach to elicit better spatial reasoning in LLMs, and may be useful in multimodal systems as well (this is future work). The evaluated tasks are chosen to capture important aspects of spatial reasoning abilities. The paper reports on systematic experiments that evaluate these abilities on the chosen tasks.
Weaknesses
There are three categories of weaknesses which I believe can be improved on by the authors.
1. Some aspects of the paper are unclear and need more explanation.
-- The paper says "This method augments LLMs with a visuospatial sketchpad". It's unclear what is meant here by visuospatial sketchpad. This sentence makes it sound like there is a new architectural feature added to the LLM, but my understanding from the rest of the paper is that VoT just asks the LLM to generate a series of "visualizations" in its output. Please make it clearer in the description what exactly you mean.
-- Figure 4 gives examples of VoT prompting. But it wasn't clear to me whether these prompts include actual images (e.g., the pictures of barriers and people in the navigation task, or the pictures of colored and empty boxes in the visual tiling task) or whether these images in the figures were actually translated into some kind of text encoding for the LLM. Please make clear exactly what is in the prompt---images or text---and if text, what exactly is the text encoding of the pictures you are showing?
-- Equation 5 defines v_i as "visualizations". What exactly are these, in terms of LLMs? Are they text or image outputs from the LLMs?
-- In several figures (e.g., Figure 4, 6, 7) prompts and responses are shown but it needs to be made very clear which part is the prompt and which part the response from the LLM. These were not clear enough for me.
-- In Figure 8, again, what is part of the prompt and what is part of the response?
-- Figure 10: The caption only says "mental images for state tracking in visual navigation tasks." Please say more in the caption about what this figure is showing. Was "Use round pin" part of the prompt, or is this just describing what the LLM did? What are you trying to show in this figure? In general, please make the figure captions more informative.
-- "Specifically, we adopt GPT-4 and GPT-4 Vision" -- there are several versions of each of these LLMs on the Azure platform -- please specify which ones you used for these experiments.
-- "We adopted sub-string matching as f_correct to determine correctness."
This didn't really explain how you determined correctness. Please give more details (enough for someone else to replicate) and explain why this sub-string matching method is reliable for assessing correctness.
2. Scaling: The tasks used were relatively simple: 3x3 maps for navigation, 5x4 rectangle for tiling. Did you do any experiments with scaling up the size of these? How does this method's spatial reasoning abilities scale as the size is increased?
3. Anthropomorphism: The paper is filled with anthropomorphic language, which makes it difficult to understand. For example:
-- "LLMs possess the ability to create and manipulate mental images in the mind's eye." Using terms like "mental images" and "the mind's eye" assumes that the LLM is doing something analogous to the brain, which is not necessarily true!
-- "visual state tracking is implemented by generating the visualization v_i as
a mental image of the internal state si after each reasoning step z". A mental image is something a human has. Using that term for LLMs is unhelpful anthropomorphizing.
-- The paper calls the intermediate outputs of LLMs "thoughts" (e.g., "this process by adding a visuospatial sketchpad to each thought z_i") -- again, an unhelpful and confusing anthropomorphism.
-- "We demonstrate LLMs’ zero-shot capability of visualizing their precepts at abstract level". Visualizing is a cognitive ability of brains. Saying that LLMs "visualize" is an incorrect anthropomorphism.
-- ""this emergent capability to create mental images to enhance spatial reasoning resembles the mind’s eye process, suggesting its promise in MLLMs." Same comments as above.
Questions
-- I was confused by Figure 8b. The left side shows the house moving -- is that correct? Also, it doesn't do what the instructions say: e.g., "move down to the end of the continuous road" -- after the move the house only moved down one square, not two. Is this considered a correct solution? The right side shows a person moving. Where did the notion of moving a person versus moving a house come from?
Limitations
The authors have adequately addressed the limitations.