Thank you for engaging in an in-depth discussion with us during the rebuttal period!
## **On implicit bias (and level of difficulty) of different modalities**
> I still feel that the task of comprehending the piece positions and relationships is easier in the text modality. Assume that the location of the rook in Figure 2 is d8 – this location is explicitly mentioned in the text representation by three dots before ‘r’. The text position encodings can easily tell that the location of the rook is 4th from the left. In addition, the text representation explicitly mentions the character ‘r’ that disambiguates the presence of the rook itself.
We just want to point out an interesting observation that we think gives some evidence against this point. In our experiments, the best-performing text representation was, in fact, PGN, which does not describe the board state nor locations of pieces (as in the other representations), but instead records a trajectory of moves from the start of the game. As an example, here’s the **PGN for the board in Fig. 2**: 1. e4 e5 2. Nf3 Nc6 $\cdots \cdots \cdots$ 27. Ke3 Qf3#. This representation is **harder than just getting the current board state** (as in ANL and image representations) as it must do state-tracking to infer the current position. This gives an example of how a more-explicit representation of the current board state, which seems intuitively easy to us, is not always the easiest for these models.
## **re: usefulness for practitioners**
Thank you for this suggestion! We will add to our paper an in-depth discussion on the usefulness of IsoBench for practitioners. We share a curtailed version below.
Regarding the graphs, there are many natural cases where a practitioner (e.g., a data scientist) starts with a symbolic (tabular) representation, and then may find it natural to convert it to a visual plot. In these cases, they have a choice of which representation to feed to the model, and our results tell them which is likely to be superior.
Similarly, for finance users who need to work with Markov Chains – even though visualization as chains/graphs are more interpretable to humans – our results tell them it’s better to provide LLMs with transition matrices of the Markov Chains instead of their graphical visualizations.
## **re: no experts can write accurate text representations**
We agree that there are tasks that are more naturally grounded in image than text, e.g. the case of analyzing scientific plots presented in papers. Our math tasks are deliberately designed to assess necessary properties for analysis automation, such as counting, curvature, and symmetry. The models’ inability to reliably perform on these tasks serves as a cautionary tale to warn practitioners from overtrusting VLMs to carry out these analyses without expert insight.
## **re: same model for image captioning**
If an expert themselves is using the LLM to help automate tasks, they could add text descriptions as needed. It's valuable for them to know that this is likely to be helpful.
On the curious case of the effectiveness of IsoSP, our hypothesis is that VLMs are better at *perceiving* than *reasoning* with visual modalities. High-quality, textual reasoning dataset are abundant (e.g. [1]), but their VL counterparts (e.g. [2]) are scarce. Thus, reasoning with imperfect captions may outperform reasoning with complete visual input, due to the lack of quality training for VL reasoning. We aim to investigate this hypothesis in future works.
[1] Platypus: Quick, Cheap, and Powerful Refinement of LLMs, https://platypus-llm.github.io/
[2] LLaVA Visual Instruct 150K, https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K