GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., $\textit{visual graph}$) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called $\textbf{G}$raph to v$\textbf{I}$sual and $\textbf{T}$extual Integr$\textbf{A}$tion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish $\textbf{G}$raph-based $\textbf{V}$ision-$\textbf{L}$anguage $\textbf{Q}$uestion $\textbf{A}$nswering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

Paper

Similar papers

Peer review

Reviewer oE8d6/10 · confidence 4/52024-06-26

Summary

The paper introduces the GITA framework, which innovatively integrates visual graphs into general graph reasoning tasks for LLMs. By combining visual and textual information of a graph, GITA improves the comprehensibility and flexibility of graph reasoning, outperforming other LLM approaches. The authors also develop the Graph-based Vision-Language Question Answering (GVLQA) dataset, the first vision-language dataset for graph reasoning. The study also highlights the benefits of layout augmentation in visual graphs and pretraining on the GVLQA dataset.

Strengths

1. Integrating visual graphs into graph reasoning tasks for LLMs, is a novel approach. This represents a creative combination of visual and textual modalities to improve graph comprehension and reasoning. The creation of the GVLQA dataset is original and fills a gap in the current datasets available for graph reasoning by incorporating both visual and textual elements. 2. Extensive experiments on both the GVLQA and five real-world datasets validate the effectiveness of the proposed framework, providing strong empirical evidence for the paper's claims. 3. The paper is clearly written, with well-defined sections and logical flow. The detailed explanations of the GITA framework's components and the dataset construction process enhance understanding. 4. By successfully integrating visual information into graph reasoning tasks, the paper addresses a significant limitation of existing LLMs and VLMs. This has the potential to substantially advance the field of graph reasoning.

Weaknesses

1. The motivation of the framework is challenged. As for the commonly used graph reasoning methods such as GNNs, the authors state that "These methods often lack generalizability, flexibility, and user-friendliness." However, the advantages of GITA in these areas are not directly demonstrated. The zero-shot performance of LLM-based methods is poor for many questions (such as MaxFlow, SP, TS). Considering this, users need to fine-tune when addressing questions on a new dataset. Additionally, GITA requires data augmentation and manual template-based construction for task-specific queries. Therefore, it is not clear that LLM-based methods like GITA are better than GNNs in terms of generalizability, flexibility, and user-friendliness, and the motivation for using LLM for graph reasoning is not justified. 2. While the paper compares GITA to various language models, it does not provide a comparison to dedicated GNN models that are designed specifically for graph reasoning tasks. A comparison to GNNs would help demonstate the performance gains of GITA and its advantages or disadvantages. 3. The paper mentions k-hop subgraph sampling for handling large graphs but does not provide a detailed analysis of GITA's scalability or performance as graph sizes increase. A more comprehensive evaluation of scalability, potentially including comparisons to dedicated graph methods, would be valuable for assessing the practical applicability of GITA in real-world scenarios. 4. The alignment between visual and textual modalities could be explored. The paper notes performance degradation in some tasks for larger models due to potential alignment issues, but it does not provide a detailed analysis or solution. Further investigation into improving modality alignment is needed. 5. Some places need double-check. For example, the meaning of GITA needs to be unified: the authors mention GITA as "Graph to vIsual and Textual Integration" in line 44, while they refer to it as "Graph to Image-Txt Assistant" in line 117. Another example is in line 120, which needs to be changed to "Firstly, V and D are designed to produce visual depictions and textual descriptions."

Questions

Please see the above weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have acknowledged several key limitations of their work but could benefit from providing more comprehensive solutions and experimental evidence. As for my suggestion for improvement, please see the “weaknessess” part.

Reviewer gmAJ6/10 · confidence 4/52024-07-03

Summary

To fill the gap that LLM overlook the rich vision modality with graph structure, this paper proposes GITA to incorporates visual graphs into general graph reasoning. A large graph vision-language dataset called GVLQA is designed to boost the general graph reasoning capabilities.

Strengths

1. This paper may involve a relatively large workload, proposing a large vision-language dataset, which can greatly promote the development of VLM. 2. Integrating visual graphs into VLM holds significant value.

Weaknesses

1. This paper appears to be quite technical or engineering-oriented, with the models largely based on existing tools, and lacks strong novelty in terms of methodology. 2. This paper primarily focuses on integrating visual graphs into large models, however, it fails to provide any specific examples of visual graphs throughout the paper, making it difficult for readers to comprehend how the visual image improve the performance. 3. In page3 line 120, the 'G' in the end actually refers to 'D'? Some explaination should be provided.

Questions

The dataset will be released in the future? If so, I appreciate it.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

See the weakness

Reviewer 6MUx5/10 · confidence 4/52024-07-04

Summary

The paper introduces an end-to-end framework called Graph to Visual and Textual Integration (GITA) to visualize graphs in order to improve LLMs’ reasoning capabilities on graph tasks. GITA consists of three main components: a Graph Describer to translate a graph into a natural language description, a Graph Visualizer to visualize the graph as an image, and a Questioner to generate task-specific queries conditioned on the task information. GITA shows improvements compared to vanilla LLMs and VLMs on graph reasoning tasks and some real-world graph datasets.

Strengths

1. The motivation is convincing. Just as humans reason over structured data, it is natural to visualize the structure first. The additional visual modality can provide rich information that can assist in reasoning over structured data. Therefore, generalizing this human behavior to LLMs/VLMs makes a lot of sense. 2. The method is simple yet effective for some graph reasoning tasks on small graphs. The idea of visualization is a form of data augmentation.

Weaknesses

1. The applications are limited. The proposed method is only applicable to tiny graphs or large graphs with k-hop sampling, which hurts its practical application value. 2. The design of key components, including the Graph Describer, Graph Visualizer, and Task-specific Query, requires human efforts to adjust to fit different datasets. 3. Lack of comparison with SOTA methods. The experiments only compared with LLMs and VLMs; some graph LLMs should also be compared, such as Graph Chain-of-Thought, GraphLLM, and GraphToken [1-3], etc. [1] Graph Chain-of-Thought, https://arxiv.org/abs/2404.07103 [2] GraphLLM, https://arxiv.org/abs/2310.05845 [3] GraphToken, https://arxiv.org/abs/2402.05862

Questions

1. Can you provide some statistics about the graph size (average number of nodes and edges) used in your dataset? 2. In Table 1, is GITA-7B (VO) equivalent to Llava-7B? 3. In Table 1, in the fine-tuning setting, how are the LLMs fine-tuned? Is GITA-7B fine-tuned on only the alignment projector or the whole model?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

See weekness above

Reviewer SxbK5/10 · confidence 4/52024-07-12

Summary

This paper introduces Graph to Visual and Textual Integration (GITA), a novel framework that enhances graph reasoning by integrating visual representations with traditional text-based processing. GITA's innovation lies in using both visual and textual inputs to address graph-structured data, a significant deviation from the typical text-only approaches in Large Language Models (LLMs). Key contributions include the development of GITA, the creation of the Graph-based Vision-Language Question Answering (GVLQA) dataset—the first for visual-textual graph reasoning—and extensive evaluations showing GITA's superior performance over existing LLMs across various datasets, highlighting the advantage of incorporating visual information into graph reasoning tasks.

Strengths

1. The paper introduces the novel idea of integrating visual graphs with textual descriptions for enhancing graph reasoning tasks. 2. The creation of the GVLQA dataset is a significant contribution, as it is the first vision-language dataset designed specifically for general graph reasoning.

Weaknesses

1. While I think it is an interesting idea to introduce vision into graph reasoning tasks, I am skeptical about its value. This is because vision is a perceptual ability, which is a fast intelligence (i.e., system one). While graph reasoning is a task that requires rigorous inference modeling and step-by-step execution to get the final precise answer, e.g., the graph reasoning tasks covered in this paper have their corresponding graph algorithms to get the precise answer. In my opinion, compared with the visual ability, the ability to rigorously execute traditional graph data structures and algorithms is the key for LLM/MLLM to have the ability to solve graph reasoning tasks. The experiments in Table 1 also show that, except for Connect and Cycle, which can be answered quickly and visually (provided the graph layout is concise and clear), the other graph reasoning tasks, GITA (VO) do not perform well. 2. I think this paper is like a data track paper, i.e., proposing novel datasets, testing the capabilities of existing LLM/MLLM, and finally proposing viable solutions for testing. Instead, this paper is counterproductive in wrapping it up as a methodological framework paper. The biggest problem is that the graph visualizer, the graph describer, and the questioner are both part of the methodology and the means of constructing the dataset, which is very confusing. I don't think they fit as part of the methodology because they are just tools for engineering the dataset, and there is no innovation at the level of ideas, modeling, or anything else. 3. Table 1 compares only text-based LLMs and lacks a comparison with visually aware MLLMs such as GPT-4V/GPT-4O/LLaVa/Gemini.

Questions

please refer to the weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Reviewer oE8d2024-08-09

I thank the author for the rebuttal. It addressed most of my questions and I am increasing the score to 6.

Reviewer gmAJ2024-08-12

Thanks for the rebuttal. My concerns are addressed, therefore I maintain my score.

Reviewer 6MUx2024-08-12

Response

Thank you to the authors for the rebuttal. I have carefully reviewed it, as well as the responses to the other reviewers. Regarding W1, my concern about the limited applications remains. I acknowledge that k-hop sampling is a widely adopted approach and may not lead to performance degradation in some cases. However, for datasets like Cora and CiteSeer, the reported accuracy is much lower compared to pure GNN methods[1] (Table 3). I am uncertain whether this is due to the k-hop sampling or because GITA only provides graph structures without node/edge attributes. If it is the latter, then GITA may not be effective enough for handling graphs with attributes, whether in numerical or text format. For W3, the average performance of GraphLLM and GITA is similar. I am conservative about the contribution of GITA over existing methods. For the new experiments during rebuttal, for table 5-7, the comparision with GNNs is an important baseline and should be displayed in the main text. Besides, for small datasets (Table 5), simple GNN method has very close performance with GITA. For large datasets (Table 6,7), pure GNN method show better performance and significantly better efficiency. I am skeptical about the necessity of using LLM with visual information to address these tasks given simple GNNs have already perform good with great efficiency. For W2 and Q1-Q3, my questions are well addressed. [1] cora leaderboad, https://paperswithcode.com/sota/node-classification-on-cora

Authorsrebuttal2024-08-12

Thanks for your detailed, patient and insightful discussion. We respond to these questions as follows. > **Q1**. Regarding W1, my concern about the limited applications remains. I acknowledge that K-hop sampling is a widely adopted approach and may not lead to performance degradation in some cases. However, for datasets like Cora and CiteSeer, the reported accuracy is much lower compared to pure GNN methods (Table 3). I am uncertain whether this is due to the k-hop sampling or because GITA only provides graph structures without node/edge attributes. If it is the latter, GITA may not be effective enough for handling graphs with attributes, whether in numerical or text format. We think the reported inferior performance in Cora and Citeseer to the leading models is mostly because we do not use node attributes, but not due to the k-hop sampling. That is because many methods (e.g., Graph Transformer like UGT, and most GNN variants where $k$ is equivalent to the number of GNN layers) in your listed leaderboard also use k-hop sampling to achieve leading performance. However, we want to point out that though using extra information beyond graph structures like node attributes is helpful (in Cora and Citeseer, they provide extra 0-1 binary word occurrences info), it requires specific models to handle them, thus hurts the generalizability, conflicting with our motivation, i.e., a "general-purpose" graph reasoning framework. Note that "generality" is a significant reason for why research works like GraphLLM and our GITA are interested in using LLMs for graph reasoning, where GraphLLM also does not consider the node attributes. The hurt for the node attributes to generalizability can be reflected in several aspects: First, handling node attributes typically requires designing specific model tailored to their shapes (e.g., vector dimensions, length, matrix size), which hinders a general and consistent solution for handling diverse tasks. Second, the meanings (e.g., word vectors of titles, embeddings of degree or index, one-hot representation of node classes) of node attributes vary in data, thus the model can overfit in a specific task and hurt the zero-shot abilities. Finally, these diverse models to handle the node attributes also increase the complexity of the model. As the major concern of GITA is how well vision+LLM performs on general graph scenarios, where node attributes are not necessary provided by default, we only concentrate on pure graph structures and do not incorporate task-specific models for handling extra node attributes. Moreover, we can provide several potential solutions for combining GITA with node attributes, and leave them for future work. 1. We can include the text node attributes with the explanation of their concrete meanings inside the text prompt. However, such an approach may not perform well for some types of attributes such as occurrence data, because they are too abstract. 2. Similar to many existing works, we can use a specific module to encode the attributes and another fusion module to combine them with the backbone model (i.e., MLLMs used in GITA). However, such an approach needs extra designs for these additional modules and needs task-specific adjustments. In the following table, We also provide an additional comparison between GITA and dedicated GNNs under non-attribute setting on Cora and Citeseer, where non-attribute setting is commonly used to evaluate how well models can understand graph structure. The experimental results show that GITA is much more effective in non-attributed Cora and Citeseer than dedicated GNNs, showcasing a more powerful awareness of their pure graph structure. | Model | Cora | Citeseer | |-------|-------|----------| | GITA | 85.24 | 75.07 | | GCN | 73.35 | 68.71 | | SAGE | 69.19 | 64.69 |

Authorsrebuttal2024-08-12

> **Q2**. For W3, the average performance of GraphLLM and GITA is similar. I am conservative about the contribution of GITA over existing methods. We would like to highlight that the contributions of GITA are unique compared with existing methods such as GraphLLM. **1. Pioneering Use of Vision in Language-Based Graph Reasoning**: GITA is the first to explore the effectiveness of vision in language-based graph reasoning. We believe that this vision benefit (our contribution) is not covered by existing methods like GraphLLM. As illustrated, our experiments show that GITA and GraphLLM excel in different types of graph understanding tasks. Therefore, the methods proposed by us (vision modality) and by them (graph modality) are expected to be combined and complement each other in our future work. **2. Superior Performance**: Though GraphLLM and GITA-7B have similar average performance, when apply our proposed vision augmentation, GITA-VO(AUGLY) can be more powerful than GraphLLM in overall performance (i.e., 63.36\% in Table 2 vs. 60.36\%). Besides, zero-shot GITA can also achieve comparable average performance (59.73\% in rebuttal supplement Table 3) with GraphLLM, where the former is training-free. ------------------------------ > **Q3.** For the new experiments during rebuttal, for table 5-7, the comparison with GNNs is an important baseline and should be displayed in the main text. According to your suggestion, we will add them accordingly in the main text of the revision. > **Q4.** I am skeptical about the necessity of using LLM with visual information to address these tasks given simple GNNs have already perform good with great efficiency. GITA and GNNs highlight the distinction solutions between general-purpose solutions and specialized solutions. As discussed in our motivation and in our response to W1 of Reviewer oE8d, GNNs are not sufficiently flexible, general, or user-friendly for addressing general graph reasoning tasks, with the following points: **1. Ease of Adaptation (Flexibility)**: GNNs often need to be combined with other models to meet specific task requirements. For instance, sequence output tasks (such as SP, HP, TS) require integration with LSTM, transformers, etc. In contrast, GITA can achieve this with a unified model architecture. Besides, adapting GNNs to specific tasks requires modifications within the model structure, necessitating a background in deep learning and coding skills. GITA, on the other hand, only requires language skills that all humans possess to fit task variations. **2. Zero-Shot Capability (Generalizability)**: GNNs lack zero-shot capabilities, whereas GITA has promising zero-shot capabilities (as demonstrated in rebuttal Table 3). **3. Human-Readable Operations and serve as agent (User-friendliness)**: GNNs operate on unreadable vectors, while GITA operates on human-understandable language and images. Therefore, though it is not as efficient as GNNs, it can serve as an agent for answering graph questions with natural language in acceptable time, whereas GNNs show inabilities because they can not handle language. Therefore, GITA is a more flexible, general and user-friendly framework for general-purpose graph reasoning than GNNs. ------------------------------------ **To sum up**, this paper is the first work to conduct general-purpose vision-language graph reasoning, and has illustrated that vision could bring overall advances on LLM-based graph reasoning. As an initial direction, it is not as mature as GNNs with years explored. However, it has shown comparable overall performance and special superiorities (flexibility, generalizability, user-friendliness, and promising zero-shot capabilities), demonstrating it is a promising direction to explore as a general solution for graph reasoning.

Reviewer 6MUx2024-08-13

Response

Thank you to the authors for the detailed response. I have no further questions and will raise my score to 5.

Reviewer SxbK2024-08-12

My concerns are addressed and I hope that the final version of the paper will be updated.

Authorsrebuttal2024-08-13

Dear Reviewers, We extend our heartfelt gratitude for your dedicated time and effort in reviewing our paper. Your constructive questions and valuable feedback have been immensely beneficial to our manuscript, and we will incorporate them into our revisions. Thank you once again for your invaluable contributions to the review process. Best regards, Authors

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC