Investigating Instruction Tuning Large Language Models on Graphs

Summary

This paper explores the application of LLMs to graph-related tasks through instruction tuning. A dataset including 14 graph tasks and approximately 63k question-graph pairs across academic and e-commerce domains is utilized to evaluate different graph representations and their impact on LLM performance. The study identifies JSON as the most effective format, achieving superior results across a range of graph types and tasks. It also examines the generalization capabilities of LLMs across three dimensions: unseen sub-tasks, domains, and answer types. The findings indicate that while LLMs exhibit strong generalization in some areas, they face challenges in others, notably inductive reasoning tasks such as link prediction.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. The exploration of representing graphs in JSON and DOT formats is innovative and provides valuable insights into optimizing LLM performance across complex graph structures. 2. The examination of domain generalization offers meaningful contributions to the field, demonstrating how LLMs can adapt to different graph structures and domains without prior exposure. 3. This paper adeptly highlights both the potential and the limitations of applying LLMs to complex graph-based problem-solving.

Reasons to reject

1. The methodology employed in graph instruction tuning is inadequately detailed and only briefly introduced in section 4.1.2. To enhance clarity and understanding, this method should be thoroughly described earlier in the paper, ideally in section 3, alongside a comprehensive explanation of the techniques and processes involved in graph instruction tuning, e.g., which loss is used. 2. The paper lacks a broad comparative analysis. It only evaluates the performance of two LLM backbones, neglecting other popular language model variants, such as those from the GPT series, which could serve as valuable baselines. Additionally, comparing these results with traditional graph methods and graph neural networks would significantly enrich the study’s insights and implications for the field. 3. There is no analysis of the scalability of the methods, which is a critical aspect to consider for practical applications. It is unkown the maximum graph size the methods can handle.

Questions to authors

1. Given the wide variety of graph datasets available, what motivated the choice of Amazon Metadata and MAPLE for your evaluation? Could you elaborate on how these particular datasets align with the objectives of your study? 2. Regarding scalability, could you detail the maximum graph size that your graph representation methods can effectively handle? What are the limitations in terms of node and edge count for each representation method? 3. The performance figures for LLaMA-2 and Mistral models without instruction tuning appear unusually low in Table 3, and seems like random guess. Could you provide some insights?

Summary

This paper presents a benchmark to evaluate the ability of large language models (LLM) in understanding graph structures. This benchmark contrains a diverse collection of 79 graph-related tasks from academic and e-commerce domains, featuring 44,240 training instances and 18,960 test samples. Besides the benchmark, the author also study the impact of different graph representations in affecting the performance of LLM and find that JSON is a better choice. In addition to that, the author also study how LLM can generalize to unseen examples.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. Present a solid benchmark containing a diverse collection of 79 graph-related tasks from academic and e-commerce domains, featuring 44,240 training instances and 18,960 test samples. 2. The paper is good written and easy to follow. Every detail is presented clearly and reading friendly for unfamiliar readers. 3. The analysis about different graph representations and the ability on unseen examples gives new insights.

Reasons to reject

1. The proposed benchmark only consider network data in academy and e-commerce and ignore widely used graph data in the domain of chemistry and biology. 2. About novelty. This is not a brand new benchmark. Many previous works has already considered the ability of LLM in understanding graph structures. For example 3. The paper only evaluates the ability of Llama2 and Mistral, which is hard to cover all representive LLMs. 4. Limited technique contribution. This paper only gives evaluation results and insights and the technique used here on how to integrate LLM and graph follows previous methods.

Questions to authors

1. Can you compare more LLMs beyond Llama 2 and Mistral? 2. Can you elaborate on the technique contribution? 3. What is the gap between small models on graphs and LLM in understanding graph structure?

Ethics concerns details

No

Summary

Authors propose a new dataset of graphs with associated tasks to answer given a graph representation. Authors use LORA to adapt LLM to enhance its understanding of the graphs. Graphs represented by one of three formats. Authors conclude that JSON is the best format with advnatage for adapted LLMs in performance.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

(1) Understanding strcutured data within the context of LLMs is quite important with wide set of applications (2) Authors propose a useful dataset that will help other researchers studying the problem.

Reasons to reject

(1) Graph representation approaches proposed are all rule based, it would have been great to propose a learnable approach to graph such as an encoder based representation. (2) The datasets are small and diverse enough.

Questions to authors

(1) Could you have written the queries in natural language, is there a need to use phrases such as 'also_buy' or 'product_of'? (2) Was the LLM allowed to generate tokens that are out of the possible set of answers? For example, if the task is counting, could the LLM generate an answer than is not in digits? Could you have tested the performance of the LLM scoring the possible outcomes instead of generating novel responses? (3)

Some current experiment results of Gemma-7b on Amazon Metadata dataset

**Add to Table 3:** | | Node | Pair | Count | Bool | Path | Graph | LP | AVG | | ----------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | | Gemma-Inst_natural | 13.85 | 25.20 | 2.59 | **65.92** | **34.64** | 28.85 | 44.96 | 24.42 | | Gemma-Inst_json | 15.50 | 26.54 | **6.67** | 65.15 | 30.00 | 29.45 | 36.00 | 24.74 | | Gemma-Inst_dot | **16.83** | **35.91** | 4.06 | 64.76 | 31.19 | **37.69** | **60.33** | **28.72** | | Gemma-GraphInst_natural | 87.34 | 76.30 | 46.51 | 97.36 | **68.06** | 77.32 | **85.17** | 75.43 | | Gemma-GraphInst_json | **90.15** | **78.11** | **49.98** | **99.24** | 65.68 | 78.08 | 82.42 | **76.98** | | Gemma-GraphInst_dot | 87.43 | 78.09 | 47.96 | 96.36 | 67.43 | **83.51** | 83.94 | 76.37 | The results show JSON format provides the best performance after graph instruction tuning. **Add to Figure 3:** | | | Node | Pair | Count | Bool | Path | Graph | LP | AVG | | -------------------- | ------ | ----- | ----- | ----- | ------ | ----- | ----- | ------ | ----- | | Gemma-GraphInst_json | Seen | 93.69 | 82.97 | 62.60 | 100.00 | 81.07 | 84.22 | 100.00 | 85.02 | | | Unseen | 87.19 | 75.69 | 37.36 | 98.47 | 50.30 | 71.94 | 47.25 | 69.71 | The results show the graph instruction tuned LLM generalizes well on unseen sub-tasks of Node, Pair, Bool, and Graph tasks, with a small drop in the performance, but fails to generalize to the unseen sub-tasks of Count, Path, and Link prediction tasks.