G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering

Summary

This paper introduces G-Retriever, which combines LLMs, GNNs and RAG for graph question-answering tasks. The authors first develop a more comprehensive benchmark named GraphQA. Then they present G-Retriever, which has four main steps including indexing, retrieval, subgraph construction and answer generation. Specifically, G-Retriever use an adapted PCST method, which also consider the importance of the edge semantics. The constructed subgraph and the query are then sent to the LLM for final answer generation. Extensive experiments validate the effectiveness and efficiency of G-Retriever.

Strengths

1. The introduction of the GraphQA benchmark fills a significant gap in the research community by providing a comprehensive benchmark for evaluating graph QA applications. 2. The motivation of the proposed G-Retriever is clear and the paper is well-structured. 3. Extensive experimental results demonstrate that G-Retriever can consistently outperform baseline models. 4. Codes are provided for reproducibility.

Weaknesses

- In Section 5.1 (Indexing), the authors use a pre-trained LM to encode the text attributes of nodes and edges into representations. However, there is no ablation study to demonstrate the importance of the node attributes or the text attributes, which leaves an unexplored gap in understanding the contribution of these features to the overall performance of the G-Retriever. - In Section 5.3 (Subgraph Construction), the authors use PCST to obtain a more refined subgraph. However, the motivation for selecting PCST is not clearly explained. Is PCST the optimal for this task? How the "optimally sized and relevant subgraphs" are defined? At least the authors could consider to justify this by comparing some naive approaches, such as fixed number or fixed similarity threshold. - The claim of the new benchmark is somewhat weak, as it primarily involves the reintroduction of an existing dataset. Additionally, the contributions related to the QA formulation and graph reformatting are limited. - The evaluation of hallucinations lacks sufficient detail and requires further elaboration.

Questions

Please see the questions in the Weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of their work in the paper.

Summary

This paper proposes a retrieval-augmented method G-Retriever for graphs with textual attributes. It introduces a graph question answering (GraphQA) benchmark by converting existing graph datasets into uniform format. Subsequently, it proposes a G-Retriever method to answer questions related to the textual graphs. Specifically, it first retrieves top-k most relevant nodes and edges from the textual graphs and then constructs a subgraph based on the retrieved nodes and edges using an existing algorithm called Prize-Collecting Steiner Tree (PCST). Subsequently, G-Retriever leverages an GAT model to obtain the pooled representation of the constructed subgraph, which is used as a soft prompt for an LLM to generate the answer.

Strengths

The idea of retrieving subgraphs from textual graphs to augment the LLM is interesting. Experimental results on three datasets in the GraphQA benchmark demonstrate the effectiveness of the proposed G-Retriever model. The code is provided for reproducibility.

Weaknesses

1. The paper mentions “chat with their graph” in the abstract and the introduction. However, it is not very clear what this concept means. Additionally, after the abstract and introduction, there are no further introductions or explanations about this concept. 2. In line 69, the paper states that the proposed G-Retriever can improve the explainability. However, there are no empirical analyses regarding the explainability of the proposed method. 3. The novelty of the proposed GraphQA benchmark seems limited, as it only converts three existing graph datasets to a uniform format. 4. I think the main novelty of the proposed G-Retriever model is that it leverages a PCST algorithm to find a connected subgraph between the retrieved nodes and edges. However, the paper does not clearly explain the advantages of retrieving a subgraph. Why is it necessary to retrieve a subgraph rather than directly using the texts of the retrieved nodes and edges as inputs for the LLM? Constructing a subgraph could potentially introduce noises, which might negatively impact the performance. Additionally, the paper does not adequately justify the use of PCST for finding the subgraph. Have the authors explored other alternative methods, such as the shortest paths between the retrieved nodes? 5. There are no quantitative or qualitative analyses regarding the quality of the retrieved subgraphs. Providing such analyses could offer some insights into the effectiveness of the subgraphs. 6. In the efficiency evaluation (section 6.3), the paper only compares the training times of G-Retriever and its variant without retrieval (I assume this is what “Before Retrieval” means, as there are no explanations about the columns in Table 4). However, it would be more appropriate to compare the inference times of G-Retriever and its variant during the inference stage. G-Retriever requires additional steps of retrieving nodes and edges, as well as constructing subgraphs using the PCST algorithm, which may result in longer inference times compared to not using retrieval. 7. A lot of necessary experimental details in section 6.4 and 6.5 are provided in the Appendix, which may hinder the readers’ understanding of the results and analyses in these sections. For instance, it is impossible to understand what “Valid Nodes”, “Valid Edges” and “Fully Valid Graphs” mean in Table 5 without referring to the Appendix. Additionally, in section 6.4, the paper uses manual examination to evaluate the hallucination performance of LLMs. However, it is unclear how many annotators were involved in this process and whether there is a potential for biases during the evaluation. 8. The texts in Figure 1 and Figure 2 are too small to read.

Questions

1. What does “chat with their graph” mean and how can the proposed G-Retriever model achieve this purpose? 2. How can the proposed G-Retriever improve the explainability? Are there any qualitative analyses to support this argument? 3. What are the advantages of using the PCST algorithm to retrieve subgraphs? 4. How to evaluate the quality of the retrieved subgraphs?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have discussed the limitations of their work in the paper.

Summary

This paper proposed a Graph Question Answering (GraphQA) benchmark with data collected from different tasks including ExplaGraphs, SceneGraphs, and WebQSP. Then, they proposed G-Retriever method, introducing the first retrieval-augmented generation (RAG) approach for general textual graphs, which can be fine-tuned to enhance graph understanding via soft prompting. G-Retriever performs RAG over a graph by formulating this task as a Prize-Collecting Steiner Tree optimization problem. The generation model is fine-tuned by a graph token and the textualized graph. Empirical evaluations show G-Retriever outperforms baselines on textual graph tasks from multiple domains, scales well with larger graph sizes.

Strengths

- This proposed GraphQA benchmark is comprehensive and comes in a timely manner. - Converting the problem of finding a connected subgraph that maximizes the total prize values of its nodes while minimizing the total costs of its edges to PCST is smart. - Strong experimental results by using the proposed G-retriever on all three datasets.

Weaknesses

I don't see major weaknesses of this paper. Several minor points: - I think several baselines are missing: (1) Simply feed the retrieved top-k nodes/edges (plus its neighbors) to the LLM would be able to show the effectiveness of PCST. (2) It would be helpful to understand the generation model by replacing the tuned generation model to a fixed LLM like GPT4, Claude, etc. - When encoding the nodes and edges, the context of the nodes/edges is missing, it could be beneficial to include the context (e.g., adjacent nodes) - Table captions should be more informative so that they are easier to understand (e.g., In Table5, what metrics are used?)

Questions

n/a

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

n/a

Summary

The work builds a new GraphQA benchmark for real-world graph question answering and presents G-Retriever, an architecture adept at complex and creative queries. Given a query and a graph, G-Retriever retrieves a connected subgraph from the original graph according to the query. The subgraph, a textualized graph, and the query are fed into an architecture consisting of a graph encoder aligned with an LLM. Experiments show that G-Retriever achieves SoTA in textual graph tasks across multiple domains, significantly improves efficiency, and demonstrates resistance to hallucination.

Strengths

1. The idea of incorporating retrieval for graph question answering is novel and interesting and could be inspiring for future works. 2. The proposed G-retriever framework performs well on the GraphQA dataset. The retrieval method scales effectively with larger graph sizes, which is not addressed in many previous works. Locating the related subgraph instead of using the entire graph also reduces hallucination. 3. The experiments are comprehensive, including analysis from various aspects such as hallucination and graph encoder selection.

Weaknesses

1. The framework seems to mainly work for larger graphs. For tasks with smaller graphs, such as the ExplaGraphs dataset with an average node number of 5.17, it is unnecessary to use such a framework. The performances of G-retrieval and GraphToken are nearly the same for the ExplaGraphs dataset.

Questions

1. After the subgraph is retrieved, is it possible to use neuro-symbolic methods (e.g. answer set programming or graph search) instead of using a graph encoder together with an LLM? It would be more efficient and cost much less.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Thank you for your response.

Thank you for your response. I am satisfied with the clarification and will maintain my positive review score.

Thanks for the response

Thank you for your response. I think adding those comparison does make the argument more solid and I will maintain my rating.

Thanks for your response

Thanks for the response. My concerns are well addressed. Overall, I think this work makes a good contribution to the area. I will improve my score to 7.