LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings

Summary

This paper proposes TEA-GLM to align the GNN representations with LLM token embeddings for zero-shot graph machine learning. TEA-GLM enables cross-dataset and cross-task learning without fine-tuning the LLM. Extensive experiments demonstrate its state-of-the-art performance on unseen tasks and datasets.

Strengths

The paper is well-written and easy to follow, effectively conveying the design and motivation behind LLM-GS. The proposed framework employs only a linear projector to incorporate with the LLM, which is efficient. The idea of mapping graph embedding to token embedding is also interesting.

Weaknesses

The experimental results are somewhat questionable. The vanilla Vicuna-7B even outperformed LLaGA, which leveraged Vicuna-7B as its base model. Furthermore, the results on E-commerce-Children show no difference between Vicuna-7B and the proposed TEA-GLM. From my perspective, this paper marks LLaGA as the strongest baseline. However, I find that the experimental settings are totally different from LLaGA. Is it possible to conduct some experiments following LLaGA? The ablation studies are also questionable. Even without GT or without FC, the model achieves at least equivalent performance to that of Vicuna-7B, right?

Questions

Please see Weaknesses.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

The paper propose a GNN plus LLM model for zero-shot learning in the graph domain. They first use contrastive learning to pretrain a GNN that can be applied to arbitrary graphs. They added a feature-wise contrastive learning objective on the projected representation to the PCA of LLM token embeddings to bridge the gap between contrastively learned embedding and LLM token embeddings. After training the GNN, they train additional linear projector to the map the GNN output to a sequence of graph tokens, and the tokens are combined with special prompt designed for graph learning tasks for LLM to perform downstream tasks. The methods does not need to fine-tune the LLM and acheived good performance on most of the datasets.

Strengths

The PCA-based alignment with the LLM input is interesting, and potentially can shed lights on future research on GNN and LLM alignment. The experimental results seem promising on most datasets compared to both LLM and GNN+LLM baselines. Especially, the model shows some-level of cross-task transferrability where the GNN and linear projector are pretrained only on node classification but, but show non-trivial performance on link prediction tasks.

Weaknesses

My major concern about the paper is its novelty. Several important components, including using self-supervised learning to generate general node embedding [1], alignning graph representation with LLM input with linear projector [2], and the prompt design are proposed and already widely adopted in the research area. The feature-wise contrastive learning seems to be the main technical contribution, yet it is not studied and invectigated in details. For example, why pre-training on node classification leads to non-trivial link prediction results? According to the ablation study, feature-wise CL seems to be the key, yet no knowledge about link-prediction is injected into the model in any step of the method. Related to the main concern, I think the experimental results do not fully justify the authors claim. The author claims that the aligned graph tokens provide the LLM with graph information. However, they still feed the node texts as direct text input to the LLM, for some datasets LLM can do well solely using node texts. To show that the pretraining process indeed brings in inductive/transferrable knowledge of the graph, the author should either show that by completely removing node text to from LLM text input (it is ok to keep task text), the model can still work or provide results of baseline models that tune the LLM on the node texts (tuning the LLM weight or use a soft prompt), and such baseline can not transfer or performs poorly. Note that the ablation study between w/o FC and w/o graph token is not enough. Essentially, if a soft prompt tuning can achieve similar effect, the proposed work is just adding linear layers as a substitute to soft prompts. [1] He, Yufei, and Bryan Hooi. "UniGraph: Learning a Cross-Domain Graph Foundation Model From Natural Language." arXiv preprint arXiv:2402.13630 (2024). [2] Tang, Jiabin, et al. "Graphgpt: Graph instruction tuning for large language models." arXiv preprint arXiv:2310.13023 (2023).

Questions

- Why do you need to map a single GNN node representation to multiple graph tokens? They should contain exactly the same amount of information. - What is the prompt used for baseline LLM? Is it the same prompt as the one used for GNN? If so, what happen when you provide it with a more detailed prompt, but still only on the target node? - For edge and graph tasks, you used the sum of node representation to compute graph tokens, wouldn't that break the aligned distribution as the pretraining only happens on node.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

The motivation of this work is to utilize the zero-shot learning capacity in LLMs for structural data. The basic idea is to connect GNNs with LLMs, by aligning its representations with the token embeddings of an LLM, such that GNNs can encode the structural information and LLMs can do zero-shot inference. The experiments were conducted with two different domains and seven baseline methods, and demonstrate the performance of the proposed method.

Strengths

- The research question is interesting and important. Generalizing the capacity of LLMs for zero-shot graph learner is a good idea to use the zero-shot capacity for structural information. - The proposed method is properly described with sufficient technical details. The pipeline of two-step adjustment is convincing. - Although more experiments can certainly be done, the current experiments answer three important research questions aligned with the topic, each of them is answered with a good amount of evidence.

Weaknesses

I do not have any major concern about this work. On the other hand, I do think the presentation of this work can be better. Some parts of the paper (especially section 2) are a little bit difficult to understand.

Questions

No further question, just some minor comments.

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations have been listed in the paper.