Summary
The paper studies inductive KG reasoning with unseen entities and relations at inference time and introduces KG-ICL, an in-context learning model for KG completion. Following Ultra [1], KG-ICL employs a two-stage approach: (1) obtaining relational representations based on the given graph; (2) performing entity prediction on the given graph using those relational representations.
The main difference with ULTRA is Stage 1, that is, instead of creating a graph of relations and learning tokens for meta-relations, KG-ICL mines several (around 5) examples of k-hop subgraphs around each relation type and applies a different labeling trick learning a different vocabulary of transferable tokens. Nodes are labeled following the distance encoding of GraIL [2] (tokens represent all combinations of pairwise distances up to hop $k$), relations are labeled simply by a binary indicator whether a given relation is the query relation or not (two tokens). Applying a GNN over each subgraph, their final representations are mean pooled to get a singe tensor of relational representations.
In Stage 2, those representations are used in a standard inductive pipeline for initializing edge type vectors, putting the query relation vector on the starting head node, and running another GNN to get the final predictions. Experimentally, KG-ICL demonstrates promising results and outperforms Ultra on inductive datasets while being marginally better on larger transductive datasets.
Strengths
**S1.** Foundation models for inductive KG reasoning on any unseen graph is a timely and important topic. New non-trivial approaches in this field are rare and this paper does a good job presenting and explaining the details clearly.
**S2.** A different approach for obtaining relational representations through few-shot subgraph examples of each relation (resembling the “in-context learning” scenario) and learning a transferable vocabulary - here it is an extended GraIL-based labeling with distance encoding for nodes and binary same/not-same indicator for relations (although Table 2 shows that learning tokens does not bring much benefits and labeling strategy is more important).
**S3**. Informative ablations, experiments on the Ultra benchmark involving 57 datasets
Weaknesses
**W1.** The main problem of KG-ICL is a different pre-training dataset mixture (inductive FB V1, inductive NELL V1, transductive Codex Small) which makes it hard to directly compare the performance numbers against Ultra (that used only transductive FB15k237, WN18RR, and Codex Medium). For a fair comparison, KG-ICL should have been either pre-trained on the same datasets as Ultra, or Ultra should have been re-trained on the new mixture. The new pre-training mixture consists of smaller graphs and is better suited for inference on smaller inductive datasets (where KG-ICL gains are the largest). While it is possible to pull the argument that LLMs and FMs are trained on different datasets (often undisclosed to the public) and evaluated on the same benchmarks, I believe the Ultra benchmarking datasets for KG reasoning are open and transparent enough for conducting fair evaluations.
**W2** Since KG-ICL focuses on the zero-shot inference performance (line 239), one missing experiment is to measure the performance as a function of training graphs in the pre-training mixture - for instance, Ultra provides several checkpoints with the growing inductive inference performance with more datasets added to the training.
References:
[1] Galkin et al. Towards Foundation Models for Knowledge Graph Reasoning. ICLR 2024.
[2] Teru et al. Inductive relation prediction by subgraph reasoning. ICML 2020.
[3] Huang et al. A Theory of Link Prediction via Relational Weisfeiler-Leman on Knowledge Graphs. NeurIPS 2023.
Questions
**Q1.** Why were 4 inductive (HM) and 10 fully-inductive (ISDEA) datasets omitted from the main results table? Those are all viable datasets and I don’t see a major reason to present the average over 43 datasets instead of 57.
**Q2.** Line 168: the formula for the total number of tokens $\frac{(k+1)(k+2)}{2} - 2$ seems to be incorrect? Setting $k=2$, the formula gives 4 tokens but we only have $(0,1), (1,0), (1,1)$ distances. Similarly, for $k=4$, there are 15 combinations of distances (without (0,0)) but the formula gives 13 options
**C1**. Line 190: citations on conditional message passing GNNs miss [3] that theoretically formalized C-MPNNs.