KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge

Summary

This paper introduces KG-FIT, a general framework that enhances the expressiveness of existing Knowledge Graph Embedding (KGE) models by integrating LLMs. KG-FIT contains four key steps: First, it utilizes an LLM to generate descriptions for a set of given entities, forming an enriched entity representation by concatenating the entity's embedding with its description. Second, it constructs a semantically coherent seed hierarchical structure. Third, it leverages the real-world entity knowledge captured by the LLM to refine this hierarchical structure. Finally, it fine-tunes the knowledge graph embeddings by integrating the hierarchical structure with textual embeddings. Extensive experiments validate the effectiveness of KG-FIT.

Strengths

1. The motivation of the proposed KG-FIT is clear and the paper is well-structured. 2. Extensive experimental results demonstrate that KG-FIT can improve most KGE baseline models. 3. Codes are provided for reproducibility.

Weaknesses

1. The performance of KG-FIT heavily relies on the LLM used for generating entity descriptions and guiding the refinement of the seed hierarchy refinement. If the LLM lacks comprehensive real-world entity knowledge about the given entities or domains, the resulting embeddings may not suboptimal. Furthermore, this dependency could make KG-FIT less effective in the situations where LLM have limited coverage. 2. If the LLM has limited coverage of a specific domain, the LLM guided hierarchy refinement process may yield incorrect results, potentially distorting rather than enhancing the structure of the well-formed seed hierarchy. 3. The paper does not justify the selection of agglomerative hierarchical clustering and the use of silhouette score. Additional ablation studies will enhance this work.

Questions

1. It is unclear the explicit description of $P$, $L$, $R$ in line 134. 2. It remains unclear whether the efficiency evaluation of training time contains the duration required for seed hierarchy construction and LLM-guided hierarchy refinement stages.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have discussed the limitations of their work in the paper.

Summary

This paper addresses the limitations of existing KGE models that focus either on graph structure or fine-tuning pre-trained language models. It introduces KG-FIT, which leverages LLM-guided refinement to incorporate hierarchical and textual knowledge, effectively capturing both global and local semantics. Experiments on benchmark datasets demonstrate KG-FIT's superiority, achieving significant performance improvements over state-of-the-art methods.

Strengths

1. The proposed method can automatically construct a semantically coherent entity hierarchy using agglomerative clustering and LLM-guided refinement, which is an interesting topic. 2. The authors provide detailed illustrations for extensive empirical study on benchmark datasets and demonstrate significant improvements in link prediction accuracy.

Weaknesses

1. The paper is not organized clearly, which is not friendly for understanding. For example, there is a lack the sensitivity study for the hyperparameters in the loss function. 2. The comparable methods are old and lack the new ones in the last 2 years such as [1][2][3][4]. The performance is not comparable with the previous work. [1] Compounding Geometric Operations for Knowledge Graph Completion [2] Geometry Interaction Knowledge Graph Embeddings [3] KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion [4] Dual Quaternion Knowledge Graph Embeddings 3. The paper lacks the analysis of time complexity as well as space complexity, which is necessary to study the efficiency of the model. 4. There are some typos and It is commanded that the writing needs to be improved. (1) On page 5, line 163 “determined by lowest common ancestor” should be “determined by the lowest common ancestor” (2) On page 6, line 195 “is sigmoid function” should be “is the sigmoid function”

Questions

Please refer to the weaknesses.

Rating

4

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

N/A

Summary

Knowledge graphs (KGs) are essential for representing structured knowledge in various domains. They consist of entities and relations, forming a graph structure for efficient reasoning and knowledge discovery. Current knowledge graph embedding (KGE) methods create low-dimensional representations of these entities and relations but often overlook extensive open-world knowledge, limiting their performance. Pre-trained language models (PLMs) and large language models (LLMs) offer a broader understanding but are computationally expensive to fine-tune with KGs. To address these issues, the authors propose KG-FIT, a framework that incorporates rich knowledge from LLMs into KG embeddings without fine-tuning the LLMs. KG-FIT generates entity descriptions from LLMs, constructs a hierarchical structure of entities, and fine-tunes KG embeddings by integrating this hierarchy with textual embeddings. This approach enhances KG representations, combining global knowledge from LLMs with local KG knowledge, significantly improving link prediction accuracy on benchmark datasets.

Strengths

1. Extensive experiments. The authors compare experiment performance with 8 baselines on three datasets and apply on 8 KG embedding backbones. 2. Extensive experimental details description. For example, the hardware environment for running the experiment, data processing, prompts for interacting with large models, and codes to reproduce their results. 3. Clear figures and presentations.

Weaknesses

1. The motivation needs to be reconsidered. The authors mention that using KG to fine-tune LLMs is computationally expensive. Many current research efforts do not fine-tune LLMs with KG. Instead, they use retrieval-based methods to explicitly provide the knowledge. 2. LLM-based baselines should be considered. The authors extensively use LLMs in their methods. They also should incorporate the LLM-based methods. For example, those LLMs retrieval-based methods. Only comparing with small LM-based methods is not enough. 3. The contribution is kind of limited. I do not know why the authors still use KG embedding methods for efficient reasoning and knowledge discovery since LLMs have very strong reasoning ability for knowledge discovery.

Questions

1. Could the authors reconsider the motivation behind their approach? 2. Why haven’t the authors considered incorporating published LLM-based baselines? 3. Why do the authors continue to use KG embedding methods for efficient reasoning and knowledge discovery when LLMs possess very strong reasoning abilities for knowledge discovery?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please refer to my weakness

Summary

The paper introduces a framework called KG-FIT for enhancing knowledge graph embeddings by integrating knowledge from large language models (LLMs). KG-FIT enriches entity descriptions using LLMs and then constructs a semantically coherent hierarchical structure of entities. It finally fine-tunes KG embeddings using this hierarchy and textual information. Experiments on benchmark datasets (FB15K-237, YAGO3-10, PrimeKG) demonstrate its effectiveness in link prediction.

Strengths

- Using LLMs to enhance KG embeddings can capture comprehensive features. The proposed method demonstrates improvements in link prediction when compared to selected baseline methods. - The presentation of the paper is good.

Weaknesses

- There are several related studies on using LLMs to enhance text information in KGs [1,2], which, however, were not discussed in the paper. Additionally, constructing a hierarchy seems redundant given the existing graph structure of the KG. More discussion and explanation are needed. - The rationale behind the technical design is unclear. For instance, the proposed method concatenates structural and textual embeddings to construct the hierarchy, and then linearly combines these embeddings for fine-tuning. What are the underlying reasons for these choices? - The method may be computationally expensive due to the use of LLMs and hierarchical refinement. In my view, it is not profitable to use LLMs to achieve ~0.02 Hits@1 improvements in link prediction. - Several recent strong baselines for KG link prediction, such as NBFNet [3] and AdaProp [4], which both achieve over 0.32 Hits@1 on FB15K-237, are absent from the experiments. It remains uncertain whether the proposed method can still improve the two baselines. [1] Derong Xu, Ziheng Zhang, Zhenxi Lin, Xian Wu, Zhihong Zhu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, Enhong Chen: Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models. LREC/COLING 2024: 11956-11968 [2] Dawei Li, Zhen Tan, Tianlong Chen, Huan Liu: Contextualization Distillation from Large Language Model for Knowledge Graph Completion. EACL (Findings) 2024: 458-477 [3] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal A. C. Xhonneux, Jian Tang: Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS 2021: 29476-29490 [4] Yongqi Zhang, Zhanke Zhou, Quanming Yao, Xiaowen Chu, Bo Han: AdaProp: Learning Adaptive Propagation for Graph Neural Network based Knowledge Graph Reasoning. KDD 2023: 3446-3457

Questions

- The proposed method leverages entity names to prompt large language models (LLMs) to generate corresponding descriptions. How does it address the issue of multiple entities having the same name? - Please see other questions in "Weaknesses".

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

NA.

Thanks for the response

Thanks for the response. I have read all the reviews and responses, and am satisfied with the response to my comments. I will maintain my positive review score.