Deep Sparse Latent Feature Models for Knowledge Graph Completion

Clarifying Model Motivation, Comparisons with GNN and TAG Models, and Baseline Selection for Text-Based KGC

Thank you for the reviewer’s thoughtful feedback and valuable suggestions regarding motivation, clarity, and baseline selection. Below, we address each of the reviewer’s comments individually, including references to the specific revisions made in the manuscript. ## Response to Weakness 1 Thank you for your insightful feedback regarding the motivation and positioning of our proposed method relative to GNN-based and TAG models. We would like to provide a detailed comparison between our method and each of the mentioned type of models. 1. Regarding the difference between GNN-based methods and our approach in integrating complex connectivity information among nodes, our approach consider **not only the features of each triple itself** when determining relationships between entities, **but also the underlying clustering information**. As illustrated in Figure 1, if there is a strong connectivity between two communities, it is likely that nodes in these communities may also share certain relationships. - GNN-based methods typically assume that **nodes with similar neighborhoods will have similar representations for learning node embeddings**. This **contrasts sharply** with our approach, which addresses KGC through **relationships between clusters**. In addition, SEA-KGC integrates pretrained language models with KGE methods to capture semantic structure **without relying on GNNs**. Studies [1][2] further indicate that **Message Passing GNN-based KGC methods struggle to effectively leverage neighborhood information**. - To clarify these points, we have **revised the manuscript**, specifically updating the **motivation** in the Introduction (Line 41, Page 1), expanding our **listed contributions** (Line 84, Page 2), and adding a dedicated section in **Related Work** comparing KGC methods that **leverage neighborhood information** (Line 479, Page 9). 2. GLEM and GraphFormers both utilize GNNs and LMs to integrate graph structure information with node text information. Below, we discuss these two models separately: - **GLEM** effectively combines global information extracted by GNNs with local information by LMs through an EM training approach, **targeting node classification tasks**. Its **stepwise extraction and integration of multimodal features via the EM algorithm** provide a valuable strategy. - **GraphFormers** are essentially a type of representation encoder for textual graphs that combines GNNs and transformers, which could directly replace the Bert encoder we currently use to generate more expressive representations, with an anticipated improvement in KGC performance. However, **for fair comparisons, most text-based models currently use bert-base-uncased during evaluation**. In general, there are several challenges when applying TAG models to text-based KGC: - Effectively incorporating additional edge types (relations) into the model. - As noted above, the GNN module needs careful design to fully utilize neighborhood features. - Text-based KGC often relies on large batch sizes for robust contrastive training, where traditional MLE training typically falls short. This necessitates lightweight implementations of models like GraphFormers to accommodate these requirements. ## Response to Weakness 2 Certainly, recent approaches have achieved notable success in KGC tasks by leveraging LLMs. These methods either directly utilize the **strong reasoning capabilities of LLMs to extract triple features** (CSProm-KG) or employ them to **generate additional information that enriches existing knowledge** (KICGPT). In contrast, our work focuses on **enhancing representation learning** of existing text-based KGC methods to achieve **better performance and improved interpretability**. We believe that directly comparing two types of models towards fundamentally different optimization direction, plus vastly different parameter scales, may lead to unfair evaluations. While we have not included some LLM-based baselines for comparison, we have also incorporated several recent state-of-the-art methods from both embedding-based (N-Former, KRACL) and text-based (KG-S2S, GHN) approaches to ensure a comprehensive evaluation in Table 1, Page 7. [1] Are Message Passing Neural Networks Really Helpful for Knowledge Graph Completion? [2] Rethinking graph convolutional networks in knowledge graph completion.

Response to Weakness 1 & 2

We are grateful to the reviewer for their insightful feedback and helpful suggestions concerning motivation, model complexity, and effectiveness. In the following, we respond to each comment individually and highlight the corresponding changes made to the revised manuscript. ## Weakness 1 > Model motivation: SBM is well-known graph clustering algorithm. There are many works that combine GNN with SBM to achieve community detection. Except from the used data structure, the key difference between the proposed method and existing works is not obvious. ### Response We understand the reviewer’s concerns regarding the **motivation and contributions** of our work. In the following content, we will make clarification on these two points, the corresponding revisions of which in the manuscript will also be mentioned. **Motivation** 1. **Connections among node communities**—a key inductive feature of graph data—have not yet been leveraged for KGC. While a few existing methods incorporate node clustering information, they typically rely on **separate clustering modules** like KMeans, resulting in **suboptimal KGC performance, limited cluster interactions, and a lack of end-to-end design**. To clarify this motivation, we have bolded the relevant content in the Introduction (Line 40, Page 1) and added a dedicated section in Related Work on KGC methods that use neighborhood information (Line 478, Page 9). 2. Although SBM and VAE have proven effective in statistical link prediction (LP) for detecting node communities, there are two significant differences between traditional LP and KGC: (1) LP methods predict the presence of a link between two nodes, while **KGC requires multi-label edge prediction** and evaluates models based on **tail entity prediction** given a head-relation pair; and (2) modern **KGs are far larger, containing millions of nodes**, unlike typical LP datasets like Cora with only thousands. As such, **directly applying SBM and VAE to KGC is challenging** (Line 82, Page 2). **Contributions** 1. We propose **a novel probabilistic generative model for KGC** based on SBM, which enables the integration of complex and meaningful latent structures within KGs in future research. 2. Our implementation **combines techniques from VAE and text-based KGC methods**, ensuring both **high performance and scalability**. Additionally, the inclusion of text information introduces a fresh approach to clustering within KGs. 3. We conduct extensive experiments demonstrating our method’s **robust performance, interpretability, and highlighting certain limitations** of leveraging clustering information. We have revised the **end of the Introduction (Line 84, Page 2)** to provide a more direct and clear listing of our contributions. ## Weakness 2 > Model Complexity: The introduction of various latent feature sampling and inference mechanisms makes the inference process relatively complex, necessitating more efficient training strategies to speed up training and reduce memory usage. ### Response We appreciate the reviewer’s emphasis on the importance of time complexity in model evaluation. In the revised manuscript, we have made the following enhancements to address this concern: 1. We provide a detailed **analysis of the time and space complexity** involved in computing the ELBO (refer to Lines 281, Page 6 and Lines 796, Page 15). 2. We have also added a **new table (Table 7, Line 930, Page 18) comparing the parameter count and training time** of our method against the baseline model, SimKGC. The results indicate that introducing SBM and VAE **only increases the parameter count by 1.4M (0.8%)**. While the ELBO training requires **additional epochs to converge**, resulting in a slight increase in total training time, we find this increase to be **within an managable range**.

Response Addressing Model Complexity, Contributions, and Baseline Issues

We would like to thank the reviewer for the thoughtful feedback and helpful suggestions on representation, clarity and experiment settings. Below, we provide a point-by-point response to the reviewer’ comments, with references to the changes made in the manuscript. ## Weakness 1 > The paper repeatedly emphasizes the advantages of the proposed model on large-scale graphs; therefore, it would be beneficial to include comparative experiments on time complexity or runtime performance between the proposed model and the baseline. ### Response We appreciate the reviewer’s emphasis on the importance of time complexity in model evaluation. In the revised manuscript, we have made the following enhancements to address this concern: 1. We provide a detailed **analysis of the time and space complexity** involved in computing the ELBO (refer to Lines 281, Page 6 and Lines 796, Page 15). 2. We have also added a **new table (Table 7, Line 930, Page 18) comparing the parameter count and training time** of our method against the baseline model, SimKGC. The results indicate that introducing SBM and VAE **only increases the parameter count by 1.4M (0.8%)**. While the ELBO training requires **additional epochs to converge**, resulting in a slight increase in total training time, we find this increase to be **within an managable range**. ## Weakness 2 > At the end of the introduction section, a more direct listing of the contributions of this paper should be provided, with particular emphasis on the novel points introduced for the first time in this work. ### Response Thank you for your valuable feedback. We have revised the end of the Introduction (Line 84, Page 2) to provide a more direct and clear listing of our contributions. ## Weakness 3 > The paper uses more baselines on WN18RR and FB15k-237 compared to Wikidata5M. Is this due to differences in dataset scale? The authors should provide a detailed explanation. ### Response We understand the reviewer’s concerns regarding the fairness and consistency of using different numbers of baselines across datasets. **The Wikidata5M dataset is primarily used by text-based methods**, and KGC performance results on this dataset are **rarely reported in embedding-based literature**. Therefore, to ensure meaningful comparisons, we selected only text-based baselines on the Wikidata5M dataset to evaluate our method’s performance. ## Weakness 4 > The primary parts in this framework, SBM and VAE, are derived from previous work, with extensive references to existing literature in the framework description. I am not entirely clear on the main innovations introduced by the authors in these two methods. It would be helpful if the authors could enhance the explanation of their contributions in the text or directly clarify these innovations in their response to me. ### Response We understand the reviewer’s concerns regarding the **motivation and contributions** of our work. In the following content, we will make clarification on these two points, the corresponding revisions of which in the manuscript will also be mentioned. **Motivation** 1. **Connections among node communities**—a key inductive feature of graph data—have not yet been leveraged for KGC. While a few existing methods incorporate node clustering information, they typically rely on **separate clustering modules** like KMeans, resulting in **suboptimal KGC performance, limited cluster interactions, and a lack of end-to-end design**. To clarify this motivation, we have bolded the relevant content in the Introduction (Line 40, Page 1) and added a dedicated section in Related Work on KGC methods that use neighborhood information (Line 478, Page 9). 2. Although SBM and VAE have proven effective in statistical link prediction (LP) for detecting node communities, there are two significant differences between traditional LP and KGC: (1) LP methods predict the presence of a link between two nodes, while **KGC requires multi-label edge prediction** and evaluates models based on **tail entity prediction** given a head-relation pair; and (2) modern **KGs are far larger, containing millions of nodes**, unlike typical LP datasets like Cora with only thousands. As such, **directly applying SBM and VAE to KGC is challenging** (Line 82, Page 2). **Contributions** 1. We propose **a novel probabilistic generative model for KGC** based on SBM, which enables the integration of complex and meaningful latent structures within KGs in future research. 2. Our implementation **combines techniques from VAE and text-based KGC methods**, ensuring both **high performance and scalability**. Additionally, the inclusion of text information introduces a fresh approach to clustering within KGs. 3. We conduct extensive experiments demonstrating our method’s **robust performance, interpretability, and highlighting certain challenges** of leveraging clustering information.