DSparsE: Dynamic Sparse Embedding for Knowledge Graph Completion

**We appreciate your valuable comments. We have revised our manuscript and clarified the contributions in the revised version. We sincerely hope you will find the time to review our improved manuscript and look forward to your valuable comments.** --- >* **W1: The difference and the superiority of DSparsE compared with baseline models are unclear.** >* **W2: The contribution of this paper is limited. It only combines some existing techniques.** **Response** Thank you for your comment. The key advantages of DSparsE can be summarized in three points: 1. Dynamic Characteristics: The relation-aware layer in ComDensE[1] focuses only on the dynamic nature of network weights as relationships change, neglecting the dynamics associated with node changes. Indeed, designing a separate weight matrix for each node would lead to an explosive increase in parameters with the scale of the graph. Therefore, we adopt a hypernetwork structure to build a network where parameter weights can change with the input entity-relation pairs. This network can consider both entity and relation node information, and works well in conjunction with the relation-aware layer. Additional ablation studies in the revised version of the paper prove that parallel dynamic and relation-aware layers can achieve better performance. | Dynamic+Relation-aware | Dynamic layer only | Relation-aware layer only | |:-------:|:-------:|:-------:| | **0.272**| 0.252 | 0.264 | 2. Sparse Characteristics: Traditional neural networks face issues of overfitting and generalization, which are more severe in knowledge graph link prediction models based on neural networks. This is due to the open-world assumption[2] in knowledge graphs, meaning the graphs are incomplete and the dataset knowledge is extremely limited compared to the actual distribution of knowledge. Thus, our model introduces a fixed sparse structure, different from the dropout feature, which is determined at model initialization. Our inspiration comes from convolutional neural networks like ConvE[3] and InteractE[4], which have sparse connections with weight sharing. We consider random sparse connections between neurons without weight sharing and use a hyperparameter alpha to control the density of feature interactions. Further experiments also show that sparse connections on top of dropout can optimize the network model's predictive performance. The results below show the effectiveness of sparse structure. Note that extra dropout represents setting dropout rate $\hat{p}$ $=$ $p + \alpha(1-p)$, where $\alpha$ denotes the sparsity degree and $p$ denotes the original dropout rate. Here we set $\alpha$ to 0.5. The results indicate that sparse structure can effectivly enhance the performance. | model | Dropout + sparse | Extra dropout | | :---: | :---: | :---: | | DSparsE | **0.272** | 0.266 | | ComDensE | 0.267 | 0.258 | 3. Residual Characteristics: Existing research has shown that deepening network layers can enhance expressiveness[5], but this comes with convergence difficulties, a problem also present in knowledge graph link prediction models. For example, the authors of ComDensE found that increasing the number of layers actually reduced network performance. We also found that simply adding front-end fully connected layers in matrix decomposition or similar models reduces their expressiveness. However, the only way for shallow models to maintain effectiveness with increasing graph scale is to increase the embedding dimension, leading to greater computational costs. If the embedding dimension is kept constant, performance declines with increasing graph scale (especially the number of entities), such as RESCAL's performance on WNRR18 being worse than on FB15k-237 with the same embedding dimension. In DSparsE, we applied expert layers to maximize feature extraction and fusion, requiring a powerful decoding module. In experiments, we found that 3-5 layers of residual neural networks provided good results on Fb15k-237 and YAGO3-10. To some extent, the introduction of residual layers allows the network to deepen further, maximizing its expressive potential. This is beneficial for large knowledge graphs like YAGO3-10. We have provided a table of the optimal number of residual layers for three models, which shows that the depth of the decoding layer needs to increase with the scale of the dataset. The introduction of residual connections helps to stabilize the network training process and maintain the predictive results (the experimental results have been presented in the paper). | Dataset | Entity | Relation | Train | optimal residual layer depth | | :---: | :---: | :---: |:---:|:---:| | FB15k-237 | 14,541 | 237 | 272,115 | 3 | | WNRR18 | 40,943 | 11 | 86,835 | 1 | | YAGO3-10 | 123,182 | 37 | 1079,040 | 5 |

**We would like to sincerely thank you for the time and effort you have dedicated to carefully reading our manuscript and providing insightful comments and suggestions. We have thoroughly considered each point raised and have addressed them in the following sections, aiming to improve and refine our manuscript based on your valuable input.** --- >* **W1: The number of datasets is only 2, relatively limited.** >* **Q1: Can you add more datasets?** **Response** Thank you for your comment! We have added a new dataset, YAGO3-10[1], the details of which are as follows in the table below: | Dataset | Entity | Relation | Train | Valid | Test | | :---: | :---: | :---: |:---:|:---:|:---:| | YAGO3-10 | 123,182 | 37 | 1079,040 | 5000 | 5000 | Accordingly, we have conducted experiments on each metric and compared them with the baseline. Due to the fewer baselines of this dataset, we only selected models with data and presented the results in the table below. | Dataset | Hits@1 | Hits@10 | MRR | | :---: | :---: | :---: |:---:| | DistMult | 0.240 | 0.540 | 0.340 | | ConvE | 0.350 | 0.620 | 0.440 | | ComplEx | 0.260 | 0.550 | 0.360 | | RotatE | 0.402 | 0.670 | 0.495 | | InteractE | 0.462 | 0.687 | 0.541 | | **DSparsE** | **0.464** | **0.690** | **0.544** | Note that the result on this dataset may change in subsequent versions.

（Following the previous text）Regarding the original Figure 6, we apologize for the confusion caused by inaccurate markings. Our intention was to distinguish a sudden change in the horizontal axis values from 5 to 100 using a dashed line and a different color. However, this approach now seems redundant. In our revised paper, we have improved the expression of this figure and the interpretation of the results. Additionally, if you have doubts about whether the network should continue to deepen, you can refer to our response to reviewer jcNV's question 5, which we will include in our revised version. Furthermore, we have added some comparative experiments, such as ablation studies of dynamic layers and relation-aware layers, as well as comparisons of the effects of sparsity structure with dropout and downscaling of the network. Interestingly, by studying the hypernetwork formed by the gated layers' responses to different entity-relation pairs, we discovered that the hypernetwork can effectively model semantic information in the latent space. Along with our analysis, we also provided visualized results. --- >* **W3: In the "Contribution" section, the authors claim that their model achieves a significant reduction in the number of parameters and a significant improvement in model efficiency. However, the subsequent text does not provide a particularly clear substantiation of this claim. It would be beneficial to provide more explicit details, such as in terms of runtime or the exact number of parameters, to support this assertion.** >* **Q3: Can you provide more explicit details of the improvement of the efficiency?** **Response** Thank you for your comment. Indeed, increasing the number of expert blocks results in reduced parameter efficiency, and the fixed structure sparsification does not significantly reduce the number of parameters in the current computing architecture. This is because non-structured pruning does not alter the number of matrix multiplications, and the optimal sparsity of 0.5 mentioned in our paper is not high enough to utilize sparse matrix techniques for optimization. To avoid misleading information, we have removed the statements about parameter reduction from the contributions section of our paper. In the table below, we present the parameter count and corresponding runtime. Several points should be noted regarding these results: | Number of parameters | FB15k-237 | WN18RR | | :---: | :---: | :---: | | InteractE | 18M | 60M | | ComDensE | 66M | 33M | | DSparsE | 69M | 29M | | model | Running time (per batch, $batch size = 128$) on FB15k-237 | on WN18RR | | :---: | :---: |:---:| | InteractE | 2.58s | 3.43s | | ComDensE | 2.61s | 3.60s | | DSparsE | 2.67s | 3.44s | 1. Increasing the number of expert cores does not significantly increase the number of parameters. 2. Relatively speaking, increasing the number of expert cores has a minimal impact on computational speed. 3. For hardware that supports unstructured pruning, introducing sparsity can save storage space and computational resources to a certain extent. --- **We would like to sincerely thank you for your valuable comments, and we hope that you are satisfied with our responses. We appreciate for your reconsideration of our revised manuscript. Thank you very much for your time and effort in reviewing our work!!** --- [1] Suchanek F M, Kasneci G, Weikum G. Yago: a core of semantic knowledge[C]//Proceedings of the 16th international conference on World Wide Web. 2007: 697-706.

**Thank you very much for your insightful comments on our manuscript. We have addressed your comments and made revisions in our new manuscript. Below are detailed responses to your questions:** --- >* **W1: Though the overall model architecture is novel, the key advantage of pure MLP-based model such as DSparsE is unclear. As I understand, it is not efficiency since 15(35) experts are set for FB15k-237(WN18RR) with each expert represented by an MLP layer, which will introduce a lot extra parameters compared to 1 expert. It is also not superior performance, since the link prediction results of DSparsE is comparable to existing methods, such as RESCAL and ComDensE.** >* **Q1: What is the key advantages of designing KGC models with pure MLP layers compared to the existing methods, such as tensor-decomposition models and translational models introduced in the related works?** **Response** Thank you for your comments. The key advantages of DSparsE can be summarized in three points: 1. Dynamic Characteristics: The relation-aware layer in ComDensE[1] only focuses on the dynamic nature of network weights as relationships change, neglecting the dynamics associated with node changes. Indeed, designing a separate weight matrix for each node would lead to an explosive increase in parameters with the scale of the graph. Therefore, we adopt a hypernetwork structure to build a network where parameter weights can change with the input entity-relation pairs. This network can consider both entity and relation node information, and works well in conjunction with the relation-aware layer. Additional ablation studies in the revised version of the paper demonstrate that parallel dynamic and relation-aware layers can achieve better performance. | Dynamic+Relation-aware | Dynamic layer only | Relation-aware layer only | |:-------:|:-------:|:-------:| | **0.272**| 0.252 | 0.264 | Furthermore, we have investigated how the hypernetwork affects the weights of the expert blocks. Here, we will quote a portion of our revised version: >Moreover, our investigation into the gating layer's outputs has unveiled some intriguing insights. Each entity-relation pair in the dataset, upon processing through the gating layer, yields an output vector $\bm{o}$. These high-dimensional vectors were subjected to tSNE reduction, with the resultant visualization displayed in Figure 8 and Figure 9. Each point in this figure represents a unique entity-relation pair, distinguished by varying colors corresponding to different relationships. The visualization result reveals the following observations: >1. A tendency for entity-relation pairs of the same relationship type to cluster together, indicating proximity within the output space of the gated layer outputs. >2. The spatial distribution of clusters is significantly influenced by the nature of the relationships. For instance, relationships denoting inverse meanings (e.g. nominee\_inv and nominee) or semantic opposites (e.g. place of birth vs. place of burial) exhibit a tendency to spatially diverge, exhibiting a unique central symmetry characteristic in the reduced dimensional space, Conversely, relationships with similar semantics (e.g. nationality and city town) are observed to be proximate in the latent space. This proved that DSparsE can capture various associations between entities and relations. >3. Alterations in the head entity of a relation pair result in minor shifts within the vector output, confined to a limited scope. Within a fixed relation, the relative positioning of nodes within its corresponding cluster does not display a discernible pattern. This phenomenon can be attributed to the relatively lower frequency of triples involving individual nodes compared to those associated with a particular relation type, posing challenges in accurately modeling semantic information for nodes. Still, certain examples, such as Mariah Carley and Dmitri Shostakovich—both notable in the music domain—demonstrate proximity within clusters pertaining to specific relations.

>* **Q5: The results of 100 depth of the residual structures. Is this mean that 100 residual MLP blocks are used in the model? If yes, how long does it take to train the model?** **Response** We apologize for any confusion caused by our unclear description. It should be clarified that for a small dataset like FB15k-237, a massive network with 100 layers is extremely redundant. For this dataset, the optimal depth of the decoding layers is around 3 layers. We fully understand your concerns. Your core question is: * **_Is it really necessary to deepen the network?_** In fact, we have observed that the larger the dataset, the deeper the required depth of the decoding layer. Blow is the relationship between the scale of the dataset and the corresponding optimal number of layers: | Dataset | Entity | Relation | Train | optimal residual layer depth | | :---: | :---: | :---: |:---:|:---:| | FB15k-237 | 14,541 | 237 | 272,115 | 3 | | WN18RR | 40,943 | 11 | 86,835 | 1 | | YAGO3-10 | 123,182 | 37 | 1079,040 | 5 | Based on this fact, we make a reasonable conjecture that for super-large-scale knowledge graphs (like FreeBase, DBPedia, etc.) with tens of billions of knowledge triples, the required number of decoding layers will increase further. Under this assumption, our experiments prove that incorporating residual connections in MLPs can effectively maintain the network's decoding performance and maximize the network's expressive power. You also mentioned the issue of training time for deep networks. We have conducted corresponding experiments, and the results are shown in the following table. Interestingly, stacking a large number of linear layers does not significantly increase the network's computational time. This may be because computational resources are mainly consumed during preprocessing and information transmission stages. To ensure rigor, we have specified the models of the GPUs and CPUs used for reference. | CPU | GPU | Running time for $depth = 3$ | running time for $depth = 100$ | | :---: | :---: |:---:|:---:| | AMD Ryzen 9 7945HX | NVIDIA GeForce RTX 4080 Laptop GPU | 10.2s per batch(512) | 13.7s per batch(512) | --- **Again, we respectfully request your review of the latest revision. We are eager to hear your comments on our revised manuscript to further improve our work.** **Thank you once again for your valuable comments on our research. It is truly rewarding that you find our work to be _very interesting_! Wish you have a nice day! :D** --- [1] Kim M, Baek S. Comdense: combined dense embedding of relation-aware and common features for knowledge graph completion[C]//2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2022: 1989-1995. [2] Drummond N, Shearer R. The open world assumption[C]//eSI Workshop: The Closed World of Databases meets the Open World of the Semantic Web. 2006, 15: 1. [3] Dettmers T, Minervini P, Stenetorp P, et al. Convolutional 2d knowledge graph embeddings[C]//Proceedings of the AAAI conference on artificial intelligence. 2018, 32(1). [4] Vashishth S, Sanyal S, Nitin V, et al. Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(03): 3009-3016. [5] Chatterjee S, Zielinski P. On the generalization mystery in deep learning[J]. arXiv preprint arXiv:2203.10036, 2022. [6] Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data[J]. Advances in neural information processing systems, 2013, 26. [7] Wang Z, Zhang J, Feng J, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the AAAI conference on artificial intelligence. 2014, 28(1). [8] Lin Y, Liu Z, Sun M, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the AAAI conference on artificial intelligence. 2015, 29(1). [9] Ji G, He S, Xu L, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers). 2015: 687-696. [10] Jacobs R A, Jordan M I, Nowlan S J, et al. ªAdaptive Mixtures of Local Experts, º Neural Computation, vol. 3[J]. 1991. [11] Vashishth S, Sanyal S, Nitin V, et al. Composition-based multi-relational graph convolutional networks[J]. arXiv preprint arXiv:1911.03082, 2019. [12] Schlichtkrull M, Kipf T N, Bloem P, et al. Modeling relational data with graph convolutional networks[C]//The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15. Springer International Publishing, 2018: 593-607.

**We would like to sincerely thank you for the time and effort you have dedicated to carefully reading our manuscript and providing insightful comments and suggestions. Here is our response.** --- >* **W1: Better explanations can be included. For instance, the paper states the expert kernels, but does not explicitly explain what is the expert kernels or why it is "expert".** >* **Q1：What is an expert kernel?** >* **Q2: How does the model do in the case of no experts? Or simple mean of the outputs of k different mlps.** >* **Q4: Why does the dynamic structure enhance the robustness of the model? Is it by some form of ensembling?** **Response** Thank you for your comment. As per your suggestion regarding the term “expert kernel,” we have revised it to “expert block” for clarity and added explanatory sentences. We have also included a figure specifically for the dynamic layer section for better understanding. The revised statements concerning the experts are as follows: >The encoding part consists of an MLP layer with $k$ expert blocks (i.e., structurally similar MLP sub-blocks) and a relation-aware MLP layer. For an explanation of the term _expert_, you can refer to the paper _Adaptive mixtures of local experts_[1] as well as _Outrageously large neural networks: The sparsely-gated mixture-of-experts layer_[2]. The explanation provided therein is as follows: >_We present a new supervised learning procedure for systems composed of many separate networks, each of which learns to handle a subset of the complete set of training cases. The new procedure can be viewed either as a modular version of a multilayer supervised network, or as an associative version of competitive learning[1]._ > _The experts are themselves neural networks, each with their own parameters. Although in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters[2]._ From the explanation, it becomes evident that what is referred to as an _expert_ is essentially a sub-neural network with similar (or the same) structure. Training a large number of such neural networks in parallel can enhance the model's robustness. This aspect has also been elaborated in the experimental section of the revised paper (which includes comparisons with a pure MLP layer without dynamic adjustments). Moreover, reviewer S6XC mentioned in W2 that the original Figure 5 lacked sufficient experimental data on the number of expert blocks. We have now increased the number of data points, which reveals a rather distinct trend. The revised elucidation for this part of the experiment is as follows: >The experimental results indicate that the predictive performance first increases and then decreases with the rising number of expert blocks. Initially, the addition of expert blocks can effectively enhance feature fusion capabilities. This can be explained from two perspectives. Firstly, in contrast to a non-partitioned fully connected structure (i.e., a very wide fully connected layer), the expert blocks in the dynamic layer represent a form of regular sparse connections (See Appendix D for details). These sparse connections are further integrated through a decision layer, namely a gating layer, forming a hyper-network structure, which brings robustness to the entire network. Secondly, the expert blocks in the dynamic layer can be viewed as sub-modules in an ensemble learning framework. This ensemble learning architecture can effectively suppress the propagation of errors, reducing the variance in prediction results. Under the hypernetwork's constraints, the multi-modular architecture can evolve towards the optimal direction (See Appendix E for details). > >However, when the number of expert blocks becomes excessive, the performance deteriorates, which can be attributed to two factors. First, an increase in network parameters introduces additional training complexity, diminishing the network's generalization performance. Second, the gating network is fundamentally a multi-classifier. An excessive number of categories increases the decision-making complexity of the network, making it more prone to difficulties. > >Another important influencing factor is the temperature of the dynamic layer. High temperature values make the weight combinations tend towards an average, leading to weight homogenization. Conversely, low temperature values can render many experts ineffective in learning, thus impacting the results.

(Following the previous text) In addition, we further investigate how the hypernetwork affects the weights of the expert blocks. In this regard, we have made some interesting observations. Here we will quote a part of the modified version: >Moreover, our investigation into the gating layer's outputs has unveiled some intriguing insights. Each entity-relation pair in the dataset, upon processing through the gating layer, yields an output vector $\bm{o}$. These high-dimensional vectors were subjected to tSNE reduction, with the resultant visualization displayed in Figure 8 and Figure 9. Each point in this figure represents a unique entity-relation pair, distinguished by varying colors corresponding to different relationships. The visualization result reveals the following observations: >1. A tendency for entity-relation pairs of the same relationship type to cluster together, indicating proximity within the output space of the gated layer outputs. >2. The spatial distribution of clusters is significantly influenced by the nature of the relationships. For instance, relationships denoting inverse meanings (e.g. nominee\_inv and nominee) or semantic opposites (e.g. place of birth vs. place of burial) exhibit a tendency to spatially diverge, exhibiting a unique central symmetry characteristic in the reduced dimensional space, Conversely, relationships with similar semantics (e.g. nationality and city town) are observed to be proximate in the latent space. This proved that DSparsE can capture various associations between entities and relations. >3. Alterations in the head entity of a relation pair result in minor shifts within the vector output, confined to a limited scope. Within a fixed relation, the relative positioning of nodes within its corresponding cluster does not display a discernible pattern. This phenomenon can be attributed to the relatively lower frequency of triples involving individual nodes compared to those associated with a particular relation type, posing challenges in accurately modeling semantic information for nodes. Still, certain examples, such as Mariah Carley and Dmitri Shostakovich—both notable in the music domain—demonstrate proximity within clusters pertaining to specific relations. For the images mentioned in the text, please refer to our revised version of the paper. --- >* **W2: Better figures and figure captions can be written. The architecture figures are small and the explanations in the captions is very limited.** **Response** Thank you for your comment. we have added detailed captions to all images and provided thorough descriptions within the text in the revised version. Regarding the figures, we have made the revisions as follows: 1. We enlarged the overall architecture diagram and bolded the key text within it. If you still see any issues, please do not hesitate to point them out, and we will continue to modify it to meet your requirements. 2. We updated the architecture diagram of the residual layer (changed from horizontal to vertical orientation) and added a dynamic layer architecture diagram to aid understanding. 3. We moved the original image for intuitive understanding of the sparse layer to the appendix due to space constraints. In addition, we have completely revamped the structure of the paper and optimized the language expression in key sections. Specifically, some of the main changes are as follows: 1. We have polished and updated the language of the article, especially the explanations in the abstract and the contributions section. 2. We have modified and added several necessary charts to aid understanding and enriched the captions for these charts. 3. We have added the YAGO3-10 dataset and provided corresponding baseline data, demonstrating that our model outperforms all similar models. 4.We have added multiple comparative experiments and ablation studies, elucidating the mechanisms of each part. 5. We have included additional data on the number of parameters and runtime. 6. Our study of the latent space output has yielded interesting findings, which were mentioned in response to the first question. However, we hope that you will review them in conjunction with the revised version's images. 7. Due to space constraints, we have placed some additional content in an appendix to aid understanding. Except for the parts related to the number of parameters, all other sections are referenced in the main text.

--- >* **W3: Some statistical testings (or critical plots) for the comparing methods would make the arguments of the paper stronger.** >* **Q3: How is imposing sparse structure different from the drop-connect (dropout on the weights)** >* **Q5: How statistically different are the results between the models (or at least the top competing ones).** **Response** Thank you for your constructive suggestions. In the revised version of the paper, we have added a dataset, YAGO3-10. Now we present the experimental results of DSparsE on the YAGO3-10 dataset across various metrics: | Dataset | Entity | Relation | Train | Valid | Test | | :---: | :---: | :---: |:---:|:---:|:---:| | YAGO3-10 | 123,182 | 37 | 1079,040 | 5000 | 5000 | | Dataset | Hits@1 | Hits@10 | MRR | | :---: | :---: | :---: |:---:| | DistMult | 0.240 | 0.540 | 0.340 | | ConvE | 0.350 | 0.620 | 0.440 | | ComplEx | 0.260 | 0.550 | 0.360 | | RotatE | 0.402 | 0.670 | 0.495 | | InteractE | 0.462 | 0.687 | 0.541 | | **DSparsE** | **0.464** | **0.690** | **0.544** | We also give the detailed results as well as the comparison on FB15k-237: | | | MRR | Hits@10 | Hits@1 | MRR | Hits@10 | Hits@1 | MRR | Hits@10 | Hits@1 | |--------------------|------|-------|---------|--------|-------|---------|--------|-------|---------|--------| | | InteractE | | | ComDensE | | | | DSparsE | | | | **Pred Head** | 1:1 | 0.386 | 0.547 | 0.245 | 0.422 | 0.557 | 0.349 | **0.434** | **0.572** | **0.358**| | | 1:N | **0.106** | **0.192** | 0.043 | 0.084 | 0.181 | 0.043 | 0.101 | 0.185 | **0.044**| | | N:1 | 0.466 | 0.647 | 0.369 | 0.466 | 0.649 | 0.372 | **0.467** | **0.655** | **0.376**| | | N:N | 0.276 | 0.476 | 0.164 | 0.279 | 0.476 | 0.187 | **0.287** | **0.494** | **0.195**| | **Pred Tail** | 1:1 | 0.368 | 0.547 | 0.229 | 0.422 | 0.563 | 0.349 | **0.428** | **0.570** | **0.351**| | | 1:N | 0.777 | 0.708 | **0.881** | **0.779** | 0.884 | 0.717 | 0.778 | **0.886** | 0.796| | | N:1 | 0.074 | 0.141 | 0.034 | 0.084 | 0.169 | **0.043** | **0.088** | **0.171** | 0.042| | | N:N | 0.395 | 0.617 | 0.272 | **0.396** | 0.618 | 0.285 | 0.395 | **0.624** | **0.286**| For the original Figure 4, we added a comparison with InteractE, demonstrating the monotonic decline of InteractE with increasing sparsity. We have explained this in the revised version of the paper as follows: >On the other hand, the performance of the InteractE model demonstrates a consistent decrease with increasing levels of sparsity. This trend is attributed to the model architecture of InteractE, where only the final feature decoding layer is an MLP layer. Experimental results indicate that introducing increased sparsity over the sparse interactions already captured by the earlier convolutional layers adversely impacts the model's predictive performance. While investigating the impact of sparsity, we also conducted partial tests of DSparsE on the WN18RR dataset. The experiments are still ongoing, but we provide some of the data as follows, which is consistent with the analytical conclusions in the original text: | Sparsity | Hits@1 | | :---: | :---: | | 0. | 0.438 | | 0.3 | 0.441 | | 0.5 | 0.443 | | 0.8 | 0.421 | Additional ablation studies in the revised version of the paper prove that parallel dynamic and relation-aware layers can achieve better performance. | Dynamic+Relation-aware | Dynamic layer only | Relation-aware layer only | |:-------:|:-------:|:-------:| | **0.272**| 0.252 | 0.264 | The Dropout method applies different masks with each training iteration, which adds to the instability of the training. In our method, the pattern of sparse connections is already determined at initialization. Essentially, our method cuts the connections between neurons, weakening the intensity of feature interactions. through empirical evidence, it was demonstrated that: 1. Our method has advantages over simply reducing the network size (i.e., the number of output neurons). 2. Our method has advantages over the use of Dropout alone.