MKGL: Mastery of a Three-Word Language

Summary

This paper proposes a method to leverage LLMs as knowledge graph completion systems. New tokens that correspond to (potentially multi-word) concepts and relations are introduced to the model’s vocabulary, and then the LLM’s embeddings for the tokens composing those concepts/relations are aggregated and upscaled to estimate embeddings for these new tokens. Given these new tokens, the goal of the system is to complete the knowledge graph triplet given two of these embeddings by retrieving the correct third token from the KG vocabulary. In a series of experiments, it is observed that the proposed system outperforms a variety of previous baselines employing diverse methods. It also excels at inductive KG completion. An ablation study reveals that each proposed part of the pipeline is necessary for achieving the best performance.

Strengths

* The proposed approach outperforms a wide variety of prior methods with respect to accuracy. It also achieves a better trade-off between accuracy and compute-efficiency. * A wide variety of baselines employing diverse methods are compared. * Informative ablation study.

Weaknesses

1. Given how many moving parts there are, reproducibility seems difficult. It would be nice to see the variance in performance of the method across multiple random seeds, where each random seed entails running the entire pipeline of optimizations from scratch. 2. Relatedly, it is unclear whether the approach will scale as LMs continue to improve (and presumably to become better bases for approaches like this). Having a comparison with other base models would be a nice way to hedge against this. 3. No detailed discussion of limitations. The checklist says it is discussed, but there are only brief comments distributed throughout the paper (which, in my opinion, do not address limitations thoroughly enough). A dedicated section would be helpful. 4. It is unclear whether the new KG embeddings encode relevant concepts to the target token, or whether they are picking up on certain spurious correlations that happen to be helpful (but may not generalize robustly). It would be nice to have an analysis where the new embeddings are directly decoded into vocabulary space, such that we can observe what concepts are included in these new representations.

Questions

1. Would it be possible to base this approach on other LMs as well? The various Llama scales would be ideal, but if scale is an issue, even just comparing Mistral, Llama 2 (7B), Llama 3 (8B), and ideally some smaller models would be nice. This is quite compute-intensive, so I wouldn’t expect it for the camera-ready, but it would definitely be nice to have. 2. Do you have any hypotheses as to why the proposed method is better at inductive KG completions than prior methods? In other words, is there a particular aspect of your pipeline that you believe makes it better for handling novel triplets than past approaches? Typos: * L226: “clear that The” -> “clear that the”

Rating

7

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

I do not believe limitations have been sufficiently addressed. There is no "Limitations" section, nor is there a dedicated space in any part of the paper that directly addresses the drawbacks of the proposed method and experiments. For example, there are many moving parts; there are multiple stages of optimization that could lead to cascading errors; only one LLM base was considered; etc.

Summary

This paper proposes what seems to be an elaborate GNN+LLM+GNN sandwich of a model for doing knowledge base completion, having a GNN pipeline to form KB-informed token embeddings, passing those to a LLM (lllama-2 ) into a knowledge base completion prompt template, and then passing that output into another GNN-like ("PNA") set of layers. That whole set-up is then used to train representations of the data optimized with contrastive loss for the entity prediction task in knowledge base completion tasks (FBK and Wordnet) (I'll admit that I found the model explanation quite hard to follow, and so it's entirely possible I'm slipping on some details -- some of this had to be inferred by glancing at their code.) It seems to outperform existing methods on these two tasks.

Strengths

If the authors work and evaluation are sound, their method outperforms other methods at two commonly used knowledge base completion tasks.

Weaknesses

- I found the model extremely rather hard to follow, and had particular trouble discerning why this collection of model assumptions would result in a meaningful improvement over prior work. Since the work is so complicated, it may be both useful to focus on very clear graphs and progressively introducing parts of the model. I'll admit that the "Retriever" framing felt very confusing as well, as this models "Retrievers" don't seem to do any retrieving. - For understanding the model, the "three world language" framing seems quite separate from the meat of what this model seems to be doing. It implies that the somewhat simple "template" setup they use is important, but it seems to be only formed to teach a model what triplet completion is, which is something that one would think would be easily addressed in fine-tuning; there is not experimental exploration showing the value of those prompts. - I'll admit to feeling suspicious at the high performance here (even after ablating nearly everything, they outperform most models?).

Questions

-Looking at the code, it was unclear whether the model is calculating the metrics (MRR, Hits@1, Hits@10) by ranking solely within a small batch containing the correct answer, or actually predicting the highest ranked entity from a full space of candidates. Could the authors clarify which one is being done? - Could the authors clarify what is removed in the second , "text" ablation reow

Rating

6

Confidence

2

Soundness

2

Presentation

2

Contribution

3

Limitations

yes, a broader impacts section is included and no major limitations are missing.

Summary

The authors introduce a SOTA method for allowing LLMs to incorporate information from knowledge graphs, relying on Knowledge Graph Language token embeddings to retrieve context, and then score it using a retriever that helps form a distribution over the possible entities to be incorporated.

Strengths

Creative, interesting paper. Introduces a novel approach in a domain that could be of great use to data analysts (LLMs for knowledge graphs). Well-supported empirically with strong evaluation on major benchmarks. Outperforms alternative methods mostly across the board. Simple and effective illustrations. Good use of formatting in the paper itself to facilitate understanding (appreciated the use of color, especially). Method may have notable benefits for reducing problems like hallucination and distribution drift, contributing to a solution for major outstanding issues with LLMs.

Weaknesses

Could use a longer and more detailed discussion section; ends a little too abruptly.

Questions

None.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

I have no concerns about this paper being published.

Summary

The paper proposes MKGL, a novel approach to integrate LLMs with KGs by instructing them in a specialized KG Language (KGL). KGL is a three-word language that mirrors the structure of KG triplets. The authors introduce a KGL context retriever and a score retriever, both based on LoRA, to efficiently encode textual and relational information into KGL token embeddings. MKGL outperforms existing KG completion methods, including LLM-based and conventional approaches, on both standard and inductive KG completion tasks. The paper also demonstrates MKGL's ability to generate valid KGL sentences and its computational efficiency compared to in-context learning methods.

Strengths

* The authors present a novel approach to LLM-KG integration using a completion of entity-relation-entity triplets. * The performance seems to be strong and the method outperforms the previous work on KG completion. * At least compared to naive in-context learning, MKGL is more efficient and also achieves better scores.

Weaknesses

* There are only limited details on certain aspects of the methodology, for example I couldn't find details about the actual implementation of the multi-layered PNA for KG information retrieval. * While the authors claim that the proposed "three-word language" parsing of natural sentences is novel, it boils down to semantic-role labeling (SRL), a well-established NLP task. I believe that the paper should include a clear comparison to past SRL methods. * The results in Table 2 should contain a column with computational cost (or at least the number of parameters of each method), to make it clear if it compares apples to apples. * The computation runtime of the proposed method and the baselines is another thing that is lacking.

Questions

* What exactly are the trainable parameters of the in-context-learning baseline in Section 4.6 and Figure 4? Isn't the point of ICL to not do any parameter updates at all and rely only on the contextual prompt? * How does MKGL scale to much larger knowledge graphs? And how is it compared to other KG completion methods?

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

No issues found.