How to Turn Your Knowledge Graph Embeddings into Generative Models

Summary

This paper provides an new perspective for looking at the popular knowledge graph embedding (KGE) method for knowledge graph completion. Specifically, KGEs models are re-interpreted as circuits. The paper first prove the circuits-based equivalence of score functions of several KGE models, and, in order to regulate KGE further as probabilisitic circuits (GeKC), two restrictions (i.e., non-negative, squaring) are introduced to modify the score functions. Additionally, two techniques are proposed to improve GeKCs: firstly, utilizing trained KGEs as initialization for efficient training of GeKCs, and secondly, introducing constraints to modify the support of GeKCs. The empirical experiments demonstrate that GeKCs exhibit competitive performance compared to PLL objective applied in KGE. Moreover, the proposed constraint method is justified to avoid constraint violations during inference. Finally, the paper compares GeKCs' ability to the uniform and NNMFAug baseline, showing GeKCs' strong capability to generate new likely triples.

Strengths

Generally the paper presents a novel perspective for the knowledge graph embedding model. This perspective enables researchers to learn KGE models through MLE and therefore build a generative model based on those models. Additionally, logical constraints can be incorporated to this framework to ensure that projected triples are valid during inference.

Weaknesses

The perspective of probabilistic circuits seems to be limited. As in the paper, the score functions of KGE models interpreted as circuits are mainly basd on tensor factorization.

Questions

1. Could other KGE models such as TransE, RotatE be viewed as circuits? 2. In line251, how is |c_k| be calculated? Does that mean constrained GeKCs generally have higher computational cost compared to the one without constraints? 3. Why does training time for squared-COMPLEX in ogbl-biokg decrease compared to COMPLEX/COMPLEX+? Theorem 1 suggests that squared-GeKCs have a higher level of complexity than that indicated in proposition 2.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I do not find out any potential negative societal impact.

Summary

The authors show that standard KGE models, such as COMPLEX, can be represented as structured computational graphs called circuits. They propose a new interpretation of these graphs and their parameters to create efficient and expressive probabilistic models over triples in a KG, called generative KGE circuits (GeKCs). These can be efficiently trained by exact MLE and scale well on large KGs with millions of entities. GeKCs can also sample new triples exactly and efficiently, and predictions at test time never violate logical constraints such as domain schema definitions. Experiments show that these advantages come with little or no loss in link prediction accuracy.

Strengths

- This paper is well-written, with clear language and a well-structured presentation. I enjoy reading it. - The idea of making KGEs generative is brilliant! There have been so many new KGEs in the past years, we constantly rely on real world datasets. Now with the proposed method, we can actually generate KG datasets conveniently. - The re-interpretation of existing training objectives as MLE is very clever. - Additionally, the proposed method makes it easier to add constraints to KGEs. - The experiments are clearly designed and well-executed, further demonstrating the value of this work. Overall, this paper is a valuable contribution to the field and deserves a wide audience for its insights and rigorous analysis.

Weaknesses

Maybe not a weakness but some extension? - The constraint adding part can be extended into another work. I can imagine, in high-stake decision-making applications like finance etc, adding constraints can play a vital role.

Questions

Q1: It is not clear from line 89 why (2) is considered a proxy for (1). Can you provide more details on the reasoning behind this? Q2: In Table 2, it is not stated why the results from [12] were not included as a baseline. Is there a specific reason for this?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitations are discussed in sec 7.1 and sec 8.

Summary

This paper converts score functions of knowledge graph embeddings into generative KGE circuits (GeKCs) by restricting activations to be non-negative or squaring their outputs. The changes come with little or no loss of link prediction performance, while enabling probabilistic interpretations, exact maximum-likelihood estimation, better scaling when trained with discriminative objectives, efficient sampling of new triples, and logical constraints such as domain schema definitions. The paper additionally proposes a metric for measuring the quality of sampled subject-predicate-object triples.

Strengths

- Novel combination of probabilistic circuits and knowledge graph embeddings. - Clear presentation. Solid theory.

Weaknesses

1. Multiple questions regarding the motivation (i.e. why the introduced properties are useful) need to be clarified. See Questions. 2. Empirical results are not motivating enough. (a) Sec. 7.1 doesn't show evidence that GeKCs are always preferred for link prediction in terms of accuracy or provide actionable knowledge about on what datasets GeKCs may perform better. (b) In Sec. 7.2 Integrating Domain Constraints, d-ComplEx2 results are 100 by design, but the baseline ComplEx also achieves very high numbers (99+) while ComplEx2 has worse-than-baseline results. Can you provide d-ComplEx2's general link prediction performance as compared to methods in Sec. 7.1's experiments, so that people can decide whether it is worth using in practice? Do you have results on more than one dataset and more than one model?

Questions

1. To confirm understanding: the model difference between GeKCs and other KGEs is constraining scores to be non-negative or squaring them, and nothing else, right? 2. About scaling. (a) Why does the GeKCs scale better, given the additional squaring operations or non-negative restrictions? (b) Is MLE intractable for the models in the paper that can be but are not yet converted to GeKCs? Can you elaborate on why? 3. About loss. When is PLL training recommended and when is MLE training recommended? Why? 4. About probabilities. (a) L29 says the scores cannot be compared across different queries, but L27 gives an example of scores being compared. Why? At least within the same model, negative energies can be compared, right? (b) L29: What are the cases where we want to compare scores across KGE models? (c) What's the benefit of having a probabilistic model instead of an energy based model? 5. About sampling. Why would people want to generate a surrogate KG or do data augmentation? The aim for knowledge graph embeddings is to answer queries over incomplete knowledge graphs. Does having a generative model help this purpose? 6. About logical constraints. Can we set energies to infinity if a query violates logical constraints? It seems that without the probabilistic framework presented in this paper, one may still be able to enforce logical constraints. 7. About generality of the method. This paper turns the score functions of KGEs into probabilistic circuits. Can the same be applied to turn score functions in all sorts of models into PCs? 8. L231: What does consistent mean? 9. Eq. (3): There are two levels of "=". Please differentiate them to avoid difficulty in parsing the formula. 10. L307: Initializing by CP and ComplEx should not be called distillation. 11. L361: How can KGEs be smaller, now that the presented circuits are already with only 2 layers? Update: The authors addressed my major concerns in the rebuttal.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The paper mentions future work which may be considered limitations. It does not seem to discuss negative societal impact, which it may not directly have.

Summary

This paper provides a reduction from several well-known knowledge graph embedding (KGE) methods (e.g., TuckER, CP, etc.) to *decomposable* circuits $\phi$ which can be efficiently computed in $O( (|E| + |R|) |\phi|)$ (note that this is mostly due to the decomposable nature of TuckER and its derivatives). The authors then address the problem that $\phi$ can be negative and hence suitable to directly represent a probability distribution by applying one of two operations---non-negativity or squaring---which allows the circuit to be efficiently learnt and sampled from. Finally, they show how the circuit formalism allows one to easily apply domain constraints as the product of the two circuits. The authors then evaluate their proposed variants of the KGE methods on four knowledge graphs, where they show (1) their squared variant produces comparable or better results to the original variants with significant computational speedups, (2) their method guarantees domain constraints, and (3) their squared method generates higher quality samples than state-of-the-art. They authors present several additional results in their appendices including: showing that it is feasible to distilling parameters from already trained circuits to the proposed variants (which allows them to be sampled from) and showing that the proposed variants are better calibrated than the original ones.

Strengths

* This is a meticulously executed paper with thorough empirical evaluation. While its key results are in the paper, there are many more interesting and insightful experiments in the appendix. * The paper presents a useful and elegant theoretical framework for computing knowledge graph embeddings that *extends* existing methods like TuckER or ComplEx allowing them to be learned and sampled from efficiently (it's not just conceptual). * The paper is extremely well-written and clear.

Weaknesses

* While the paper reduces several existing KGE methods to circuits, the reduction is quite straightforward and only practical for this family of circuits. There are other KGE methods (e.g. graph neural networks) that are not covered by this paper or its related work.

Questions

The paper is extremely well-written and clear, and I don't have any questions.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

* While the paper reduces several existing KGE methods to circuits, the reduction is quite straightforward and only practical for this family of circuits.