TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph

Summary

This paper proposes TFLEX, a framework that reasons over temporal knowledge graphs (TKG). It takes a complex query about either an entity or a timestamp and a set of constraints as input. The query is then converted into a directed acyclic graph (DAG), which is a computation graph that projects the query into an embedding space. The paper also defines operations that happens in the graph, including projection, intersection, complement, and other set/temporal logic operations (e.g. After, Before, and Between). When the query embedding is produced, its distances between all candidate entity/timestamp embeddings are computed. The loss function is designed to minimise the distance between the embedding of the query and that of the correct entity/timestamp and maximise the distance between the query and the incorrect candidates. The framework was evaluated on three standard datasets with modifications. The assessment process involves predefined complex queries on top of each dataset, including 27 types for training and 40 types for test. Several variants of the proposed framework were compared, as well as a few previous works on embedding query for static knowledge graphs.

Strengths

- This paper discusses complex query embedding in temporal knowledge graphs, which is an important yet under-investigated topic. - The paper provides a definition to the problem and some attempts to solves the problem.

Weaknesses

Related Works - From my understanding, TKGC is an important subset of the task (Temporal Complex Query) defined in this paper. There are many TKGC works missing in the related work part. For example, [1] and [2]. - The discussion on complex query embedding works is insufficient. The paper should provide a more substantial explanation as to why existing works on static KG cannot be easily adapted for temporal KGs. "They cannot utilize temporal information" is a result, not a reason. Method - In line 146 the paper introduces "the relational embedding" without saying how it is defined. Is it a trainable parameter? How is it initialised? - The definition to operators appears arbitrary. In equation (3), $V_q + r + V_t$ are adding the embedding for entities, relations, and timestamps together. What is the semantics of addition here? - Again, in equation (7), something like $q_f^t+\frac{1+q_l^t}{2}$ is in essence adding the entity feature with its a truth value. What is the semantics of addition between these two values? - Later in the definition of distance function (line 199), $q_l^e + ||v_f^e-q_f^e||_1$ is adding a truth value with a L1-norm. What is the semantics of addition here? Experiments: - The method is evaluated on 40 types of generated queries from three datasets for TKGC. The paper did no discuss the reason for choosing these specific 40 types or how complete these types are. Nor has the it discussed whether TFLEX can generalise to queries outside of those 40 types. - There is no comparison with TKGC works. Since the TKGC task aims at completing a simpler query, it is naturally a subtask of what is proposed in this work. In other words, a (s, r, ?, t+1) completion task can be converted to a query and TFLEX should be able to handle it. I would expect at least a separate table comparing TFLEX and state-of-the-art TKGC on a completion task. Otherwise, it is difficult to assess how well TFLEX captures temporal information. Miscellaneous: - There are technical details being moved to appendix due to page limit. Please note that the page limit is nine content pages, not eight. The current main text is not self-contained to provide readers a high-level idea about the benefits of the framework. For example, as I pointed out above, many definitions and designs about the operators lack intuition or rationale, and look like something randomly popping out. - The source code provided in the supplementary material and the link in the paper reveal a non-anonymous email "l****9@sysu.edu.cn". While I believe this is an honest mistake, I urge the author to remove this information. --- Reference [1] Chronor: Rotation based temporal knowledge graph embedding, AAAI 2021. [2] Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning, SIGIR 2021.

Questions

Please see my points above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The paper has a good discussion about its limitations in page 11.

Summary

The authors introduce an embedding-based method for answering complex, i.e., multi-hop, queries on temporal knowledge graphs. The Temporal Feature-Logic Embedding framework (TFLEX) uses fuzzy logic to model first-order logic operations on the entity and timestamp sets. The queries, with answers being either entities or timestamps, are embedded via four components, including feature and logic vectors for both entities and timestamps. The correct answer to the query is determined as the entity or timestamp whose embedding is closest to the query embedding according to a predefined distance function. Based on existing benchmark datasets, three new datasets were created that include a variety of complex queries. The proposed method TFLEX is compared against state-of-the-art query embedding methods, where TFLEX outperforms all baselines. Ablation studies show the effectiveness of the particular design of the method.

Strengths

- The paper seems to be the first one to address the multi-hop logical reasoning problem on temporal knowledge graphs, where queries consist of disjunctions of atomic formulas. The authors introduce this new task, which can find application and are relevant in many domains. - The proposed method supports FOL operations as well as temporal operations and is able to perform multi-hop reasoning. - Three new datasets are generated, including 40 kinds of complex queries, which can be used for further benchmark experiments. - The experiments and ablation studies are extensive and confirm the effectiveness of the proposed method. - The source code and datasets are available online, which supports reproducibility and further evaluation.

Weaknesses

The main weakness concerns the clarity and preciseness of the technical content, which makes it difficult to follow and to understand the methodology, thereby making it challenging to judge the soundness. Here are some examples: - Figure 1: The query example "During François Hollande was the president of France, which countries did Xi Jinping visit but Obama did not visit?" is given. The logical formulation states that the answer is an entity (country) so that there exists a timestamp T_1 when Xi visited this country and there exists a timestamp T_2 when Obama did not visit this country. This does not seem to reflect the query correctly. The answer should be a country so that there exists a timestamp T_1 when Xi visited this country and for all T_2, Obama did not visit this country. Also, the picture might be easier to understand if the entities for Hollande (e_1, e_3) and France (e_2, e_4) are represented by one node instead of two. - Line 87: The fact set should be a subset of the complete graph. The set on the right-hand side of the equation, however, denotes the complete graph (all possible quadruples). - The definitions for the entity query and the timestamp query are difficult to understand. Maybe it is possible to first introduce atomic formulas, then literals, then conjunctions, and last the disjunctive normal form. Some query examples with corresponding query structures could make it more comprehensible. Why should there be the same number (k) of bound variables V_i and T_i? There is only one relation r in the query definition, but is it not common to have several relations (see Figure 1)? Should “Between” also be one of the possibilities for f? - The method builds on fuzzy logic, which is important for modeling the operators. A short introduction to fuzzy logic directly in the paper (instead of being in the appendix) would help understanding. - The MLPs in Equation 3 are different, so indices could be added (as in later cases) to make it clearer. - Line 160: The Alignment Rule is mentioned and is part of the subsequent equations. Since both formulas (for I_e and I_t) look exactly the same, how are the AND operators calculated differently? - Line 131: ||q|| should be a query set, so is V_q the embedding of all queries or just one query? - Existing literature for temporal KG completion does not only include the two groups tensor decomposition and translation. There are also methods based on dynamic (time-dependent) embeddings, logical rules, autoregressive models, Markov processes, … (see [1] for a survey from January 2022). [1] Temporal Knowledge Graph Completion: A Survey. Borui Cai, Yong Xiang, Longxiang Gao, He Zhang, Yunfeng Li, Jianxin Li. https://arxiv.org/abs/2201.08236

Questions

The definitions section and method section contain the formulation for the two cases entity query and timestamp query and include a lot of notation, which impedes the readability. It is sometimes not clear, which equations are needed for which query type. Since both cases are similar in many ways, one suggestion would be to focus on one case, add more details/examples/descriptions, and include the other case in the appendix or mention the differences in a separate section. Minor comments: Figure 1: “Obama” -> “Barack Obama” (to be consistent with the other names) Line 21: The statement “results are inevitably incorrect” is rather strong. Depending on size and content of the KG, results might be correct. Table 1: The best results could be marked in bold to make it easier to identify the best method.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The authors addressed the limitations and corresponding future work adequately in a separate section. Possible broader impact and application areas are also stated in a designated section.

Summary

This paper proposes a method to learn on temporal knowledge graphs, using a combination of fuzzy logic with a temporal extension and node embeddings. To test the method, three new datasets were generated. The choices made for the model seem logical and are over all well motivated in the paper (e.g. figure 2). The final model was tested on multiple data sets and compared to several non-trivial baselines. Furthermore, ablation studies were performed to analyze the effect of several components of the TFLEX model. While this paper has several strong points when it comes to presentation (e.g. figure 2 and the colors in line 164), there are some lacking areas. Firstly, having to much mathematical notation spread throughout the text itself (e.g. lines 173-180) compromises readability. Secondly, given the novelty of the model, I would have liked to see a detailed diagram representing the model. This paper provides a novel approach for learning a temporal graphs and provides three data sets for this task. Research on temporal graphs is somewhat limited (as seen by the limitations of the baselines), making this a welcome addtion to the sub-area of machine learning on graphs.

Strengths

This paper performs a analysis by motivating choices and thoroughly evaluating the model. The figures that are present are very clear.

Weaknesses

Some of the math could have been beter separated from the text, making the paper more readable. There is not figure to visually convey the model itself.

Questions

To me, it is not clear why we would want up to n compositions of the identity, negation, before and after operations (see line 105). In lines 124-125 you seem to assume a query only has one answer. Is this only for training purposes, or is this a limitation of the model?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The model seems to be limit the answer set of queries to just one answer (the embedding closest to the query embedding).

Summary

This paper studied the multi-hop logical reasoning problem on temporal knowledge graphs and proposed the first temporal complex query embedding framework named Temporal Feature-Logic Embedding framework(TFLEX).Firstly, they defined the task of multi-hop logical reasoning over TKGs. Secondly, they designed the first multi-hop logical reasoning framework which utilizes fuzzy logic to compute the logic part and extend fuzzy logic on timestamp set for supporting all FOL operation and extra temporal operations(After, Before and Between). At last, they generated three new TKG datasets for the task of multi-hop logical reasoning. Experiments on benchmark datasets demonstrate the efficacy of the proposed framework in deal with different operations in complex queries.

Strengths

1. They creatively conducted research multi-hop logical reasoning problem on the TKGs and proposed the first TFLEX framework to answer the temporal complex queries. 2. they relatively proposed using fuzzy logic to handle temporal feature-logical embedding and expanded extra temporal operations. 3. They provided three new TKG datasets and compared them with multiple benchmarks for experimental verification in order to better validate the reliability of their framework. 4. They had clear explanations and explanations of the definitions and methods used, with reasonable classification and appropriate illustrations. 5. There are more detailed compensations and explanations in the appendix to better explain the details of the method and experiment. 6. There are also corresponding open source code and datasets.

Weaknesses

1. In the section on related work, there is a lack of appropriate explanations for the problems and shortcomings of the methods cited in the relevant articles, as well as a comparison with current work. 2. The appendix section provides some overly detailed supplements to the overall concept, such as a more concise description of fuzzy logic. 3. It is not recommended to directly apply source code for explanation and display in the appendix section.

Questions

1. It is recommended that the author compare the methods of relevant articles with the improvements of the author's article to highlight the efficiency of their methods 2. It is suggested that the author can further optimize the content of the paper to streamline the appendix content, and simplify the subsection: Explaining answers with the framework and Experimental analysis into the main body of the paper to better demonstrate its experiments and principles. 3. It is suggested that Pseudocode should be used for interpretation, which is helpful for reading and analysis and is more standardized.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The author thoroughly analyzed its limitations in the article, including insufficient time operators, improved time embedding, long query generation time, MRR and Hits@k Weak. Also proposed its plan for future improvement work

Summary

Authors present new embedding framework called TFLEX to embed complex temporal queries over temporal knowledge graphs to perform multi-hop reasoning with time constraints on TKGs. Authors present the overall embedding framework using Fuzzy logic to model complex logical queries and extending fuzzy logic to include three temporal operators After, Before and Between. It presents new benchmark datasets to evaluate embedding on TKGs and show experimental results to show the benefits of proposed approach.

Strengths

paper presents a new embedding framework and dataset for complex temporal queries, which can benefit the community in extending research in this direction.

Weaknesses

Not sure if this framework can handle complex queries without temporal constraints on par with other complex query embedding methods? suppose if we ignore the temporal aspect of current framework and run the standard benchmarks on complex query handling does it work as well as other methods in the literature.

Questions

Can this solve complex query answering equally well on other bench mark tasks and how does it fare against the prior art?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Reviewer please acknowledge having read the rebuttal

Review SQqW please let me know if your questions are addressed and want to revise your score if applicable.