Complex Query Answering on Eventuality Knowledge Graph with Implicit Logical Constraints

Summary

The paper extends the traditional complex query-answering task into an eventuality-centric complex query-answering task to understand the reasoning at the eventuality level. Specifically, the paper divides the discourse rations into two types of implicit constraints: occurrence constraints and temporal constraints. The occurrence constraints determine whether certain eventuality happens or not. The temporal constraints provide the order of the eventualities' occurrence. In addition, the paper proposes a new memory-enhanced query encoding to reason based on eventuality-centric knowledge graphs. The model first introduces a computational graph that encodes queries, including operations such as relational projection and intersections. The paper proposes a memory-enhanced encoding component that utilizes a memory module to encode constraint information. The operation output is used as a query to access each head eventuality based on the relevance score. The paper then uses an attentional aggregation to sum over the constraint relation and tails. The final model is optimized based on similarity scores with cross-entropy loss.

Strengths

1. The paper introduces a new eventuality-centric complex query-answering task to better model the reasoning at the eventuality level. The paper proposes a new way to provide discourse relations with two implicit logic constraints. The idea of occurrence constraints and temporal constraints is interesting. 2. The paper introduces a new memory-enhanced query encoding to update the query representation with relevance-based constraint representations. 3. The paper tests the new framework with a new dataset sampled from ASER. The model shows strong performance over multiple different baselines. The paper also provides code and dataset construction details in the Appendix. The paper also includes a case study in the Appendix.

Weaknesses

Some parts of the paper are not very clear: 1. In section 3.1, what function is used for relation projection and intersection operation? The paper said that the intersection is a permutation-invariant neural network. However, it needs to be clarified in detail to readers. 2. In section 3.2, what relevance score is used for Equation 5? Suppose the paper used a semantic relevance score such as cosine similarity. In that case, the motivation for this part is unclear because the constraints with higher similarity might not be the ones with closed relevance. Moreover, Figure 4 needs to be clarified. The paper needs to briefly explain the right part (constraint memory in Figure 4.) I suggest going through the walkthrough example in section 3.2. The code does not include a ReadMe file. 3. In section 4.1, the paper only focuses on the answer with constraints. Has the paper also tested the new model for answers without any contradictions? 4. The abbreviation in Table 4 needs to be clarified. It seems that p is the projection, i is the intersection. However, those abbreviations are not clarified in the caption. Readers also find it hard to figure out what e represents. The analysis of Table 4 is superficial. The paper needs to add more qualitative analysis with more concrete examples. The paper needs to conduct an ablation study to show the contribution of each component.

Questions

1. In section 3.1, what function is used for relation projection and intersection operation? 2. In section 3.2, what relevance score is used for Equation 5? 3. Has the paper also tested the new model for answers without any contradictions?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper provides a limitation section and broader impact in the Appendix.

Summary

The paper proposed a reasoning task "Complex Eventuality Query Answering (CEQA)". CEQA is performed over EVKG and is different from traditional CQA over entity-centric KG. Authors of the paper clearly explain the new task, and further discussed a memory-augmented method to improve models' performance on CEQA.

Strengths

The paper made two contributions. First, it proposed the CEQA task and discussed its difference from the traditional CQA task. Then, the paper proposed a memory augmentation method for query encoding. The authors clearly discussed the CEQA task and emphasized its importance in logical reasoning.

Weaknesses

I have a few quick comments of the paper. Maybe it worth briefly discussing System 1 and 2 for the completeness of the paper. Readers may be unfamiliar with the terms. Please consider discussing your dataset created from ASER earlier in the paper and maybe go through a few examples. This task is new, to my knowledge, to many readers including myself. Maybe also discuss ASER a bit more. How is ASER constructed? I have another concern about the quality of ASER, the backbone database of the proposed task. Some statements and/or reasoning can be ambiguous or debatable. For example, someone can say, "PersonX adds soy sauce" is the cause of "Food is bad" due to some other implicit information in addition to the temporal one. How did ASER (or other similar database resolve this problem)? How is ASER related and different from Commonsense QA? Both require reasoning with implicit information. Are there other relevant tasks in addition to the ones you described in the paper. Please include them in the Related Work section. Are you able to experiment with a few more datasets? I understand this is a new task with limited resources. However, experiment results in this paper is limited to fully prove the effectiveness of your proposed method.

Questions

Please see above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The paper proposed a novel reasoning task potentially useful for building more powerful and general reasoning systems. The paper has made substantial contribution in proposing new tasks, but is limited in modeling and experiments.

Summary

This work aims to conduct complex logical query task over eventuality-centric KG (EVKG) and propose the Complex Eventuality Query Answering (CEQA) setting that considers the implicit constraint of the temporal order and occurrence of eventualities. The authors also propose a memory-enhanced query encoding method and achieve state-of-the-art performance on the CEQA task.

Strengths

1. very interesting and important research problem 2. the proposed MEQE module enhances the state-of-the-art query encoder on the CEQA task 3. the paper is overall well-organized and well-written

Weaknesses

1. the novelty of memory-enhanced module is limited since the memory mechanism has been proposed in many prior works such as [1] 2. Recently, path-based KG reasoning methods such as QE-GNN [2] have achieved much research progress and show stronger reasoning ability than embedding-based methods. However, the proposed MEQE seems cannot incorporate the path-based method that didn't learn the representation for each node and relation in KGs. [1] Rossi, E., Chamberlain, B., Frasca, F., Eynard, D., Monti, F., & Bronstein, M. (2020). Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637. [2] Zhu, Z., Galkin, M., Zhang, Z., & Tang, J. (2022, June). Neural-symbolic models for logical queries on knowledge graphs. In International Conference on Machine Learning (pp. 27454-27478). PMLR.

Questions

please refer to the weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes an approach to address the challenge of complex query answering on eventuality knowledge graphs by integrating implicit logical constraints. The authors introduce the task of complex eventuality query answering (CEQA), which requires considering the occurrence and temporal order of eventualities. Methodologically, the paper encodes the edges of the knowledge graph containing these constraints as key-value pairs, which are then integrated into the attention mechanism. The authors extracted eligible data from the ASER dataset and conducted experiments combining their proposed method with various query encoding models, demonstrating improved performance.

Strengths

- The paper's motivation is solid, aiming to incorporate logical information from eventualities into complex query answers. - The method serves as an effective additional information exploitation approach, adaptable to various query encoding models. - The constructed CEQA dataset can provide insights for related tasks.

Weaknesses

- This paper dedicates excessive description to the introduction. The use of multiple representations of first-order logic may be misleading, as the actual encoding is relational. - The work assumes the existence of a knowledge graph and constructs datasets using theorem provers, limiting its generalizability to other tasks. - The paper introduces two types of constraints and presents contradictory answers in the data, but lacks analysis of these categories in the methods and experiments. While Table 2 provides statistics on constraints and contradictory answers, conducting further experiments specifically targeting these categories would provide valuable insights into the functioning of the model.

Questions

- In Figure 3, please provide a brief explanation of the confliction of the "Precedence" relation in the two subfigures. - Could you provide a concise overview of the structure of the permutation-invariant neural network for the intersection operation in Section 3.1? - Consider reducing the number of method-independent first-order logic examples and representations in order to allocate more space for comprehensive experimental and methodological analyses of the defined categories in the paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I have no concerns about the limitations.

Summary

The paper proposes a new task of complex event knowledge graph completion, curates the datasets from existing event knowledge graph, and designs a solution. However, the tasks motivation is not very clear regarding the formal logic form as each event is still in the form of natural language, especially considering the event chains are likely break down along with increasing reasoning paths. The authors need to further discuss the different and importance of the proposed task compared with commonsense reasoning or commonsense knowledge graph completion. Weaknesses 1. The paper is hard to follow and needs further polish. For example, 1. line 53-60, what ‘s the difference between entity-centric and event centric KGC? The given example is not convincing and seems artificial. If the event does not occur, it may not be necessary to included in the KG. So, the difference lies on the Open-world assumption or closed world assumption, not entity- and event-centric. 2. What is the motivation to formulate the commonsense knowledge as the formal logic form? For commonsense knowledge, it has a great chance to hold true with probability. That is, along with the increasing paths, the reasoning chains are more likely false. 2. The proposed task looks similar with commonsense reasoning. So, it is necessary to discuss and compare with the datasets and baseline methods for commonsense question answering or commonsense knowledge graph completion. 3. What is the quality of the curated benchmark?

Strengths

see summary

Weaknesses

see summary

Questions

see summary

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Limitations

see summary

Thanks for the rebuttal

The authors have answered most of my questions. I have raised my score from 4 to 5. I want to clarify the W3 and Q3. So basically, I wonder about the performance of the dataset without any constraints. Will the model cause a performance drop in those instances?