Multi-hop Question Answering under Temporal Knowledge Editing

Summary

The paper proposes a novel framework called TEMPLE-MQA (TEMPoral knowLEdge augmented Multi-hop Question Answering) for multi-hop question answering under temporal knowledge editing. TEMPLE-MQA constructs a time-aware graph (TAG) to store knowledge edits in a structured manner, preserving temporal information. It utilizes pre-trained language models (LLMs) to devise an inference path and perform joint reasoning using the TAG and LLMs. The key contributions include the TAG construction method, inference path generation, structural retrieval for knowledge retrieval, and a new dataset called TKEMQA for evaluating temporal MQA. Experimental results on benchmark datasets demonstrate that TEMPLE-MQA outperforms existing baselines for MQA under massive knowledge edits.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. The paper addresses an important problem of handling temporal contexts in multi-hop question answering with knowledge editing, which has not been well-explored in previous work. 2. The introduction of the TKEMQA dataset, the first benchmark tailored for MQA with temporal scopes, is a valuable contribution to the research community. 3. Extensive experiments on multiple datasets demonstrate the superior performance of TEMPLE-MQA over existing baselines, validating the effectiveness of the proposed approach. 4. The paper is well-written, with clear explanations of the methodology and experimental setup.

Reasons to reject

1. While the paper introduces the TKEMQA dataset, more details on the dataset construction process and the quality of the data could be provided. 2. The paper could benefit from more extensive ablation studies and analyses to better understand the contributions of different components in TEMPLE-MQA. 3. The computational complexity and scalability of TEMPLE-MQA, especially for large-scale knowledge graphs, are not thoroughly discussed. 4. The paper lacks discussions on the limitations of the proposed approach and potential future directions.

Questions to authors

1. In Section 4.1 on constructing the Time-Aware Graph (TAG), you mention using data augmentation to capture aliases for entity names to aid entity disambiguation. Could you provide some more details on how this data augmentation is performed and how effective it is compared to not using data augmentation? 2. The results in Table 2 show that on TKEMQA, TEMPLE-MQA underperforms MeLLo on the HES metric in the All-edited GPT setting. What are some potential reasons for this and are there any planned improvements to address it?

Summary

The paper proposes a novel method to multi-hop question answering especially focusing on how to correctly answer temporal questions. The authors use a temporal aware graph (TAG) representing temporal information. The graph is used to retrieve the correct answer to temporal questions. Comparison with previous approaches shows superiority of the new method for two benchmark datasets. In addition, a new data set is proposed.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

- Novel extension of LLM-based Mulit-hop question answering using temporal information - The representation of a temporal graph indicates an important step towards reasoning capabilities of temporal information - The new method shows strong performance compared to previous approaches

Reasons to reject

The paper is sometimes difficult to follow, several important aspects are hidden in the appendix or refer to other papers: - Construction of the temporal graph generation - the metrics used.

Questions to authors

While the overall contribution is definitely novel and the results are impressive, I found it difficult to understand the different steps of the algorithm that was developed. Sometimes the authors referred to the appendix and some of the explanations seemed confusing and incomplete: - 4.1. Comparison to existing approaches:”..store knowledge edits in an unstructured format.. “ please give an example. Does unstructured format simply mean text? - “Historical and current facts are typically stored in unstructured format as f_ols and f_current” this sentence does not add much to the explanation - “See appendix A for details.” PLese add more information in the text because this is an important part of your paper - Data augmentation (SPARQL) and graph construction refer to the appendix again and the text lacks clarity. It may be helpful to include a running example so that the reader can see the effects of the different steps. - “alternate between multiple different and solve phases. “ noun missing after different? - Evaluation metrics: the metrics are specifically developed for the temporal Multi-hop QA and it would be good to describe them in more detail in the paper because otherwise the reader will have a hard time identify the significance of the difference in performance (i.e., Historical Explicit time Question Score (HES) and Current Explicit time Question Score (CES) ) - Joint reasoning: “there is no need to use self- checking Zhong et al. (2023) to check if the retrieved fact contradicts the generated answer. “ why?

Summary

The proposed method introduces a novel approach for handling multi-hop question answering that involves substantial volumes of temporal knowledge edits. The proposed approach consists of 1) extending a fact triple knowledge graph with timestamps (referred to as a time-aware graph or TAG by the authors). Such structured information can then be converted into unstructured text by an LLM for Data Augmentation and Contextual Filtering to enhance retrieval performance. Given these improvements, they also propose using LLM and TAG for chain-of-thought type of step-by-step joint reasoning.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- The proposed approach addresses an interesting and challenging problem on multiple-hop QA with temporal time stamps. - Expanding the knowledge graph with time stamps is an interesting direction to explore. - The authors develop a temporal multi-hop QA dataset that can benefit the community. - The proposed approach is novel, although seems to stitch multiple different existing papers.

Reasons to reject

- My main concern is with the evaluation. Are there quality controls on the created TKEMQA dataset? In section 3, the temporal KE from start time to end time is added to the original fact triple. However, it was not clear to me how reliable the augmented facts are. Are these being evaluated? It seems that they are only evaluated on downstream multi-hop QAs with accuracy. - There are some recent papers on temporal multi-hop QA datasets that are worth comparison to the differences. e.g. PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering https://arxiv.org/abs/2402.11034

Summary

This paper proposed a temporal knowledge augmented multi-hop question answering method called Temple-MQA. This method built a time-aware graph to store edits as a time-conscious knowledge structure. The benefit of this graph is to store the temporal information in a structured format. During the retrieval, Temple-MQA utilized a LLM to devise a path and do a step-by-step reasoning. It is interesting to use LLM to strategize an inference path. In general, the whole idea is clear with a reasonable performance.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

Strengths: 1. This paper presented a temple-MQA method that achieves a high accuracy for multi-hop temporal QA task. The whole architecture includes the following key steps: 1) construct a time-aware graph (TAG) to store time information; 2) build an inference path for each multi-hop question using LLMs; 3) design a joint reasoning way to combine LLMs with TAG. 2. The performance of proposed temple-MQA achieves the state of art performance on multiple metrics. It proves the effectiveness of proposed method.

Reasons to reject

Weaknesses: 1. The temporal questions include two types: questions with explicit time (like 01/01/2024) and questions with hidden time (like before, after). It will be wondering if more deep analysis cross different types are provided. The questions with hidden time will require more support about multi-step reasoning. Maybe this method can perform well on various types. 2. Using the LLMs to generate the inference path is a great idea. But this section missed some details. I would like to see more comparisons and analysis about this direction. The accuracy of inference path will impact the performance a lot.

Questions to authors

Look forward to more analysis on various temporal question types.

Re: Rebuttal by Authors

Thanks a lot for providing more results and clarifying the questions. It is pretty interesting about the implicit time analysis. I would like to recommend to add it into the final version. In general, the whole paper looks great. I'd like to raise my rating.