AmbigDocs: Reasoning across Documents on Different Entities under the Same Name

Summary

The paper focuses on evaluation of LLMs in factoid question answering where the salient entities in the question are ambiguous and the answer to the question could change depending on the ambiguity resolution of the salient entities. The authors create AmbigDocs dataset to evaluate multiple LLMs and conclude that LLMs are not currently created with proper strength considering such questions.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

* The paper studies the knowledgabilty of LLMs and the factuality of their generated answers in a realistic setting. * The idea is quite well thought and the paper asks the right questions and properly answers those questions. * Multiple LLMs are studied so the results are more trustable.

Reasons to reject

* The write up has a few missing citations and could be improved. Please refer to the suggested improvements in the next section.

Questions to authors

Suggested write-up improvements: * Page 1 / Section 1: I'd suggest a retouch on this sentence: " Then, we generate a question and answers from the set of documents, ensuring full annotation of gold document for each answer." * Page 2/ Section 1: For the sentence: "We find that both open-source LLMs and GPT models suffer at identifying different entities, failing to generate complete answers." I would replace GPT with "OpenAI's GPT" for clarification and I would add a citation for it as well (You have citations for LLaMA and Mixtral earlier but (OpenAI, 2023) is missing and is cited in Section 3.2). * Page 2/ Section 2 / Task definition: I believe this definition could be improved by citing and properly explaning the task of Retrieval-augmented generation (RAG; Lewis et al., 2020b; Izacard and Grave, 2021; Singh et al., 2021) which is briefly mentioned in the related works section. * Page 7/ Section 5.1 and 5.2/ Gold only setting: authors may find the work of Kandpal et al. (2022; https://arxiv.org/pdf/2211.08411) related to oracle or gold retrieval settings (for completion of related works coverage). * Page 8/ Section 5.3: "We categorize model outputs into five output categories with the automatic metric we have built in Section 4.2" a period is missing at the end of this sentence.

Summary

Distinguishing between different entities that share the same name poses a significant challenge for language models (LMs). The paper addresses this challenge by introducing a novel benchmark, AmbigDocs, which leverages Wikipedia's disambiguation pages to identify documents related to entities sharing an ambiguous name. The method involves generating questions containing ambiguous names and their corresponding sets of answers from these documents. The goal of this benchmark is to evaluate the ability of LMs to reason across multiple documents with ambiguous entities and provide cohesive answers despite the confusion caused by entities with the same name.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

1. The paper introduces a new benchmark, AmbigDocs, designed specifically to tackle the task of reasoning across documents with entities that share the same name. This benchmark provides a solid resource for evaluating the performance of LMs in handling such ambiguity. 2. Empirical findings presented in the paper shed light on the performance of current LLMs in generating answers for questions involving ambiguous entity mentions. By categorizing incomplete answers and establishing an ontology for evaluation, the paper offers insightful analysis into the challenges faced by LMs when dealing with ambiguity.

Reasons to reject

1. The paper proposes an ontology categorizing incomplete answers and automatic evaluation metrics to identify them. However, how accurate are these metrics in distinguishing between incomplete answers caused by genuine ambiguity and those due to other factors such as model limitations or dataset biases? 2. In the section of "3. Creating AmbigDocs", some details still require further clarification. Please see my questions in "Questions To Authors".

Questions to authors

1. Regarding seed document pair generation (Section 3.1), could the authors elaborate on how the set of documents D is collected from Wikipedia's disambiguation pages? Are there any specific criteria used to select these documents? How to ensure the quality and relevance of the documents obtained from the disambiguation pages? 2. Also in Section 3.1, the method for calculating the relevancy score between document pairs based on n-gram overlap is interesting. However, could the authors explain why a threshold of 3 and 10 is chosen for determining relevancy? How was this threshold decided? The paper mentioned setting relevancy to 0 when n ≥ 10 to indicate almost identical content. However, could this threshold potentially exclude relevant document pairs where the content is similar but not identical? 3. In section 3.2, the paper mentioned obtaining a total of 43,839 questions after filtering low-quality examples. What criteria were used to determine the quality of the generated questions? Could the authors provide some examples of what constitutes a low-quality question in this context? 4. In section 3.3, the paper mentioned two criteria for retaining newly generated answers: passing an NLI entailment check and having a length-normalized log perplexity below a threshold value of 0.2. Could the authors explain the rationale behind these criteria? How effective are these criteria in filtering out incorrect or irrelevant answers?

Summary

The paper defines a methodology for analysing entity disambiguation performance of LLMs. A part of the methodology is the creation of a large scale data set that leverages Wikipedia for entity disambiguation. I have concerns about this paper because it misses a baseline. There are many Wikipedia based tools that allow entity disambiguation, e.g. https://www.dbpedia-spotlight.org/ In addition also since several years, in industry NLP libraries there are tools to create entity linking based on custom data, e.g. https://github.com/explosion/projects/blob/master/nel-emerson/scripts/notebook_video.ipynb Without a baseline using existing tools or an existing language model data set (e.g. based on Spacy), the usefulness of the new benchmark is hard to evalute.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

A novel benchmark about entity disambiguation for LLMs is introduced.

Reasons to reject

The benchmark misses a baseline against which the LLMs are compared. I am withdrawing this concern, the authors explained in the rebuttal that they have something else in mind.

Summary

This paper proposes AmbigDocs, a new benchmark for studying the behavior of LLMs on ambiguous document-based QA, and a methodological pipeline that can be used to create similar benchmarks from Wikipedia disambiguation pages using LLMs. The questions in the benchmark are ambiguous in the sense that they contain a surface form of a named entity, which may refer to multiple actual entities in the answer documents. The paper also gives a new spin to the definition of an answer in the context of LLMs and document-based QA and provides a taxonomy of 5 answer types, against which several SOTA closed- and open-source LLMs (LLaMa 7b and 13b, Mistral 7b, GPT-3.5 and GPT-4) are evaluated. The benchmark and the answer taxonomy are the main contributions. The key findings are twofold: 1) generating complete answers for ambiguous questions in AmbigDocs is a challenging task for the open- and close-source LLMs 2) prompting with in-domain in-context examples significantly improves the performance of LLMs. The paper is generally well-written and the benchmark creation pipeline is clearly presented, although the analysis of results is quite short and shallow, which is the main limitation of this paper. Another weakness is the artificial nature of ambiguity in benchmark questions and its unclear practical applications.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. The behavior of LLMs on ambiguous document-based QA has not been studied before 2. Taxonomy of answer types with heuristics for answer classification

Reasons to reject

1. Detailed analysis of results is lacking 2. The ambiguity in questions seems quite artificial, unclear whether this benchmark has any practical applications

Questions to authors

1. Some of your findings are quite counter-intuitive (e.g. LLaMa 2 with 13b parameters performing worse than the one with 7b parameters, Mistral demonstrating the best overall performance). Do you have any intuition why? 2. What are some practical applications of your benchmark besides examining the performance of LLMs on the ambiguous QA task?

Discussion period now open

Hi reviewers, please take a look at the author's rebuttals and the other reviews for this paper! If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum. It seems there is significant disagreement among the reviewers about this paper. mQPG, you are the most negative about the paper, and your concern is that the paper doesn't compare with existing benchmarks for the task. Does the authors' response clarify this concern, or do you still hold this concern? dFmw and v8yv, you are both very positive about the paper. Would you like to argue for its acceptance?

Re: Rebuttal by Authors

I read the authors' responses and chose to keep my original evaluation.