Fine-grained Hallucination Detection and Editing for Language Models

Summary

This paper introduces a taxonomy of hallucinations in large language models (LMs) and proposes a novel task of automatic fine-grained hallucination detection. The authors construct an evaluation benchmark, FavaBench, with over 1,000 human judgments on LM outputs. The authors then train FAVA, a retrieval-augmented LM, to detect and correct fine-grained hallucinations. The results show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection and improves the factuality of LM-generated text.

Rating

3

Confidence

5

Ethics flag

1

Reasons to accept

(1) An important task; (2) An new method to create synthetic data for hallucination detection

Reasons to reject

(1) The authors make a very poor literature about LLM hallucination. Just list two highly related studies -The dawn after the dark: An empirical study on factuality hallucination in large language models Both papers are similar in many aspects: similar taxonomy for hallucination, detection baseline, and metrics. -Halueval: A large-scale hallucination evaluation benchmark for large language models Both papers are similar in many aspects: task, detection baseline, and result findings. In addition, there are also several hallucination dection methods that are not compared by this work. Thus, I find that the literature is highly incomplete. (2) The method to create synthetic data for hallucination detection, while a side effect is that LLM-generated errors may not be similar to the actual genrations of LLMs. That is, the detection method may only learn some shortcut patterns to detect hallucinations.

Summary

This paper proposes a new taxonomy for hallucinations and introduces a new fine-grained hallucination detection and editing task based on it. To evaluate this task, the authors provided FAVABENCH, a manually annotated benchmark. Additionally, instead of using existing LLMs for hallucination detection and editing, the authors also fine-tuned a retrieval-augmented LM called FAVA, which outperformed existing LLMs in hallucination detection and editing. The training of the FAVA model was conducted through synthetic data generation, which also follows the proposed taxonomy. The paper is well-written, and the included figures significantly aid understanding. The topic of LLM hallucinations is also timely and appropriate, and the models and datasets provided by the authors would be beneficial to the community. However, apart from providing a taxonomy, there is no notable technological impact.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

- The paper is clearly well-written. - The paper provides models and datasets (benchmarks). - Well-designed experiments and their results demonstrate the usefulness of the proposed method.

Reasons to reject

I have some concerns about the appropriateness of the taxonomy proposed by the authors: 1. Entity and Relation errors: wouldn't it be possible that relation errors could also be resolved by replacing entities? Additionally, according to Figure 3, it seems that relation errors make up a very small percentage. Do relation errors introduce any unique impacts? 2. Ambiguity between Invented and Unverifiable: The difference between 'invented' and 'unverifiable' seems ambiguous. In Table 1's examples, 'invented' could also occur due to a lack of evidence. The authors state, "Invented are the hallucinations where we can verify that some core entities or subjects of the sentences don’t exist," but how can we be sure that these aren't issues with the retrieval stage? - Regarding 2, could you provide the agreement scores among human annotators specifically for 'invented' and 'unverifiable' errors? It appears that the currently submitted agreement scores do not specify this for each specific type.

Questions to authors

- If Task 1, Fine-grained hallucination detection, involves binary classification for each error type, it's possible that a sentence might contain multiple error types. Then, for Task 2, Hallucination editing, how is editing performed according to each error type? Are there issues such as overlapping span boundaries in each edition? - In fine-grained hallucination detection, does the process involve separate binary classifications for each error type, or can the classification of one error type influence another (e.g., multi-label classification)? - Could you provide more detailed information about the third item on the list of source prompts for FAVABENCH, which states, "50 instructions requiring more reasoning and knowledge by the authors"?

Summary

The authors introduce a comprehensive taxonomy that categorizes hallucinations into six detailed types and develop a novel fine-grained hallucination detection task. They also present FAVABENCH, an innovative benchmark for evaluating LM outputs with approximately 1,000 fine-grained human annotations. Leveraging this benchmark, the authors develop FAVA, a retrieval-augmented LM trained on synthetic data designed to detect and correct hallucinations at the span level. The evaluations demonstrate that FAVA significantly outperforms existing systems in both fine-grained and binary hallucination detection, thereby providing a robust solution for enhancing the factuality and reliability of LM-generated text.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1. The introduction of a comprehensive taxonomy that classifies hallucinations into six distinct categories based on their characteristics and the sources from which they need verification is a significant novelty. This classification allows for a more nuanced understanding and detection of errors than the traditional binary models of hallucination detection. 2. The creation of FAVA, a retrieval-augmented LM that operates at the span-level for both detection and correction of hallucinations, further underscores the innovative nature of the work. 3. The introduction of FAVABENCH as a new standard for evaluating language models could become a pivotal resource for future studies, encouraging more detailed and precise improvements in model outputs. 4. The paper is well-organized. Each section logically builds upon the previous, with clear explanations that make the sophisticated content accessible to readers with a background in the field.

Reasons to reject

1. LLMs are generally trained for broad applications, yet the proposed task demands detailed, fine-grained operations. Therefore, it is plausible that a smaller, task-specific LLM might outperform its more generic counterparts. However, the assertion that smaller LLMs exhibit remarkable performance over models like GPT and Llama is not definitive. If these larger models were trained using the same specialized dataset, one might anticipate even more significant improvements than the smaller, task-tailored models. 2. The authors dedicate substantial space to discussing the proposed task and datasets; however, the description of the experimental settings, especially in Section 7, lacks clarity. The absence of detailed experimental setup information for Table 4 complicates the distinction between the generators used in Tables 3 and 4. This ambiguity significantly delays comprehension of the comparative analysis. I recommend that the authors refine the description of the experimental setup.

Summary

This paper describes an approach to detect and edit fine-grained hallucinations. Several fine-grained hallucination types are defined. A strategy is proposed to generate hallucinations by prompting LLMs. A model is trained on generated data containing references-hallucination-correct triples. The model is asked to detect hallucinations and to edit hallucinations to produce the correct text. Experiments are done on a set of cases. The method is compare to the ones based on LLMs, and shown to outperform the latter.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

The categorization of hallucination is slightly different from the existing ones. This leads to more fine-grained types of hallucination. The way to generate hallucination cases is interesting. LLMs are instructed to insert hallucinations from a correct text. This allows the authors to create a quite large set of hallucination cases (35k). The use of a specifically trained model for hallucination detection and editing is reasonable.

Reasons to reject

Although the authors claim the types of fine-grained hallucination types are new, they are not very different from what others have proposed. The main categories are similar. The details for training the model based on the generated data are missing. The baselines considered in the paper are simple. In the literature, other approaches have been developed (e.g. checking factuality, fine-tuning a model. etc.). These models should also be included a baselines.

Questions to authors

Fig. 2: the tests used in the figure to determine types of hallucination are difficult to follow. For example, from the question "Does the information contradict with evidence?", there are two yes outputs, and from the "Is it verifiable based off web knowledge?" test, there is only one no output. The criteria to determine the types are confusing. While it is interesting to include more (fine-grained) types of hallucination, it is unclear if "subjective" should be included as hallucination. The risk of considering it as hallucination is that many utterances in everyday dialogues (e.g. "I think ...") would be considered hallucinations. This does not make this type of hallucination very interesting. What is the reason to include this type of hallucination? On the other hand some other types of hallucination may require broad knowledge on the topic (invented, unverifiable). Looking at the approach, it seems that only 5 reference documents are included to cover the topic. This is not sufficient to verify if a statement is invented or verifiable. To detect these types of hallucination, more references (or larger models) should be considered. Therefore, the experiments done in the paper may not reflect the difficulty to detect these types of hallucination. Have you looked at the coverage of the 5 references on the topic? Are there cases where the statement is unverifiable with the 5 reference texts, but is verifiable with more reference texts? The type "invented" is interesting. In addition to require broad knowledge to detect this type of hallucination, the generated data should also try to generate statement that are close to the topic. Looking at the examples, it seems that the "invented" hallucinations are randomly generated. This makes it artificially easy to detect, by checking if the topic of the statement is related to the remaining of the text. Have you tested different ways to generate invented hallucinations? Table 4: I guess the numbers are FActScore. Please specify. For hallucination editing, I believe the Human evaluation results are more meaningful than FActScore. It mat be appropriate to move Human evaluation results to the main body of the paper (instead of appendix). The experiments only compare with simple LLM-based methods. There are more methods proposed in the literature. The paper mentions several ones in the related work. Another strongly related work is https://arxiv.org/pdf/2401.03205, which also detects hallucination cases and uses different methods to mitigate hallucinations, such as fine-tuning, different promptings, RLHF, etc. It is appropriate to compare with these approaches.