ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models

Summary

The paper proposes to augment hallucination annotation dataset and improve the performance of the hallucination annotator simultaneously in an iterative self-training framework. During each iteration, they use a Expectation-Maximization Algorithm for data annotation and hallucination annotator training. The trained hallucination annotator can be further used for downstream tasks such hallucination evaluation and hallucination mitigation. Extensive experiments were performed to show the superiority in hallucination detection and the effectiveness of the proposed approach.

Strengths

- The paper is very easy and pleasant to read. Each section and their subsections are well connected and explained without adding much redundant content. Figure 2 is especially illustrative. - I like how they explain a rather practical work in a theoretical manner using all the math equations in Section 3. It makes the work more sound with less confusions. - Although the model name is ANAH-v2, it actually brings big novelty compared to the original ANAH model. The model is in a self-training manner and the training prompt is expanded to do three tasks. - Large amount of experiments show impressive performance on hallucination detection. The thorough ablation study also shows the importance of each component.

Weaknesses

- My biggest concern for this paper is that the model essentially does three things: Factual Existence Judgment, Reference Information Extraction, and Hallucination-Type Judgment. Each step relies on the results from previous step. For example, if the extracted reference information is limited, it will greatly impact the hallucination judgement as well. The experiments mainly focus on the third step without discussions on the previous two steps. So I’m not certain how stable the model actually is. - The hallucination mitigation is a little weak, it basically generates multiple responses and select the response with the least portion of hallucination sentences. There are multiple other ways to mitigate hallucinations that are not compared in Table 8.

Questions

- Typo in ine 76-77, "we reduce the hallucination of the final LLM generations from 25% to 37%." - Line 156, when you use the majority vote to select the the most common hallucination type, wouldn't it be dominanted by "No Hallucination"? As you are also showing in Table 7, the hallucination rate for InternLM2 is less than 20%. If you generate n candidates (depending on how big your n is) and use a majority votes, not many hallucination types will be selected. - How many iterations did you do to get the dataset in Table 1? - In Table 2, any discussions on GPT4 having much better RougeL and BertScore?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations are discussed in Appendix which actually addresses some of the questions I was gonna ask, such as more benchmarks or more backbone models for the hallucination annotator.

Summary

This paper introduces an iterative self-training framework to address hallucinations in large language models (LLMs), enhancing the accuracy of annotators and scaling up hallucination detection datasets. The framework utilizes the Expectation Maximization algorithm to progressively improve the hallucination annotator's performance by annotating a scaled dataset and training a more accurate annotator in each iteration. Experimental results demonstrate that the enhanced annotator surpasses GPT-4 in hallucination detection, achieving state-of-the-art results on HaluEval and HalluQA.

Strengths

1. The paper addresses a unique perspective on the issue, as many current works focus on hallucination detection and mitigation. The focus on automatically constructing high-quality hallucination datasets is crucial for addressing hallucinations in large language models. This is an important and valuable contribution to the field. 2. The structure of the paper is clear, and the writing is concise. The experimental section is detailed and thorough, enhancing the credibility of the results.

Weaknesses

1. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice? 2. How does the paper ensure that the EM algorithm converges through iterations?

Questions

1. The difficulty in automatically constructing high-quality hallucination datasets is not clearly explained. The relevant works in this area, the unresolved issues, and why these difficulties persist are not thoroughly discussed by the authors in the paper. 2. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice? 3. How does the paper ensure that the EM algorithm converges through iterations?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

none

Summary

The authors propose an innovative approach to tackle the persistent issue of hallucinations in large language models (LLMs) during long-form question-answering tasks. Current methods for detecting and mitigating these hallucinations are constrained by limited data and high labor costs. To address this, the paper introduces an iterative self-training framework that scales up the hallucination annotation dataset while simultaneously enhancing the accuracy of the annotators. Using the Expectation Maximization algorithm, the process involves multiple iterations where a pipeline annotates a growing dataset, and the improved annotator is used for the next cycle. This approach not only leads to a highly accurate annotator that surpasses GPT-4 but also effectively reduces hallucinations in LLM outputs. The results show significant improvements in key benchmarks, offering a scalable and efficient solution for managing LLM hallucinations.

Strengths

1. One of the major strengths of this paper is its scalable framework. By automating the annotation process and iterating through larger datasets, the authors manage to overcome the usual limitations of manual data labeling, making the approach both cost-effective and efficient. 2. The application of the Expectation Maximization algorithm to improve annotator accuracy through iterative training is a standout feature. This method not only refines the annotations with each cycle but also ensures that the annotator becomes progressively more reliable, which is a clever and effective use of existing statistical techniques.

Weaknesses

1. One of the big drawbacks of the EM algorithm is its sensitivity to initial conditions. If you don't start with the right parameters, the algorithm can easily get stuck in a local maximum instead of finding the best possible solution, which can be frustrating. 2. The EM algorithm can be quite demanding in terms of computational resources. Each iteration involves a lot of number crunching, which means it can be slow and resource-intensive, especially when dealing with large datasets or complex models. 3. The authors could enhance their literature review by including several highly relevant papers on data annotation. The annotation task has been discussed in [1] [2] [3] [4] [5]. [1] https://arxiv.org/abs/2310.04668 [2] https://arxiv.org/abs/2303.15056 [3] https://arxiv.org/abs/2306.04349 [4] https://dl.acm.org/doi/pdf/10.1145/3613904.3642834 [5] https://dl.acm.org/doi/pdf/10.1145/3594536.3595161

Questions

See weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

No

Summary

This paper proposes an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset and improves the accuracy of the hallucination annotator. The framework is based on the expectation maximization algorithm, alternately annotating a scaled dataset and training a more accurate hallucination annotator on the dataset. A 7 billion model trained by this framework can surpass GPT-4 and obtains state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference.

Strengths

- The paper is well motivated by the fact that large language model hallucination significantly hinders applications but its annotation is difficult and very labor intensive. - The solution based on self-training effectively and feasibly addresses the above challenge. (1) The paper defines a procedure of analytical hallucination annotation, that aligns with human cognitive processes. (2) Staged multi-dimensional data scaling, collecting synthetic data from more large language models and for more numbers of topics and questions, ensures the richness of the dataset. (3) Leveraging EM algorithm is suitable. - Strong empirical results are shown for both in-domain hallucination detection and on existing benchmarks, HaluEval and HalluQA. - Very well-written paper.

Weaknesses

The RougeL and BertScore may not be the most capable metrics for evaluating generated texts.

Questions

Is it possible to leverage large language model based evaluation for hallucination detection?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Appendix E describes the limitations.