Estimating the Hallucination Rate of Generative AI

Summary

This paper presents a Bayesian interpretation of in-context learning. This interpretation enables us to calculate the hallucination rate. In other words, by considering in-context examples as observations, the posterior distribution can be computed and the hallucination rate is derived. Numerical experiments verify the practicality of the interpretation.

Strengths

1. The hallucination is one of the most important problems of large language models. This paper explored a theoretical understanding of the problem in in-context learning. 2. The idea to interpret the in-context examples as the observation, described in Section 2.1, is original and interesting.

Weaknesses

In summary, I suggests that the authors clarify the focus of this paper either on the hallucination rate for NLP tasks or on the error rate for general in-context learning tasks. 1. As discussed in L222, the proposed theory cannot be applied as is in NLP tasks and an approximated metric is required. Although the introduction of this paper focused on NLP tasks, the usefulness of the proposed theory in this area is limited, as reported in L265 with Figure 6. 2. Hallucination is a complex phenomenon with a wide range of causes [1]. As mentioned in the above weakness, the effectiveness of this theory is limited to the synthetic regression task. In light of these, I am concerned whether it is appropriate to call the phenomenon examined in this paper a hallucination rate rather than an error rate. 3. The claim of Section 2.2 is to use the cumulative probability of the token generation probability distribution as the confidence for token y. However, the use of cumulative probability as the confidence of token y is a common technique in NLP, for example in nucleus sampling [2]. To demonstrate the importance of the theory presented in this section, I consider it necessary to show its universality when other statistics S are used. Otherwise, I also suggest moving this section to the appendix as noted in the following weakness. 4. L146 says `Algorithm 1 can be understood intuitively without appealing to any Bayesian arguments’. I agree with this statement, and I consider that the necessity of the presented theory depends largely on the discussion in Appendix B.1. Because the appendix is just supplementary material and cannot play a role in justifying the main claim, I suggest the authors should revise the structure of the paper. [1] Huang+, A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv:2311.05232. [2] Holtzman+, The Curious Case of Neural Text Degeneration, ICLR2020.

Questions

See Weaknesses.

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

4

Limitations

This paper discussed broader social impact.

Summary

This paper presents a method for predicting the hallucination rate of in-context learning with conditional generative models.

Strengths

- NA

Weaknesses

- Unclear how many queries would be require to validate the approach, as this will be very dependent of task, context, and LLM - this is clearly a missing element of the evaluation - Evaluation on Llama 2 only - more is definitely require to validate the approach - No identification of where the hallucination might come from - Title is misleading - this is basically a hallucination identification method - Non cost effective as this require multiple LLMs calls

Questions

- Why not more LLMs used for experimentation? - What could be the right number of LLM call needed? - What parameters is the function of hallucinate rate? What are the optimal parameter? for which task? for which LLMs?

Rating

2

Confidence

5

Soundness

2

Presentation

2

Contribution

1

Limitations

cf. weakness and questions

Summary

The paper focuses on the in-context learning setting of generative AIs, such as large language models, and proposes a new definition for hallucination. It introduces a novel metric, PHR, along with a corresponding estimation method. Unlike traditional metrics, the proposed metric accounts for label ambiguity resulting from unclear task specifications.

Strengths

- The paper highlights a significant issue with using error rates to evaluate hallucination, that the ambiguity on labels caused by the lack of task specification has been overlooked. - The definition and algorithm of PHR are logically sound and make sense to me.

Weaknesses

- The notations are messy and confusing. - - There is misuse and abuse of uppercase random variables and lowercase instantiations. For example, the distribution of $F$ is referred to as $p(f)$ in lines 87 and 121, while it is referred to as $p(F)$ in line 124. This problem is exacerbated in Theorem 1, making it hard to read. - - The notations PHR/THR can refer to both the definitions and the algorithms, which adds to the confusion. For instance, in the sentence on line 174, “we evaluate the PHR in a setting where we know the true mechanism f* so that we can compare it directly against the true hallucination rate (THR),” PHR appears to refer to the algorithm, while THR seems to refer to Definition 3. - The experimental setup is difficult to understand. - - For example, the experiments in Section 3.1 are supposed to evaluate the proposed method in a setting where $f*$ is known. However, what $f*$ is and how THR is calculated are not explained. To understand this, at least the way $p(y|x, f*)$ is set should be specified. Given the unused space at the end of the paper, there is no reason for the authors not to elaborate on the experimental settings. - - In Section 3.2, the authors attempt to justify the use of PHR by introducing another proposed metric called MHR. It is unclear how MHR could serve as a gold metric. Even if MHR is a gold metric, why not just use MHR? - Typos: Line 31 "the" should be deleted.

Questions

Please refer to the weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations have been well discussed.

Summary

The paper proposes a method to estimate the hallucination rate for in-context learning (ICL) in a conditional generative model (CGM) from a Bayesian perspective. The authors assume the CGM samples from a posterior predictive distribution over a Bayesian model of a latent parameter and data. They define Posterior Hallucination Rate (PHR), along with True Hallucination Rate (THR) and Model Hallucination Rate (MHR), and provide theoretical proofs and empirical demonstrations to show that they can accurately estimate the true probability of hallucination.

Strengths

S1. The paper tackles an important and well-known issue of hallucination in language models. By focusing on estimating the hallucination rate, the research contributes to improving the reliability and trustworthiness of outputs from CGMs. This makes the work highly relevant in the context of ongoing AI issues. S2. The paper introduces new metrics, particularly PHR, for future evaluation of model hallucinations. This can potentially advance methodology for assessing model output reliability. S3. The metrics are well-defined and the pseudo-algorithms help in understanding how the metrics are used.

Weaknesses

W1. While the authors tested their methods on six datasets from various domains, the range of label content and types may be limited. This limitation could introduce confounding factors. For instance, tasks where one label can be a substring of another (e.g. “similar” vs “dissimilar”) might not fully represent the complexity of real-world scenarios. W2. The labels used in the datasets, such as “entailment” vs “not”, do not have a clear semantic meaning and relation like “positive” vs “negative” OR “subjective” vs “objective.” This absence of semantic relation might influence the ICL performance. Testing whether the semantic meaning of the labels affect performance and hallucination rate would strengthen the findings. A suggested approach is to use neutral labels such as letters (A,B,etc.) to determine if the lack of semantic meaning of the labels impacts the hallucination detection. W3. Related to W1, the datasets used in the experiments have either 2 or at most 4 categories. This limited variety raises questions about how the method performs with more complex classification tasks. For example, how would the method fare in English-to-French word translation, where the number of possible categories is significantly higher? Expanding the experiments to include tasks with a larger number of categories would provide a more comprehensive assessment of the method’s robustness.

Questions

Q1. What constitutes an “in-capability” task and “out-of-capability” task? Q2. In Figure 10, why do the lines, especially for context length 50 graphs, stop earlier? What does that imply about the performance or behavior of the model at longer context lengths? Q3. The paper references Fong et al. [22] proposing Martingale Posterior distribution that uses posterior predictive. Are there any baseline results for this approximation that could be compared to PHR to evaluate which distribution better defined models?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have addressed the positive and negative societal impacts of their work. It would be beneficial to further discuss and address the dataset choices and potential limitations, such as label diversity and number of categories, to provide a comprehensive view of the conditions under which the method has been tested. This would enhance the understanding of the method’s applicability and generalizability ensuring a more thorough assessment of its limitations.