HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection

Summary

This paper proposes a new way of harnessing unlabeled LLM generation as the training set for fact verification. The key assumption is "LLM generates factually correct statements more than hallucinogenic statements", which guarantees the existence of clear subspace between hallucination and non-hallucination statements. As the mean of identifying such subspace, singular decomposition is applied to find out orthonormal vector to project LLM's embeddings. The distance from origin is regarded as the value to define "abnormality". The experiments demonstrate that the proposed idea works out and outperforms several existing methods.

Strengths

* The idea of using unlabeled LLM generation is interesting and show the potential over other counter parts. * Ablation study may include key components we can alter. * The paper is clearly written and solve an important challenge - distinguishing hallucination or not.

Weaknesses

There are many weaknesses and things to be considered for further improvements. **Limited Training Set:** The main advantage of the propose method is the usage of unlabeled LLM generation. It means, we can create much larger training set by generating diverse QA pairs (including multi-domain, subjects, etc). But this study is limited to using only the give training set from a few data sets. Also, the classification performance is much lower than what we can obtain by using labeled set in Figure 5. Showing the potential how this framework gets synergies by adding more LLM generations. (we can simply create such large data by using LLMs with simple prompts or we can add LLM generations on other large-scale QA bench not for fact check). Can the fact verification performance improve by adding more LLM generations? **Issue on Robustness Comparison:** I don't understand why the proposed method gets higher robustness compared with other methods. The propose one still relies on training on a specific data. How could this increase robustness over others? The latent space extracted by a specific dataset cannot be generalized to all domains. I think this is the similarity between the two datasets, TRIVIA QA, TrustfulQA. Please clearly explain what data distribution, domain difference the two data have, and what components help the robustness against distribution shift. **Explanation on Hyperparameter:** In Lines 150-151, I think the methods need $T$ to split the LLM generations into two different groups of $\mathcal{H}$ and $\mathcal{T}$. But, there is no mention about this, and any other studies on ablating the value of $T$. **Visualization Study**: A key assumption is the existence of clear subspace to distinguish hallu and non-hallu statements. I think that the author should provide some qualitative analysis using visualization. We can visualize the projected embedding space with oracle labels on 2D space. This clearly show that how such two classes are separable well in latent space.

Questions

No Questions.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, author stated the limitation on Page 17.

Summary

The paper attempts to address the popular problem of hallucination in texts generated by today's LLMs (large language models). Hallucination refers to false or misleading text generated by LLMs. The paper attempts to address hallucination by proposing a truthfulness classifier, HaloScope, that operates on the unlabeled text generated by an LLM during operation. The paper compares Haloscope with existing algorithms and shows its superior performance. The key idea behind Haloscope is identifying a hallucination subspace in the space of latent (Euclidean) representations of an LLM generation. Thus, when when the latent representation of a generation aligns strongly with this hallucination subspace, the text is classified as potentially hallucinated.

Strengths

The paper is very well-written. The key idea behind HaloScope while being simple in retrospect is novel and a welcome contribution. The exposition in the paper is very clear and the simulations have been excellently presented with all the relevant details. The work on ablation studies (Sections 4.2-4.4) is also a welcome contribution and adds to the overall value of the work. The reviewer believes the research community will benefit and draw upon the ideas and simulations presented in this paper.

Weaknesses

One point I would like to note is with regard to the problem setup. Currently, the way problem is formulated (Equations 1, 2, 3, 4), it appears that the fact that an LLM generation is a hallucination is independent of the corresponding user prompt. This does not seem to be a good assumption to make. However, the empirical results do outweigh the weakness of such an assumption. The reviewer suggests the authors to describe their modeling choice (Eq 4) a little more in detail, specially how it relates to the user-prompt conditioning.

Questions

Line 40-41. By assuming the specific mixture of two distributions. The authors are assuming that hallucinated data is generated with probability \pi independent of the input. Can authors please elaborate. Line 158. What is the difference between lambda and capital T?

Rating

9

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The distribution shift has been identified as a potential barrier to a reliable usage of HaloScope by the authors. The reviewer agrees with the authors and appreciate their candidness. The reviewer would like to suggest mentioning this as potential future work direction in the conclusion.

Summary

This paper presents a technique for detecting hallucinations by leveraging unlabeled data generation. Instead of relying on human annotation, the method automatically distinguishes between truthful and untruthful generations using network embeddings and their projection onto singular vectors with high singular values. Subsequently, this information is utilized to train a classifier. The study demonstrates enhancements across multiple benchmarks compared to several baseline methods.

Strengths

* Leveraging unlabeled generation for hallucination detection without any need for human annotation. * The method scales well for larger models. * The method does not require sampling multiple generations.

Weaknesses

* The method relies on BLUERT for ground truth evaluation. Can you justify the use of BLUERT score?

Questions

* Have you explored the effectiveness of the method on tasks beyond QA, such as summarization? * Can you provide more details on the result of Table 3? What data is used to perform this ablation? This is important because where to extract the embeddings has huge effect on the effectiveness of the method.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The paper is very clear in the problem it is facing and the solution it proposes is also clearly described. Specifically it is looking at detecting hallucinations produced by generative language models. It does so by taking internal representations of the LLM, projecting these onto SVD factorisation which identifies important directions of variation for the models subspace(s) over some relevant data. Membership of unlabelled data (hallucination, or not) is then obtained by projecting the centred representation for the datapoint at hand against the principle singular vector. This provides an unsupervised labelling method. This is used to label all data available, and then a binary classifier is trained on this. This is the final hallucination classifier. Experimental results are given over a few Q+A datasets and against several related baselines for hallucination detection. The proposed method is shown to be the most accurate consistently. It's actually rather amazing I think that the projection against the primary SVD vector (with scores aggregated across these from different parts of the network) works so well as an unsupervised labeller of truthfullness/hallucination. There's no specific information here at all about the problem of hallucination, and this could have captured the information of any other trait (or even meant nothing at all).

Strengths

* Very interesting empirical results on the provided Q+A datasets. * Method is clear and simple, assuming access available to the internals of the LLM. * The ablations across all of section 4.3 are very thorough. These answered the questions I had noted down reading up to that point of the paper.

Weaknesses

* Hallucinations do vary a lot depending on the specific problem at hand, e.g. in question-answering with or without a prompt, or in data-to-text NLG. This paper only looks at question-answer based tasks. It would be interesting to see the method applied on other tasks. There is nothing about the proposed method which limits where it can be applied, it's purely an empirical question as to whether the results are the similar or rather different. * Access to the internals of proprietary, cloud hosted LLMs is not a given. Often only the predictions are obtainable for these. The method won't be applicable in this case.

Questions

* Why is projecting against these subspace primary vectors so informative for hallucination detection? I'm quite surprised at how well this works, given that could capture any other feature of the model/data other than hallucination. The variation of this across the different network layers is significant as shown in Figure 3 c. Based on that, the practical way to apply this would be to measure on every layer before selecting which to use right? * Would it be possible/sensible/silly to incorporate an objective on certain subspaces into the training of the model (if not just zero-shoting the task of interest with an already capable LLM)?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

* Access to the internals of the LLM is the main one, as already noted. * More empirical results over different types of language generation tasks which each have their own nuances to what is a hallucination would be of interest.

Response to Authors

Thanks for the clear response on my concerns and questions. All the things has been resolved, so I increase my score to 6. Thanks!