Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space

Summary

The paper introduces the metric "Semantic Density" to quantify the uncertainty of LLMs by measuring the distance of response embeddings in a semantic space.

Strengths

- **Structure**: The problem is well-motivated and the paper is clearly and coherently written. - **Theory**: The adaptation of kernel density estimation techniques to semantic space for uncertainty quantification in NLG is novel. - **Experiments**: The experiments were performed on multiple datasets and various SOTA LLMs, where Semantic Density shows good empirical performance given the considered correctness metric.

Weaknesses

- **Theory**: Semantic Density is theoretically not well grounded in uncertainty quantification theory, but proposed as an ad-hoc solution for practical application. So, although it seems intriguing to assign an uncertainty score response-wise rather than prompt-wise, uncertainty theory suggests that the (aleatoric semantic) uncertainty has to be quantified prompt-wise, no matter what ends up being sampled as the response [1]. However, since the true uncertainty score for a given prompt is unknown, the performance of an uncertainty measure is usually evaluated by how well the scores align with the correctness of the most-likely response. - **Experiments**: According to the experimental setup in line 897, an answer is considered to be correct if its Rouge-L to any of the reference answers is larger than 0.3 for all datasets. However, since the considered datasets usually result in short responses, a fixed threshold of 0.3 is not sufficient. For instance, in the TriviaQA question *What is the last Grand Slam tennis tournament played in a calendar year?* the reference answer *The US Open* and the sampled response *The Australian Open* achieve a Rouge-L of approximately 0.66. This highlights the fact that considering multiple thresholds is essential to assess the performance of uncertainty metrics. --- [1] L. Aichberger, K. Schweighofer, M. Ielanskyi, and S. Hochreiter. Semantically Diverse Language Generation for Uncertainty Estimation in Language Models. arXiv preprint arXiv:2406.04306, 2024.

Questions

- What do the authors exactly mean by Semantic Density rebuilding the output probability distribution from a semantic perspective? (line 58) - In line 277, the authors write "After investigation, the inherent sequence likelihood returned by the original LLM was badly calibrated in these two cases.". What is meant by badly calibrated sequence likelihoods? Why didn't the authors account for this by recomputing the sequence likelihood? - There exists independent work (Kernel Language Entropy) that is closely related to Semantic Density [2]. The authors should consider discussing this in their paper. - It would be insightful to additionally report the performance of existing uncertainty estimation methods when decreasing the number of reference responses instead of only reporting Semantic Density with different models (section 4.2) - Semantic Density uses diverse beam search to sample responses, requiring hyperparameters such as diversity penalty and number of groups. How have these hyperparameters been chosen? Is the method sensitive to the specific sampling strategy? An ablation study on the performance of Semantic Density compared to existing uncertainty estimation methods would be insightful. - Have the authors considered using sampling strategies other than diverse beam search? For instance, Aichberger et al. (2024) introduced sampling semantically diverse sequences, which could be utilized to sample unique reference responses [1]. --- [2] A. Nikitin, J. Kossen, Y. Gal, and P. Marttinen. Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities. arXiv preprint arXiv:2405.20003, 2024.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors did not address the limitations of their work, only the limitations of existing uncertainty estimation methods.

Summary

The paper proposes a novel framework for quantifying uncertainty in LLM's responses. The authors introduce the concept of "semantic density" (SD), which measures uncertainty from a probability distribution perspective in semantic space, applicable to any pre-trained LLM without additional training. Experiments on various state-of-the-art LLMs and benchmarks demonstrate that SD outperforms existing methods in accuracy and robustness. The paper presentation is overall easy to read.

Strengths

1. The paper introduces a novel method, semantic density, that quantifies uncertainty in LLM responses using semantic information. 2. The proposed framework is clear and detailed, including its theoretical foundations and implementation details. 3. The proposed method is off-the-shelf and can be applied to other scenarios.

Weaknesses

1. The comparison with existing methods, although extensive, may not cover all possible alternatives. There are recent or lesser-known methods that also warrant consideration. 2. While the method shows promise for free-form question-answering tasks, its applicability and performance in other types of tasks (e.g., summarization, translation) are not explored, which limits the generalizability of the findings.

Questions

1. The literature review is not comprehensive, there are lots of related works [1, 2, 3] not mentioned or adopted in the experiment. 2. The AUROC is not enough the show the strength of the proposed model. It would be more beneficial to show Area under precision-recall curve for the misclassification detection task as well. 3. The prompt construction can have more details. In Sec. 3.2, it's worth denoting the final prompt structure, since the in-context learning examples and prompt structures can make the final result different. 4. The dataset are mostly QA tasks, it seems the model can also handle other NLG tasks. Can you elaborate more on this? [1] Ling, Chen, et al., "Uncertainty Quantification for In-Context Learning of Large Language Models." (NAACL 2024) [2] Fadeeva, Ekaterina, et al. "LM-polygraph: Uncertainty estimation for language models." (EMNLP 2023) [3] Duan, Jinhao, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. "Shifting attention to relevance: Towards the uncertainty estimation of large language models." ACL (2024).

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper addresses the problem of LLM’s lack of uncertainty metric for the response it generates. The authors propose to use semantic density to quantify such uncertainty, as it is not restricted to any specific downstream task. In particular, the approach samples reference responses, analyzes semantic relationships, and then calculates the semantic density. Experiments on four QA benchmarks of several Llama and Mistral models are conducted, including the latest Llama 3 13B and Mistral-8x22B, to demonstrate the effectiveness and the robustness of semantic density compared to prior metrics.

Strengths

* The proposed metric is not restricted to any specific task but measuring the general LLM ability * The experiments show the effectiveness of the proposed approach, outperforming the other previous metrics

Weaknesses

* Only some of the llama and mistral models are tested * The results shown in Table 1 seem to be quite close for all the models (on each task), which are not so indicative for system comparison * No further analysis/discussions on different model sizes (7B vs. 70B), nor different architecture (MoE vs. Non-MoE)

Questions

In Figure 1, different models have inconsistent performance across different tasks. Do you have any explanations?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper proposes an approach for estimating uncertainty of LLM outputs. The proposed approach begins by sampling a diverse set of responses, analyses their equivalence using a NLI classification model, and then computes the semantic density using a kernel density estimate.

Strengths

- The proposed approach seems sufficiently different from prior work and combines existing ideas meaningfully. The approach is convincingly validated in experiments. - The paper is clear for the most part. - The problem of uncertainty estimation for generation tasks is meaningful.

Weaknesses

- The paper is somewhat vague in distinguishing itself from the closest prior work on Semantic Entropy - this makes its specific contribution unclear. - Some details of the implementation of semantic density are unclear.

Questions

- Sec 3.5: Please describe how the NLI model is used for computing p_c, p_n and p_e. NLI models typically accept pairs of texts, but these probabilities seem to contain three: y_*, y_i and x. - Do I understand correctly that embedding models (discussed in Sec 3.2) are never explicitly used in Semantic Density? Instead the NLI model is relied upon for establishing equivalence between texts and for the kernel function? If yes, the current presentation comes across as a bit confusing - please consider rewriting parts of the paper to make this more explicit. - As I understand, the proposed approach only estimates uncertainty rather than explicitly calibrating the underlying model - and yet the proposed uncertainty estimation method helps improve predictive performance in Table 1. Am I understanding correctly that the underlying models are somewhat well calibrated to begin with? To put it another way, are there generation tasks where one might expect SD to not be as reliable? For example, a summarization task. If this is a reasonable understanding, please consider discussing where SD is unlikely to work well. - The differences highlighted against Semantic Entropy in Lines 105-107 are vague or inaccurate - SE considers responses in computing the SE(x) value so it is unclear how it is "prompt-wise", that being said, what are the advantages of obtaining uncertainty for a specific output and prompt pair? What does a "one-cut equivalence relationship" mean? Please consider highlighting differences more clearly and discussing the specific advantages/disadvantages that the differences result in.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Response to Reviewer koQE

Thanks for your comment. The motivation of most summarization benchmarks are to evaluate a model’s ability in compressing the information, so the DUC summarization task used here requires the language models to summarize a paragraph within a predefined length limit, usually no more than a single long sentence. The current NLI model we used (Deberta-large-mnli) still works reasonably in such tasks. However, we think you did bring up a good point for future work in this research direction, i.e., how should we quantify the uncertainty of a long paragraph? Recent solutions to this scenario focuses on decomposing the long paragraph into pieces of “claims” [1] or “factoid” [2], each one being at sentence level, then we can apply existing uncertainty quantification methods to each of them. This strategy makes sense since it is indeed more informative if we could quantify the uncertainty for each part when the response is a long paragraph containing multiple claims. However, it would also be interesting if we can further develop a NLI model or embedding model that can properly measure the overall semantic similarity between two long paragraphs, or we can directly ask another LLM to make this overall similarity evaluation quantitatively. This discussion may be a little out-of-scope, but we will add this discussion in the future work section. Thanks again for your inspiring comment. [1] Xin Liu, Muhammad Khalifa, and Lu Wang. “LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses”. In The Twelfth International Conference on Learning Representations (ICLR), 2024 [2] Farquhar, Sebastian and Kossen, Jannik and Kuhn, Lorenz and Gal, Yarin, “Detecting hallucinations in large language models using semantic entropy”, Nature 630, 625–630, 2024

Follow-up response to the general rebuttal

Due to the space limitation of initial rebuttal, we did not include all the detailed results for the newly added experiments. As per the request by Reviewer koQE, we have now added the detailed results for all the newly added experiments. The detailed results for experiment 3, 4 and 5 are added in the comments replying to Reviewer koQE. The detailed results for experiment 6 are directly included below: ## Results for the modified version of semantic entropy [1]: | AUROC | CoQA | TriviaQA | SciQ | NQ | |---------------|-------|----------|-------|-------| | Llama-2-13B | 0.587 | 0.726 | 0.576 | 0.576 | | Llama-2-70B | 0.570 | 0.696 | 0.596 | 0.555 | | Llama-3-8B | 0.606 | 0.736 | 0.587 | 0.582 | | Llama-3-70B | 0.571 | 0.694 | 0.582 | 0.563 | | Mistral-7B | 0.572 | 0.734 | 0.613 | 0.568 | | Mixtral-8x7B | 0.567 | 0.699 | 0.576 | 0.612 | | Mixtral-8x22B | 0.562 | 0.690 | 0.592 | 0.565 | Its performance is comparative to the original semantic entropy [2], and the proposed semantic density still outperforms it consistently across all the base LLMs and tasks. [1] Farquhar, Sebastian and Kossen, Jannik and Kuhn, Lorenz and Gal, Yarin, “Detecting hallucinations in large language models using semantic entropy”, Nature 630, 625–630, 2024 [2] Lorenz Kuhn and Yarin Gal and Sebastian Farquhar, “Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation”, The Eleventh International Conference on Learning Representations (ICLR), 2023

Follow-up comment to Reviewer kNB4

As per the request by Reviewer koQE, we have now added the detailed results for all the newly added experiments, including the one suggested by your that adds a new variant of semantic entropy. Please see our follow-up comment in the general rebuttal above for detailed results. Thank you.

Thank you

Thank you for the additional results and the responses.