Large Language Models Must Be Taught to Know What They Don't Know

Summary

This work investigates the calibration of LLMs, i.e. how to estimate reliable confidence in their responses that correlates well with the actual likelihood of being correct. Through experiments, the authors reveal that existing methods fall short of achieving accurate confidence estimates, and their calibration performance does not improve with the increased size of LLMs. The authors argue that fine-tuning is necessary to obtain reliable confidence estimates. They explore three fine-tuning approaches: Probe, LoRA, and LoRA + Prompt, and their experiments confirm the effectiveness of these paradigms. Additionally, the authors examine the generalizability of these methods and their benefits for users in making decisions.

Strengths

Originality: This paper proposes three fine-tuning methods for estimating confidence in LLM responses. While previous works have used similar techniques like Probe to gauge uncertainty, the combination of LoRA and prompts for calibration is novel. Quality: The ideas presented in this paper are well-motivated, and the experiments effectively verify the authors' hypotheses. Clarity: The paper is well-written and easy to follow. Significance: The paper's conclusion, that LLMs need to be fine-tuned to predict accurate confidence, is valuable for the community. Additionally, the investigation into the benefits for users in making decisions is both novel and significant.

Weaknesses

1. One clear drawback of the proposed methods is that they require access to the parameters of the LLM, making them inapplicable to black-box LLMs. Figure 4 (right) indicates that a fine-tuned smaller model can estimate the confidence of larger ones. However, it would be preferable to see a comparison between these fine-tuning-based methods and other techniques that do not require access to the LLM, such as sampling. 2. The authors claim that "(LoRA + Prompt) performs almost as well in transfer as on in-distribution data." However, in open-ended generation, the gap between transfer and in-distribution is almost 10 points for both ECE and Auroc, which is a significant margin. For instance, ECE increases by nearly 80%. 3. Section 6.1 is somewhat confusing. While the two possibilities presented seem reasonable, the experimental investigation appears problematic (correct me if I'm wrong). The topic information is not only available in model generation but could also be reflected in the questions. The experiment that replaces the model generation with a random sentence does not adequately address this situation. On the contrary, the results seem to support that the model can predict reliable confidence based solely on the question.

Questions

1. I don't quite understand Figure 3 (Left). The figure with the y-axis labeled "# train" shows the distribution of training samples. What does the figure with "# MMLU" represent? 2. From my experience, in the sampling baseline, the size of the largest cluster can be used to represent uncertainty (size of the largest cluster/# of clusters), and sometimes it performs better than probability. Have you tried this approach? 3. Do you use the correct and incorrect answers provided in the datasets, or do you sample LLM generations to construct incorrect answers? If it is the former, would the latter approach help to better evaluate the uncertainty of the LLM's own generation? 4. The following related studies are missing: [1] LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses [2] Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors acknowledge the limitation of needing to train separate models. Personally, I believe that at least the Probe method can be directly applied to the LLM without altering the LLM itself. Exploring how to better evaluate confidence directly from the LLM could be a promising direction.

Summary

The paper investigates uncertainty calibration via fine-tuning models with correct and incorrect answers. The paper concludes that training through the features of a model on a thousand graded examples using LoRA outperforms prompting-based method and linear probes. The paper also finds that models can be used not just to estimate their own uncertainties but also the uncertainties of other models. Human study shows LLM uncertainty can affect human decision making.

Strengths

1, The paper emphasizes the importance of training-based uncertainty estimation methods. 2, The experiments cover various aspects, including data amount, distribution shifting, cross-model generalization, and a human study.

Weaknesses

1. Lack of technical novelty. The training methods regarding Probe, LoRA, and LoRA+Prompt are highly similar to previous works. No new technical method is proposed. 2. Limited experimental scope: The experiments are mainly based on the MMLU dataset. This raises concerns about the generalizability of the findings to more tasks and datasets. Previous research has shown that task/dataset complexity can significantly impact uncertainty calibration performance. 3. Inconsistent results across MMLU subsets: As shown in Figures 8, 9, and 10, sometimes LoRA + Prompt performs better, sometimes LoRA performs better, and sometimes Probe performs better. Additionally, there is no consensus between AUROC and ECE on MMLU subsets. Methods that perform best on AUROC sometimes perform poorly on ECE. The inconsistency is a common observation rather than an exception. Based on that, the average performance among subsets (Figure 2, left) is not convincing when performance varies significantly across different subsets. 4. Lack of confidence distribution for correct and incorrect predictions on MMLU: The paper does not show confidence distribution for correct/incorrect predictions on the MMLU dataset. Although Figure 3 (right) shows confidence distribution on answerable/unanswerable sets on the SelfAware dataset, a calibrated uncertainty method should also distinguish between correct and incorrect predictions on the MMLU dataset.

Questions

Upon the statement of self-knowledge, “We show that, by contrast, Mistral 7B actually has better AUROC values when applied to LLaMA-2 7B than LLaMA-2 7B applied to itself” (line 295), how is Mistral 7B applied to LLaMA-2 7B? Specially, 1, Which model is used for training? 2, Which models generate the answers for the training data and test data, respectively?

Rating

3

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

good discussion

Summary

This paper studies calibration of large language models. Firstly, the paper points out the zero-shot outputs of LLMs are poorly calibrated, especially in an open-ended generation setting. Secondly, the paper explores fine-tuning a separate LLM for calibration, and demonstrates its good properties such as OOD generalization, data efficiency, etc. Lastly, the paper presents a preliminary study of presenting model uncertainties to human users and investigating how this affects human decision making process.

Strengths

* The paper is very comprehensive and presents an in-depth analysis on the calibration of large language models. * The takeaway messages are summarized clearly after each section. The insights are useful to users and developers of large language models. Plus, the proposed future directions are interesting.

Weaknesses

* The paper mainly focuses on calibration of zero-shot prompting LLMs, while prompting the model with few-shot examples may also be a viable calibration method. I think this should be added into the discussion and maybe added to the experiments to further strengthen the work. * Due to the page limit, many important details are left in the appendix and the paper may be improved with a better organization.

Questions

Clarification questions: * Line 162: Do your results contradict with those in Kadavath et al. 2024? If yes what are differences in your experiment settings that may lead to this? * Line 177: Does each dot in Figure 1 (right) represent one task in MMLU, or representing one LLM? Were different sizes of Llama2 and Llama3 used here? * Figure 1: Do you observe any patterns for base and instruct models? As discussed in https://arxiv.org/pdf/2303.08774 Figure 8, post-training may hurt calibration. * Section 5: I think more details are needed for me to understand how the fine-tuned calibration model is trained. * How do you get P(correct) from the model. Did you train a classification head? Or is it obtained in a way similar to the zero-shot classifier in Line 159? * How do you formulate your training set? It was mentioned in the abstract that a mixture of correct and incorrect answers are needed. However this was not explained clearly in Sec 5. * How is the trained calibration model different from a trained “verifier”, e.g., as done in https://arxiv.org/abs/2110.14168 * Line 287: “model trained on incorrect answers perform surprisingly well”, does this mean the two hypotheses are both true and account for part of the success of the the trained calibration model? * Figure 5: I’m not sure how to interpret the figure. How do you get “an increased reliance on the LLM for more confident answers, and decreased reliance for less confident answers” from subfigure (b)? Is this done by comparing subfigure (a) and (b)? Discussion questions: * Line 132: “Early work on LLMs such as GPT-3 showed … poorly calibrated”. GPT-3 actually showed reasonable calibration on MMLU in a few-shot in-context learning setting (Figure 11 in https://arxiv.org/abs/2009.03300). I wonder what’s the authors comments on this. It seems that the paper mainly focuses on zero-shot settings. Is there any reason this is not included in the paper? Would few-shot ICL also be a viable method for calibration? * I like the idea of infusing calibration into the pre-training phase, discussed in Sec 8. Can you elaborate on the challenges of online learning here? From what I understand, the calibration training data used in Sec 5 is stationery. Missing Reference: You may be interested in * https://arxiv.org/abs/2104.08315 (Corresponds to the phenomenon that you mentioned in Line 137) * https://arxiv.org/abs/2302.13439; https://arxiv.org/abs/2401.06730 (Studies verbalized uncertainty in LLMs) * https://arxiv.org/abs/2305.14802 (Studies task-level calibration by training a meta-model, also studies open-ended generation settings) Overall I think this is a solid and insightful submission. I'm open to adjust my ratings if some of these confusions are resolved.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed in Section 8.

Summary

### Summary This paper presents a method to quantify the confidence of predictions from an LLM. Unlike some previous work, the authors propose a training-based approach to teach a model to generate reliable confidence scores. The experiments report Expected Calibration Error (ECE) and AUROC, commonly used in calibration literature, to ensure that the model's predicted confidence aligns with actual likelihood. Typically, papers in this area either tune the model or do some post-processing on the model to obtain calibrated confidence scores. However, this paper is slightly different in its setup, and I found the title somewhat misleading. The main difference here is that there is an auxiliary model to emit the confidence scores, entirely independent of the primary model, as stated in Line 327 of the paper, "Currently, finetuning relies on two separate models for question answering and uncertainty estimation." In this sense, this auxiliary model serves better as a potential guardrail to filter out less confident predictions, and framing the paper in this light would make it easier to convey its message. I also dislike using the terms "uncertainty estimation" and "confidence estimation" interchangeably. Please refer to Section 3.2 of the paper "Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models," which does a good job of eliciting the difference between uncertainty and confidence. #### Main Contribution of the Paper: The central contributions of the paper are - Assembling a finetuning dataset to calibrate the model on. - LoRA finetuning of the auxiliary model for reliable uncertainty estimate. - Testing the robustness of this model to distribution shifts. The paper also contradicts findings from previous papers, such as - Frozen features are insufficient for uncertainty estimates (The internal state of an LLM knows when it's lying) - Transferability of uncertainty estimates across models, which is different from the findings in Kadavnath et al. (Language Models (Mostly) Know what they know) While I haven't read the previous works in detail, my understanding is that they attempt to predict the uncertainty from the model that generates it. There is no auxiliary model involved. #### Experiments and Results - Figure 1 shows that perplexity is not indicative of model uncertainty. - However, the open-ended generation setting seems slightly contrived and is not truly open-ended. **What is the average length of the answer in the MMLU dataset?** Since a Multiple Choice Dataset was converted to open-ended generation, the answers might comprise only a few tokens. - The paper proposes three approaches to emit the confidence score. - "Probe," which trains a small feedforward neural network on the last layer - LoRA (same as Probe with LoRA weights) - LoRA + Prompt (same as LoRA with prompt posing it as a multiple choice question (i) and (ii) The dataset used for finetuning was assembled from a diverse collection of benchmark datasets, such as AI2 Reasoning Challenge, Boolean Questions, CosmosQA, etc. I looked at the MMLU paper and found no evidence of any data contamination, i.e., test data accidentally leaking into the training data. I suggest calling this out explicitly in the paper. - Table 1 shows regularization improves calibration. - Figure 2 shows that the three proposed approaches improve calibration and selective prediction. However, since the open-ended generation setting is not truly open-ended (see my comment above), I suggest conducting additional experiments to predict the uncertainties of longer sequences. This could be one reason why the findings are contradictory to previous papers. - Regarding my comment about training an auxiliary model, the confidence generation model is entirely detached from the primary model, which generates the answer. Based on the result of the transferability of these scores, you can arbitrarily mix and match any open-weight model with any other model to get reliable uncertainty estimates. This suggests that this auxiliary model has learned a notion of "correctness" independent of the original model from the training data. It's unlikely that this technique would work on niche domains like finance and medicine since the training data is unlikely to contain such examples. - Line 253: Humanities and social sciences overlap, so this claim is not compelling. - Question Figure 3(b): I see that a tuned model leads to a more uniform distribution of probability scores in the first figure compared to the zero-shot which has a mode around 70%. Why is this calibrated? (Line 279) Shouldn't you use a calibration curve to claim if scores are calibrated? - The experiment presented in section 6.1 is convincing in showing that learned uncertainty estimates depend both on the intrinsic difficulty of the question and the learned correspondence between the question and the correct answer. This also corroborates my earlier skepticism about using this technique to niche unseen domains. I like Section 7 a lot. Thank you for this study on how uncertainty estimates correlate with user decision-making. ### Nits and suggested improvements: - Calling out contributions of the paper more explicitly towards the beginning of the paper. - Line 39-40: "In particular, we consider whether it's possible to have good uncertainties over correctness (rather than tokens) out of the box." - consider rephrasing this; this sentence is unclear. - Line 218: "The probability of each cluster becomes the probability assigned to each sequence in that," this sentence is unclear. - Line 959 in appendix is incomplete sentence

Strengths

### Strengths The paper is concise yet comprehensive, with clear and accessible writing. The appendix offers thorough explanations of hyperparameter selections, auxiliary techniques, and evaluation metrics. While the results align with expectations, they don't break significant new ground. The literature review provides a solid contextual foundation for the work.

Weaknesses

See comments above

Questions

See comments above

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

See comments above

Summary

This paper investigates fine-tuning on a small dataset of correct and incorrect answers for an uncertainty estimator. Through various experiments, the authors show the trained estimator not only surpasses the previous prompting approach, also has generalizability to other models and subjects. Lastly, the authors conducted a user study to verify that calibrated confidence could improve human-machine collaboration by helping users modulate their decision to use the model.

Strengths

- Uncertainty and calibration to reduce hallucination are currently crucial issues. - This paper provides various experiments and analyses. Moreover, the experiments are soundly conducted and most of the details are presented for reproducing. - The user study is fascinating where users solve MMLU tasks accompanied by a confidence score and predicted answers from an LLM. However, there remains future work to in-depth analyze the results. - This paper is well-written, neatly organized, and well-presented.

Weaknesses

- The evaluation task only considers MMLU, which is highly likely to be contaminated. - In the section “When and Why Do These Estimates Generalize?” - In both Figure 4 left and right, some results (Probe, Incorrect, sBERT) seem to have near AUROC of 0.5, which is random. Therefore, it’s hard to agree with the result drawn in this paper. > Learned uncertainty estimates generalize to new formatting, subject matter, and even the generations of other models. This generalization appears to stem not simply from judging a question’s difficulty based on its subject matter (a short-cut) but also learning the correspondence between questions and correct answers. > - The following paper is related in terms of a fine-tuned confidence estimator. - Ulmer, Dennis, et al. "Calibrating Large Language Models Using Their Generations Only." *arXiv preprint arXiv:2403.05973* (2024). - In the section, “How Should We Use Labeled Examples?” - The architectural details of the “small network” used in probing and others are absent. And, what is the regularization hyperparameter for KLD? What prompt is exactly used for the “LoRA+Prompt” setting?

Questions

- In Figure 1 (right), there are 6 value points of “verbal” and 11 of “zero-shot classifier”. How are they selected? Furthermore, it seems to have insufficient values to draw the trend line. - In Figure 1 (left), isn’t AUROC sensitive to the scale of perplexity range?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors adequately addressed the limitations.