Do LLMs estimate uncertainty well in instruction-following?

Thank you for your thoughtful review and for recognizing the importance and timeliness of our work in addressing the uncertainty estimation challenges faced by current LLMs. We deeply appreciate your acknowledgment of our contributions to the community and your recognition of the potential in leveraging internal states to enhance uncertainty estimation in LLMs. --- ### W. Suggestion on future work > *The poor uncertainty estimation by modern LLM is also not surprising because of its strong alliance with MLE methodology instead of MAP. Therefore, besides future directions outlined by authors, I strongly suggest trying more inference methods other than greedy, e.g. setting temperature, beam search, etc. These are also tunable parameters to closed-source models. This way you can measure closed models although the insights will be limited.* > Thank you for this insightful suggestion. Exploring different sampling strategies at inference time, such as temperature adjustment and beam search, could indeed reveal further findings in LLMs’ uncertainty estimation abilities. While our current work focuses on greedy sampling for consistency across evaluations, extending the evaluation to include various sampling methods presents an exciting direction for future work, especially for closed-source models. ### Q. Suggestion on future work > *Besides trying more inference method, please also consider impact of regularization such as dropout rate if you plan to do any pre- or post-training works. This reveals another problem which is training details are not sufficiently exposed even for open LLMs.* > The impact of regularization, such as dropout rates, is another intriguing direction to consider. We agree that training details, including regularization techniques, can significantly influence LLMs’ uncertainty estimation capabilities. Thank you for highlighting these important points, which could guide future work. --- Thank you once again for your thoughtful feedback and for suggesting valuable directions to expand our work. We greatly appreciate your recognition of the significance of our study and the potential it holds for advancing uncertainty estimation in LLMs.

Thank you for your review and for acknowledging that our work provides crucial insights into the LLMs’ uncertainty estimation in instruction-following tasks. --- ### W1. Dataset scope > *The paper relies on a single IFEval dataset for its initial evaluation. It would be beneficial to include additional datasets to validate the findings across different contexts and domains.* > Thank you for your comments. Please refer to the general response section for a more detailed explanation, where we address these aspects and discuss our rationale for the current benchmark design and its applicability to future datasets. ### W2 & Q > *The paper raises many novel concepts and findings, but it does not seem to provide much direct help in enhancing instruction-following capabilities.* > First of all, we would like to clarify that our focus is specifically on the “uncertainty estimation” capabilities of LLMs within instruction-following tasks, rather than directly improving instruction-following performance itself. While the two abilities are related, they address different aspects of model behavior. For instance, if the instruction is, “Please do not use punctuation in your response,” a model may struggle to comply with this if it conflicts with what it has learned in training — it is very unlikely to not include punctuation in training dataset. However, the model could still recognize and assess whether it has correctly followed the instruction, offering an uncertainty estimate on its compliance. In this case, the model demonstrates **low instruction-following ability** but **high uncertainty estimation ability.** To highlight our contribution once again in the uncertainty estimation side, our findings shed light on the strengths and limitations of widely used uncertainty estimation methods, potentially paving the way for improvements in how LLMs gauge their own uncertainty. For example, discovering that all methods struggle with subtle instruction-following errors highlights inherent limitations in LLMs. We hope this work will inspire further research to enhance LLMs’ ability to estimate uncertainty, ultimately leading to more reliable AI systems. ### W3. Concerns on models > *The article did not use LLMs larger than 13B in its tests, so the conclusions may not be sufficient.* > Due to computational constraints, we were unable to test models larger than 13B. We would like to share that there are many previous studies on trustworthy LLMs also focus on models up to 7B or 13B due to similar limitations [1, 2]. Instead, we included a variety of LLMs (e.g., Mistral, Phi, LLaMA) to capture a range of model architectures and configurations. We hope our benchmark dataset will facilitate further research, including evaluations of larger models in future studies. [1] Liu, Linyu, et al. "Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach." *arXiv preprint arXiv:2404.15993* (2024). [2] Li, Kenneth, et al. "Inference-time intervention: Eliciting truthful answers from a language model." *Advances in Neural Information Processing Systems* 36 (2024). --- Thank you for your review and for acknowledging the insights our work provides into the uncertainty estimation capabilities of LLMs in instruction-following tasks. We hope our detailed responses clarify the focus and contributions of our study.

Thank you for the positive and constructive feedback. We deeply appreciate your recognition of our contributions, particularly the development of new benchmark datasets that address critical gaps in uncertainty estimation for instruction-following tasks. We are pleased that you found the methodologies rigorous, the experimental setups comprehensive, and the writing and presentation clear. Your acknowledgment of our findings—especially the potential of leveraging models’ internal states for improved uncertainty estimation—motivates us to continue advancing this area. Thank you for highlighting these key aspects of our study. --- ### W1. Dataset scope > *the analysis might benefit from extending the scope of instruction types and domains included in the benchmark to cover more diverse real-world tasks* > Thank you for your thoughtful comments. Please refer to the general response section for a more detailed explanation, where we address these aspects and discuss our rationale for the current benchmark design and its applicability to future datasets. ### W3. GPT-4 evaluator > *the use of GPT-4 for task quality assessment introduces a potential risk of pre-training overlap affecting the evaluation, though this is acknowledged as a limitation* > Thank you for highlighting this point. The potential bias from using GPT-4 is indeed a valid concern. To address this limitation, we plan to investigate multiple large models as evaluators in future work, which would reduce the risk of bias that may arise from relying on a single model’s perspective. You can refer to Zeng et al. (2023) [1] for a detailed evaluation of the reliability of the GPT-4 evaluator. [1] Zeng, Zhiyuan, et al. "Evaluating large language models at evaluating instruction following." *arXiv preprint arXiv:2310.07641* (2023). ### Q1. Suggestion on future work > *Can the authors elaborate on how training procedures like RLHF can affect the models' performance in uncertainty estimation, especially in Controlled-Hard tasks?* > This is a thought-provoking question. Comparing models before and after reinforcement learning from human feedback (RLHF), specifically comparing models such as LLaMA2-7B and LLaMA2-chat-7B, seems very interesting and worth exploring. This will help clarify how RLHF might impact uncertainty estimation, especially under challenging Controlled-Hard settings, by influencing model confidence calibration and response tendencies. Thank you again for your suggestion. ### Q2. Suggestion on future work > *Are there potential methods beyond linear probing that could better utilize internal model states for uncertainty estimation in nuanced tasks?* > This is an interesting suggestion. We agree that more complex models could potentially capture nonlinear patterns in the representation space, providing a richer uncertainty signal. However, a critical consideration here is the need for a sufficiently large number of high-quality, objectively verifiable instruction-following datasets to avoid overfitting. We hope that as these datasets become available, future work can explore such models more robustly. ### Q3. Suggestion on future work > *Would expanding the evaluation to include LLM agents provide more comprehensive insights?* > This is another valuable idea. If we understand you correctly, evaluating LLM agents specifically fine-tuned or in-context trained for instruction-following tasks in particular domains would allow us to investigate whether domain-specific training impacts uncertainty estimation. Such training might improve uncertainty estimation ability, or lead to overconfidence in uncertainty estimation, potentially affecting reliability in nuanced tasks. We hope to explore this direction in future work and thank you for the suggestion. --- Thank you again for your thoughtful and constructive feedback. Your detailed suggestions and acknowledgment of our contributions reinforce the significance of our work and provide valuable guidance for future research directions.

Thank you for your constructive feedback and for acknowledging our systematic approach in devising a controlled evaluation framework. By isolating multiple factors that impact uncertainty estimation in naturally generated responses, we aim to provide a clearer and more fine-grained assessment of each method’s performance. --- ### W1. Clarification >*The Probe method is mentioned in Table 4 and highlighted as the best-performing method. However, it is not introduced or discussed in Section 3, leading to some confusion when comparing results in Section 3.2.1. A clearer explanation of this method earlier in the paper would improve the comprehensibility of the findings. Or at least keep the scope of methods consistent when discussing "best-performing"* Thank you for this suggestion. We agree that introducing the Probe method earlier in the paper would clarify the findings. In our revision, we will integrate Section 4 into Section 3 to provide a consistent explanation of all methods, including Probe, for improved clarity in comparisons. ### W2. About dataset scope > *As noted by the authors in their limitations section, the types of instructions included in the benchmark may not comprehensively reflect real-world tasks. Additionally, the dataset size is relatively small, potentially impacting the reliability of the benchmark.* > Thank you for your thoughtful feedback. For a more detailed explanation, please refer to the general response section, where we address these aspects and discuss our rationale for the current benchmark design and its applicability to future datasets. ### Q1. Details in benchmark data generation process > *Could you please provide more details in the benchmark data generation process? For example, what’s the prompt for generating a correct response in the controlled version and natural responses in the realistic version? What are the domains for the arguments (e.g., Type, Kwargs) in the prompts? Are these parameters inherited from IFEval?* > We really appreciate your question in the detailed benchmark generation process. Yes, the argument structures in our prompts (such as Type and Kwargs) are inherited from the IFEval dataset. Type specifies the instruction type (e.g., ‘keywords:forbidden_words’, etc.), while Kwargs details task-specific parameters (e.g.,{”forbidden_words”:[”night”, “happy”, …]}). Full versions of these prompts are available in the Appendix. We will clarify this in the revision. ### Q2. Normalization of token length > *It seems that all the token length distributions are not normalized by the total number of responses in each class (Figure 2, 3, and 10). This may mean that incorrect responses appear to be longer simply because of different class sizes. Normalizing the distributions would help, even if it may not change the conclusions about length biases.* > Thank you for noting this. We agree that normalizing token length distributions would clarify our conclusions. We have since normalized these distributions and found that, as expected, the conclusions remain consistent. We will include these normalized plots in the revised submission. --- Thank you again for recognizing the importance of our contributions and for your thoughtful suggestions to improve the paper.