HonestLLM: Toward an Honest and Helpful Large Language Model

Summary

The paper presents an approach to ensure that LLMs are helpful and honest. The paper curates and releases a dataset that can be used to assess the LLMs honesty and helpfulness. The paper’s evaluation demonstrates that the proposed approach can improve the LLMs helpfulness and honesty by 65% for Llama3-8b and 124% in Mistral-7b.

Strengths

- The paper focuses on an important and timely problem that affects large language models - The paper curates and makes available the HoneSET dataset that can assist future research in assessing and improving the helpfulness and honesty of large language models - The paper undertakes an in-depth evaluation of the proposed method, demonstrating its merits on multiple large language models

Weaknesses

- The paper includes a 3d pie chart that significantly hurts the readability of the paper. I suggest to the authors to change the type of plot in Figure 2. - It is unclear what the paper means by human experts when presenting the creation of the HoneSET dataset. I suggest to the authors to provide more details on the expertise of these humans and why they are suitable for constructing the dataset. - The paper does not go into details on how the proposed approach can be used or help in Retrieval Augmented Generation settings (RAG). Also, there is no comparison on how the proposed approach compares to RAG in terms of honesty. The main idea of RAG is that it improves the model’s honesty, yet, the paper does not consider RAG at all, which in my opinion, is a major limitation that is not acknowledged by the paper.

Questions

1. What do you mean by human experts in the creation of the dataset? Experts in what field? 2. How does the proposed approach compares to RAG settings? Also, can the proposed approach help improving LLMs honesty in RAG settings?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper presents some of the limitations of the work. I do not anticipate any negative societal impact arising from this work.

Summary

In this paper, the authors proposed methods for improving the helpfulness of LLMs while preserving their honesty. To this end, the authors proposed a training-free and a fine-tuning-based method. The main contributions of this paper is the redefinition of Honesty and the proposed improvement methods.

Strengths

1. A new definition of Honesty of LLMs which is more practical and data-agnostic.

Weaknesses

1. As the construction of the dataset requires human validation and there are 7 human experts. There is no statistical indicator such as agreement provided in the paper. Besides, the proportion of each category lacks more justification. Why is the Latest inf. has almost twice of the queries compared with other categories? What is the reason for building an unbalanced dataset? 2. The fine-tuning process may have a negative influence on the safety standard of LLMs. However, this is not studied in the paper.

Questions

1. the proposed curiosity-driven prompt is fixed for each input or is adaptive for different input? 2. The authors should give more justifications to the construction of D1 and D2 as this is important for obtaining a helpful yet honesty LLM. 3. 1000 pairs seems to be enough for obtaining significant improvement on Honesty. Do the authors consider to analyze the influence of data size on the honesty performance?

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

1. The authors did not study the influence of stage one and stage fine-tuning.

Summary

This paper presents a method to simultaneously enhance the honesty and helpfulness of large language models. The authors start by constructing an evaluation dataset of about 1,000 questions named HONESET. Two types of approaches are proposed: one based on prompt engineering combined with multiple model invocations, and the other based on DPO training. The DPO training employs a two-stage process aimed at separately improving honesty and helpfulness. Experiments conducted on HONESET demonstrate the effectiveness of both methods.

Strengths

- The paper is well-written with a clear structure. - The proposed methods are clear and easy to understand. - The authors conduct extensive experiments on both open-source and proprietary LLMs using two evaluation protocols, showing overall promising results.

Weaknesses

- The experimental evaluation is solely conducted on the authors’ custom HONESET dataset. It is unclear whether the models' general capabilities are compromised under the two proposed methods. It would be beneficial to include standard benchmarks such as mtbench to observe changes in general metrics. - There is no ablation study on the necessity of the two-stage training process in DPO.

Questions

Please refer to the weaknesses mentioned above.

Rating

5

Confidence

3

Soundness

3

Presentation

4

Contribution

2

Limitations

Yes.

Summary

The authors develop a fine-grained dataset and metric for measuring honesty and helpfulness tradeoffs, that consider specific honesty failure modes and demonstrate prompting and training based techniques to improve along this metric

Strengths

Significance: Honeset is potentially another useful contribution to honesty benchmarks, that is more nuanced and fine-grained than say truthfulQA (I need a data quality reviewer to verify this more thoroughly though) H2 assessment is potentially a useful and novel new metric for honesty, good to have diversity there. Quality: the breakdown of honesty 'dimensions' is fairly detailed and nuanced, and seems to be more thorough than any such thinking in the field. However I'm concerned the 'common failure modes' identified may change as models change, so this analysis/dataset has risk of becoming outdated fairly quickly - Dataset construction methodology seems well thought out, though I'm no expert on this matter Clarity: Writing and structure is mostly clear and easy to skim. I appreciate the use of concrete prompt examples Originality: Not groundbreaking given it's just a combination of known steering techniques, but the thoughtfulness put into assembling the new techniques is perhaps better than the existing work in the field

Weaknesses

Significance: - Though the H2 metric and honeset are somewhat better thought-out and fine-grained than most honesty metrics I'm aware of, it is still only a marginal improvement on metrics and datasets for evaluating/improving one aspect of model desiderata. Though this is a net positive contribution to the field, it seems like a relatively minor one to me (i.e. is less impressive than say a paper introducing novel techniques/breakthroughs) - I could see some LLM users finding the honesty/helpfulness desiderata to be overly specified as well, and may have a different vision of the maximally honest and helpful answer (though seems fairly easy to just swap out the prompt to fit their vision) or value different things in a honesty metric (eg maybe they care about explanation/some other aspect of honesty only, but not solutions/guidance, such as in a context where less tokens generated is desirable. or they are concerned by some other failure mode not well categorised by the 6 you identified). - But given the main contribution of the paper (IMO) is providing a fine-grained breakdown of what honest and helpful model outputs concretely look like (and implementing a eval pipeline from this), I could see this paper not being useful for someone with a different operationalisation of honesty/helpfulness - assessment of whether the proposed techniques affect performance/accuracy other than helpfulness measured by H2 would be important if the proposed honesty-enhancing techniques are used commercially (though the authors admit this limitation) Quality: - Unclear how responses are classified as honest/dishonest for calculating honesty rate. Don't think this is mentioned at all in the paper? - A baseline for how the model does just by prompting it to avoid the 6 concrete failure models (maybe with examples) seems much needed. The honesty training isn't worth it if it doesn't beat pure prompting. A compute/time comparison of pure prompting vs (though maybe the two-stage curiosity driven prompting is still worth it, but I'd like to still see a comparison with zero/few-shot, single stage prompting) - Probably should compare your results with existing/accepted honesty benchmarks (such as truthfulQA, but not sure if there's a best practice/consensus for honesty evaluation). Seems a little suspect if you only evaluate the techniques/dataset you develop only with metrics you decided on, as there's some potential for cherry picking/gaming. Clarity: - More detailed prompt examples + examples of honesty failures before and after your technique seem much needed (beyond the brief examples given) - Unclear what 1∼3 (Poor) 4∼6 (Medium) 7∼10 (Excellent) means in table - It took me a long time to figure out what the labels in table 2 like "Lat. Inf." are short for. Try to make this clearer that it's the 6 honesty dimensions.

Questions

Unclear how responses are classified as honest/dishonest for calculating honesty rate. Don't think this is mentioned at all in the paper?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Seems adequate, though I'd add the concerns raised in the weaknesses section