FLAME: Factuality-Aware Alignment for Large Language Models

Summary

This work studies how to do alignment for large language models to improve their factuality. The focus of this work is on SFT and DPO. The motivation behind this work is a pilot study which shows more factual data does not always lead to a more factual model. To resolve this issue, the proposed Flame framework (1) handles fact-based and non-fact-based examples differently; (2) uses few-shot generated examples from the model itself for fact-based SFT; (3) builds a reward model specifically for factuality (via atomic fact decomposition, retrieval augmented claim verification, etc.) Experiments on multiple datasets demonstrate that Flame can improve the model's factuality without hurting other capabilities (e.g., instruction following). Ablations are also conducted to measure the gain from each individual step.

Strengths

1. The motivation is clear and reasonable. I like using a simple and quick pilot experiment to demonstrate the main motivation of this paper. 2. The idea is straightforward and effective. The high level framework can applied to many different systems. 3. Ablation experiments are conducted to show the gain from each step. The effectiveness for both SFT and DPO are clear.

Weaknesses

1. No external baselines are used in the comparison. It would be great to compare the flame model with other related approaches (e.g., few-shot prompting, sampling multiple responses, and reranking using FactScore or the reward model). I know these approaches are not directly comparable, however, it will still be valuable to understand the relative trends, especially since approaches such as few-shot prompting are used in data generation. 2. It will be great to conduct human evaluations even just on a few examples. 3. The whole pipeline involves a number of components. While many details are presented in the appendix, low-level details like few-shot prompts, and implementation of fact decomposition are omitted. Adding these details will be super valuable for future work to build similar systems. It would be even better if the authors decide to release the code.

Questions

1. In the pilot experiment, doing DPO with FS seems to work reasonably well. Have you tried similar approaches in the real experiments?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed in Sec. A.6 in the appendix.

Summary

This paper shows that training on new or unfamiliar knowledge can promote hallucination and that reward functions in standard RL often inadequately capture factuality. The authors propose a factuality-aware alignment method that first identifies instructions as fact-based or non-fact-based. For fact-based instructions, they employ adapted techniques in respective SFT and RL to generate additional training data, thereby reducing the hallucination of the model's responses.

Strengths

* The paper conducts a pilot study that highlights the limitations of SFT and RL in capturing factual knowledge. This study provides valuable insights into data selection for LLM alignment training. * The proposed dual-stage factuality-aware method improves factuality without compromising the instruction-following capabilities for both SFT and RL stages.

Weaknesses

* The proposed strategy to create SFT and DPO training data using the generated responses from the LLM itself is limited to the knowledge learned within the original model. This approach may struggle with instructions that the original model cannot generate factual answers for. * The proposed strategy relies on accurately identifying the instruction type initially, which is limited by the model's ability to correctly classify the instruction type. * In the pilot study, it is unclear whether the $PT$ and $PT^{RAG}$ are evaluated using the same protocol as other methods. If they are, the FS score decreases after both SFT and DPO, which contradicts the claim that "fine-tuning LLMs on their own generations appears to be crucial for factual alignment." * While the results in Table 2 and 3 indicate that eliciting knowledge from the model itself can enhance factuality compared to introducing more factual but unknown knowledge, it does not improve the FS of the $PT$, which achieves a score of 53.1 on the Biography task with just 5-shot demonstrations. * As discussed in Sec 5.5, conducting fact checks and computing factuality rewards solely for fact-based sentences can lead to more factuality errors. Clarification is needed on how FS is calculated for the experiments in Sec 5.2 and 5.3.

Questions

Please refer to the Weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Yes, the authors have discussed the limitations of the metric for evaluating factuality.

Summary

This paper addresses the issue of factual inaccuracy, or "hallucination," in Large Language Models (LLMs). The authors identify factors that lead to the generation of false facts during supervised fine-tuning (SFT) and reinforcement learning (RL). They propose FLAME, a novel alignment method that incorporates factuality-aware SFT and direct preference optimization (DPO) to guide LLMs towards more factual responses without compromising their ability to follow instructions. Experiments demonstrate FLAME's effectiveness in enhancing factuality while maintaining helpfulness.

Strengths

1. The ablation experiments provides comprehensive insights into the effectiveness of DPO and SFT in mitigating hallucination. 2. The method proposed in this paper attempts to balance instruction following and factuality. It relies on model self-construction data, and does not depend on external proprietary models.

Weaknesses

1. The baselines compared in this work are limited to different settings of SFT and DPO only. The baselines in the paper should at least include the work [1]. This prior work also uses DPO and algorithms, and the only difference seems to be data construction. The paper should compare with this work to demonstrate that its algorithm truly achieves a balance between instruction following and factuality. 2. In addition to the works listed in the related work, there are some works whose methods are somewhat similar to this paper, such as [2] [3], etc. The paper may need to add explanations of the differences between these methods to clarify its own novelty. [1] Fine-tuning Language Models for Factuality. https://arxiv.org/abs/2311.08401 [2] Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation. https://arxiv.org/abs/2402.09267 [3] GRATH: Gradual Self-Truthifying for Large Language Models. https://arxiv.org/pdf/2401.12292

Questions

1. In comparison to training, have the authors considered comparing representation editing baseline methods? 2. Could the authors supplement experiments on the TruthfulQA-MC in Table 4 to provide a measure of multi-choice performance?

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have addressed some limitations of their work in the Appendix, which is commendable.

Summary

The paper discusses a novel alignment method to enhance the factual accuracy of LLMs. The authors observe that conventional alignment processes, which include SFT and RL, often result in the generation of false facts or 'hallucinations'. To address this, they introduce factuality-aware alignment (FLAME), which includes factuality-aware SFT and RL through direct preference optimization. FLAME identifies factors leading to hallucination and adapts the training process to reduce the generation of false claims. Experiments demonstrate that FLAME guides LLMs to produce more factual responses without compromising their ability to follow instructions. The paper contributes to the field by tackling the issue of maintaining helpfulness while improving the factuality of AI-generated content.

Strengths

- Clear and Logical Structure: This paper is well-organized and presents its findings with a logical flow, making it easy to follow. - In-depth Analysis of Hallucination: The paper thoroughly analyzes the factors contributing to hallucination during the SFT and RL phases of language model alignment. It identifies key issues: training on unfamiliar data can reduce factual accuracy, and standard RL reward functions often prioritize longer, more detailed responses, potentially encouraging the model to fabricate information. - Innovative Solution: The proposed FLAME is a novel alignment approach that effectively addresses hallucination without compromising the model's ability to follow instructions. By extending both SFT and RL, FLAME tackles a critical issue in LLMs, ensuring more accurate and reliable information generation. - Comprehensive Evaluation: The paper thoroughly evaluates FLAME's effectiveness in improving both factuality and instruction-following abilities. Experiments demonstrate that models aligned using FLAME achieve significantly higher FactScore compared to standard alignment methods, without sacrificing their helpfulness.

Weaknesses

This paper is well-written and makes a valuable contribution to the LLM alignments. I only have several minor concerns as follows: - Model Size and Generalizability: The paper focuses solely on the LLaMA2-70B model. It would be beneficial to investigate whether FLAME's effectiveness extends to smaller models, such as 7B or even smaller, given that the factuality-aware SFT relies on self-supervision through few-shot prompting. - Evaluation Metrics and Human Assessment: While FactScore is a valuable metric, it has limitations. It assumes Wikipedia as the definitive source of truth and may not be suitable for broader domains. Using a more comprehensive metric like Veriscore [1] could provide a more nuanced evaluation (I understand that Veriscore is a recently released method, so this is a suggestion for the future version of this paper). Additionally, incorporating human evaluation would strengthen the analysis. A manual assessment of factuality and helpfulness would provide valuable insights and increase the persuasiveness of the findings. - Multi-faceted Evaluation: The paper primarily focuses on instruction following and factuality. However, other crucial aspects of LLM capabilities, including knowledge, reasoning, and code generation, should also be considered. It would be insightful to evaluate the performance of FLAME-trained models on standard benchmarks like MMLU, GSM8K, and HumanEval to assess potential trade-offs in these areas.

Questions

- While FLAME primarily focuses on DPO, can it also be applied to conventional reinforcement learning from human feedback (RLHF) methods like PPO? - Are there plans to release the code and models trained using FLAME for the research community to replicate your methods?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations and broader impacts.

Reply

I thank the authors for their insightful replies to my review. They are indeed helpful. I still have some concerns regarding your evaluation assumption that all the sentences in the responses are fact-based, which might not be entirely fair. The baseline model fine-tuned on high-quality instruction data, as pointed out by [1], might generate more stylistic tokens, such as "Of course." Could fact-checking these sentences decrease the FactScore? The evaluation deserves more rigorous and comprehensive discussion. (Note that Reviewer HVe5 also pointed out the limitation of your evaluation protocol.) Given most of my concerns have been addressed, I would like to raise my score from 4 to 5. [1] [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://arxiv.org/abs/2312.01552)

Reply to the authors

Thank you for your response regarding the additional results. I've read the response in detail and decided to keep my score and judgments.