Teaching Language Models to Hallucinate Less with Synthetic Tasks

Response to Reviewer tStg

Thanks for your review and comments! We’re glad you found SynTra a valuable tool, and respond to your questions below. --- _Important insights, such as the significant reduction in grounded entities generated by Orca and Vicuna (by 66% and 99% respectively) when using SYNTRA without reference data, are buried within tables and figures_ Sorry for the confusion; **the 66% and 99% reduction in entities are for the fine-tuned model, not the system message**: the numbers for the system message are (1.4% and 14.8% respectively). We also find that this reduction is small for the Llama 2 models, (e.g., the reduction for Llama 2 7B is only 6.2%). Moreover, this reduction is consistently less than the reduction in ungrounded entities, which are what we aim to remove. We hope this assuages your concern, and if it changes your impression of our results that you’ll consider increasing your score. --- _The evaluation of SYNTRA is somewhat limited, as it focuses on only 2 models and 3 realistic tasks. This raises concerns about the method's ability to generalize across diverse models and tasks._ Thanks for raising this point; based on this feedback, we ran SynTra on Llama 2 7B chat and Llama 2 13B chat, both with RLHF. We find that SynTra manages to reduce hallucination in both models under these datasets in a comparable way to the existing models. We include full results in Appendix B.8, but find substantial improvements; for example, SynTra doubles the BLEU score for MS MARCO 13B. --- _The authors mentioned at the end of the Related Work that the direct optimization of a discrete prompt might be viable in SYNTRA as well, can they give more discussions about the pros and cons compared to SYNTRA? Will it be more difficult in terms of the optimization?_ Yes, doing the discrete optimization is likely harder (e.g., see https://arxiv.org/abs/2306.15447 which shows that even the best discrete optimizers over tokens struggle), but may produce more in-distribution system messages that generalize more. We have added a note about this to the discussion. --- _I find the axis label "Hallucination rate" misleading in Figure 2 - I think the number presented therein should instead be 1-hallucination rate? or did I miss anything?_ Yes you are right; thanks for raising this! We’ve updated the figure. --- _How is the convergence of the optimization? Is there any constraint on the postfix such as any constraint on the matrix?_ The optimization converges; we’ve added plots in Appendix B.8. There is no constraint on the postfix. --- _Can SYNTRA be added on top of other classes of methods mentioned in section 2 to further reduce hallucination?_ This is possible; SynTra already reduces hallucination on top of instruction tuning (for Vicuna and Orca), which already reduced hallucination. We believe this is an exciting direction for subsequent work. --- _From Figure 2 and Table 1, it looks like when LLM weights are fine-tuned, they are overly optimized for the synthetic task and as such it might ends up hallucinate more on realistic tasks. Is there any possibility that an early-stopping of the fine-tuning process might actually help in finding better parameters for both synthetic and realistic tasks?_ This is possible; there are ways to both improve fine-tuning and postfix-tuning. Nevertheless, the broader paradigm of supervising on known tasks then transferring is a promising approach, and developing better ways to transfer (e.g., other fine-tuning methods or beyond) are promising future directions.

Thoughts on our rebuttal

Dear Reviewer tStg, Thanks again for your review! As we near the end of the discussion period, we wanted to ask if we addressed questions you raised, and especially if our clarifications about the entity removal change your opinion of the paper. We hope you find our responses useful and would love to engage with you further if there are any remaining points. We understand that the discussion period is short, and we sincerely appreciate your time and help!

Thank the authors for the clarifications

>That makes sense, thanks for clarifying this! The 66% and 99% numbers are a baseline within a baseline: they come from training on just the names data (without mixed in reference) using fine-tuning (instead of tuning the system message). The numbers with just fine-tuning are 12.2% and 17.6% for Orca and Vicuna respectively (Table 3). These are nonzero, but fine-tuning reduces the number of "ungrounded" entities by much more: 36.5% and 43.2% respectively, which might be an ok tradeoff for practitioners. We're space constrained with the edits we can make in the rebuttal, but will add a discussion about how (i) fine-tuning lowers the entity counts by more (i.e., beyond the "Fine-tuning and training without the reference data eliminate more grounded and ungrounded entities" that is already there) and (ii) fine-tuning without reference data can be degenerate to the discussion in 4.3. We can also preview this in the appendix if it's helpful, and overall think this largely enhances the message by adding additional evidence that optimizing the system message transfers better. Thank you for providing additional explanations. They have given me a clearer understanding of the problem. Yes, I agree that this paper would greatly benefit from more in-depth discussions, as pointed out by the authors. It would be beneficial if the authors could include such discussions in the appendix, at least, to make this paper more complete. > Yes, there is no result analogous to Table 1: we mention this in the response to reviewers (but should've also propagated it here). This is due to quota limits on GPT-4; we can likely get results for the smaller ACI-Bench dataset if it's helpful by tomorrow, but probably cannot get results for the two larger datasets. Both of the Llama-2 and Vicuna experiments use default hyperparameters from Orca without any adjustment, which is part of the reason the Orca gap is larger (see end of Section 4.2). I understand the computational difficulties and do not wish to add to the burden. However, if feasible, please consider including the experiment again after the rebuttal period for the paper's completeness. Finally, to acknowledge authors' effort in including the additional experimental evidence, I have raised my score to 6.

Response to Reviewer C5mJ

Thanks for your review and comments! We’re glad you appreciated how our method is transferable and novel. It seems like your main concern is that the reduction in hallucination rate is due to the ShareGPT / original LLM fine-tuning set, rather than SynTra. **We think this is a misunderstanding: we compare SynTra trained on top of the fine-tuned models to the fine-tuned models themselves, not to Llama**. In particular, we show that SynTra reduces hallucination _on top of_ any hallucination reduction obtained by the original fine-tuning. This highlights the complementary nature of SynTra to other approaches; SynTra further reduces hallucination after fine-tuning. We updated all tables to say e.g., “Original Orca”, instead of “Original Model” to emphasize this and hope that if this and our other responses help resolve your concerns, you’ll consider increasing your score. We address your additional questions below: --- _Comparison with the code datasets: previous work [1] has found that training the model on code datasets is beneficial for reasoning. I wonder how the quality of the SynTra task compared to directly fine-tuning the model on the code dataset?_ We haven’t tried fine-tuning on code datasets; it’s possible this would help, although reasoning ability may not directly reduce hallucination. However, we expect that SynTra may be complementary to this approach; SynTra reduces hallucination further after fine-tuning on high-quality GPT-4 generated data for Orca, suggesting it improves performance even on fine-tuned models. --- _The transfer from synthetic to downstream tasks could be brittle: Sometimes full fine-tuning is better than soft-prompt tuning in system messages, but sometimes it's not. Sometimes reference data regularization is helping, sometimes it's not._ This is true; for specific datasets some approaches work better than others (e.g., training without the reference data is better on MS MARCO). We think this is a byproduct of transferring the same model weights / system message across multiple diverse tasks, and note that on average, training on the system message + regularization data is best for both models. --- _In figure 2, the y-axis says hallucination rate (the lower the better), but according to the passages, the y-axis should be the accuracy (the higher is better). Please clarify this._ Thanks for bringing this up! We updated the figure accordingly. --- _Citation error: (yew Lin & Rey, 2004) -> (Lin & Rey, 2004)_ Fixed, thank you!

Thoughts on our rebuttal

Dear Reviewer C5mJ, Thanks again for your review! As we near the end of the discussion period, we wanted to ask if we addressed questions you raised, and if our clarifications about the experimental setup change your opinion of the paper. We hope you find our responses useful and would love to engage with you further if there are any remaining points. We understand that the discussion period is short, and we sincerely appreciate your time and help!

Thanks for your response

Thanks so much for getting back to us and for your thoughtful comment! It is indeed probably also the case that the ShareGPT / Orca fine-tuning reduces a lot of hallucinations (and instruction tuning in general is probably required for these tasks, which require cohesive long-form outputs). We didn't run the hallucination evaluation on llama 1 because it seems to struggle with our tasks without instruction tuning, but think exploring the comparative benefits of synthetic and real data on top of the instruction tuned models is interesting, and exciting for future work. The last paragraph in our discussion section discusses the tradeoffs between synthetic data and demonstrations, and we've updated it based on your feedback (see new revision). Thanks again for engaging in the rebuttal, and please reach out if any other questions come up!

Response to Reviewer MYvf

Thank you for your feedback on our work! We appreciate that you liked our high level approach and our experimental design, and thought that circumventing human preferences / demonstrations was promising. You highlighted two places to expand the experiments: we only experimented with one model size, and it would be nice to compare to other baselines (e.g., running on Llama 2). Based on this feedback, we **ran SynTra on Llama 2 7B chat and Llama 2 13B chat, both with RLHF**. We find the following: * Prior to using SynTra, both Llama 2 chat models hallucinate more (under the automated metrics), than the non-RLHF Vicuna and Orca models. * SynTra manages to **reduce hallucination** in both models under these datasets in a comparable way to the existing models. We include full results in Appendix B.8, but find substantial improvements; for example, SynTra doubles the BLEU score for MS MARCO 13B, and always reduces the number of ungrounded entities on ACI-Bench by substantially more than the grounded entities. We respond to your additional questions below. --- _The authors only experiments with one model size (13B) and it’s hard to say how well their method scales. Will be benefits of SynTra increase or decrease for finetuned LLaMA 30B or 65B? What about LLaMA 6.5B?_ SynTra performs comparably on Llama 2 7B chat and Llama 2 13B chat. We do think that SynTra provides an interesting alternative approach to human feedback, in the limit where human feedback becomes harder to elicit (e.g., in higher quality outputs, humans may increasingly struggle to identify hallucinations). --- _Relatedly, it would be good to compare SynTra to some simpler baselines, e.g. finetuning on gold labels for some other summarization dataset (e.g. OpenAI’s cleaned Reddit data). I’d also be curious to see how RLHF’d models (e.g. LLaMA-2-chat series) do on those benchmarks._ On the automated metrics, Llama-2-chat seems to perform _worse than both Vicuna and Orca_, although SynTra manages to regain some of the lost performance. We hypothesize that this might be a negative side-effect of the RLHF; humans may prefer responses that embellish slightly, or add ungrounded content in order to be helpful. We see evidence of this in the new ACI-bench entity evaluation of Llama 2; Llama 2 chat models tend to generate far more entities than Orca and Vicuna (Appendix B.8). We think exploring this further would be interesting for subsequent work. In terms of fine-tuning on gold summarization labels, we think this is out of scope; these labels require expensive human annotations that SynTra seeks to avoid. Nevertheless, we hypothesize that these approaches are complementary. SynTra already demonstrates improvements on top of models that have been instruction tuned with high-quality data (e.g., Orca is fine-tuned on process-based feedback generated by GPT-4), and we expect that it could further improve performance even after models have been fine-tuned on other gold data. --- _The absolute improvements in hallucination rate (Table 1) don’t strike me as very high. I’m not sure they justify the software complexity of soft system message optimization._ For some of the models and datasets the change in hallucination rate isn’t high, but for others it is: for example, in ACI-Bench, SynTra reduces the hallucination rate from 47 -> 29, and on Vicuna from 50 -> 42. We also think there are other ways of improving SynTra in practice; while we opt for generality by making a single system message that generalizes across different realistic tasks, practitioners could design synthetic tasks that are “closer” to a specific practical task they want to transfer to, or hyperparameter tune further. We believe this is an exciting direction for subsequent work. --- _It’s confusing that “hallucination rate” in Fig. 2 is actually non-hallucination rate (higher is better) while for “hallucination rate” in Table 1 lower is better._ Updated the Figure: thanks for raising this! --- _It was a mildly surprising finding for me that soft system message optimisation is so much better than supervised finetuning. Have you tried manipulating the $\alpha$ hyperparameter to increase regularisation for supervised finetuning? I could imagine this being helpful_ This is a good question; we only test two values for $\alpha$ (either 1 and 0.5), but it is possible that this, or other interventions, might improve fine-tuning (just as similarly, regularization + optimization \alpha could help SynTra). Intuitively, we expect that the system message helps because it optimizes over “high-level instructions’ that are likely to transfer, instead of actually changing the model mechanism. However, we note that better fine-tuning might help (see second paragraph of section 5), and think this is an interesting direction for subsequent work. --- _Is the KL in eq. 2 forward (KL(base, new)) or reverse (KL(new, base))?_ It should be KL(base, new) — thanks for catching this.

Thoughts on our rebuttal

Dear Reviewer MYvf, Thanks again for your review! As we near the end of the discussion period, we wanted to ask if we addressed questions you raised, and if our additional experiments help assuage some of your concerns. We hope you find our responses useful and would love to engage with you further if there are any remaining points. We understand that the discussion period is short, and we sincerely appreciate your time and help!