Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability

Summary

This paper presents a comprehensive study on the relationship between LM hallucinations and their scale. The authors construct a knowledge graph (KG) based dataset to train LMs, creating a controllable experimental environment to study hallucinations. The authors also design several hallucination detectors to study how hallucination detectors depend on scale (e.g., LM size, training epochs, datasets).

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The presentation of the methods and experiments is clear and methodical. And this paper provides a comprehensive understanding of the hallucination problem with the scale. The results and findings presented in the paper are insightful and interesting, for example, this paper reveals a trade-off between the model's ability to generalize and its tendency to hallucinate.

Reasons to reject

1.It will be better if the authors could provide more detail analysis about the causes of hallucinations, such as noise in the dataset (training on KG may limit this setting), etc., may provide richer insights. I also wonder what will happens if training LMs on KG with incorporating textual context. 2.Providing some case studies of hallucination examples on the scale of LMs can help readers to understand the nature of hallucination and the working mechanism of the detector more intuitively. 3.The Related Work section should be included in the main tex, rather than in the Appendix. And this paper mentions some correlations with existing methods but does not provide direct comparisons.

Summary

The paper examines hallucinations in language models trained on knowledge graphs, focusing on cases where correct answers are present in the training data. It reveals that larger models have lower hallucination rates but require more computational power. The study also finds that the detectability of hallucinations decreases as model size increases, suggesting a trade-off between model scale and generalization capabilities.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. The approach of training LMs from scratch using knowledge graphs is innovative and brings a fresh perspective to the study. 2. The paper presents intriguing conclusions that hold significant implications for future research in the field.

Reasons to reject

1. This paper studies the phenomenon of hallucinations, particularly focusing on the task of predicting the object entity in a triplet. However, the research scenario presented in this work is relatively singular, primarily concentrating on the direct prediction of facts, which seems limited compared to the broader types of hallucinations observed in real-world tasks, such as text generation, dialogue systems, summarization, translation, etc. As a result, the conclusions drawn from this study may lack universality and may not adequately reflect the performance of language models across a diverse array of tasks. 2. Some findings in this paper are interesting but potentially insufficient and possibly incorrect. For example, the analysis presented in Figures 2 and 3 indicates that an increase in dataset size leads to a higher rate of hallucinations. However, this might be due to the models not being sufficiently converged when the same amounts of FLOPs are applied across different dataset sizes. To achieve more reliable conclusions, models should be allowed more FLOPs for larger datasets to ensure adequate training and convergence. 3. The manuscript's complexity and lack of clarity in certain experimental descriptions make it difficult to follow. The figures, with their numerous variables, are not clearly articulated, hindering the extraction of useful conclusions. The paper would benefit from a more concise presentation and clearer figure design to enhance readability and comprehension.

Questions to authors

1. Could the authors please elaborate on the necessity of incorporating special tokens during the training phase, and discuss whether the deviation from the natural language text format could potentially influence the conclusions drawn from the study? 2. The results presented in some of the figures, such as Figures 4 and 7, are not as intuitive as they could be, making it challenging to derive clear conclusions. With the presence of multiple variables in these charts, it seems difficult to reach a valid conclusion. The authors should consider revising the presentation to allow for a more straightforward interpretation of the results. 3. In Appendix B, I notice that the authors used different learning rates for models of different sizes. But apparently, the values of these learning rates are very strange, how did the authors get these particular learning rate values (e.g., 0.00047 for the 113M model)?

Summary

This paper investigates the scaling law of language models (LMs) and hallucination. The authors control the train/test facts by constructing a knowledge graph (KG) based dataset and train language models from scratch. They ablate LMs of varying sizes and evaluate their hallucination rates on seen and unseen data. They find that while increasing the LM size and training compute reduces hallucinations on the training set, eliminating hallucinations entirely requires an order of magnitude more compute than currently considered optimal. They also study hallucination detectors and find an inverse relationship between LM scale and the detectability of hallucinations by these detectors.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The use of a controlled KG-based dataset is a novel approach that enables quantifying hallucinations in a precise manner not applicable to previous data. This controlled setup reveals interesting insights. 2. Comprehensive analysis of how hallucination rates depend on LM scale, training compute, and dataset size. 3. Investigation into the scaling properties of hallucination detectors and their ability to catch hallucinations from LMs of different scales. The inverse scaling of hallucination detectability with LM scale is an interesting result with potential ramifications for detector-based mitigation strategies. 4. Clear writing and good visualization of key results.

Reasons to reject

1. Limited to studying hallucinations on factual knowledge present verbatim in the training data (which is not natural language sentences). Other types of hallucinations are not explored. 2. Some discussion on similarities/differences compared to natural language could have strengthened the connection to real-world LM applications. 3. LM and dataset sizes studied are still significantly smaller than state-of-the-art LLMs, raising potential generalizability concerns.

Questions to authors

1. Can the authors comment on how the findings may change when considering more advanced language understanding capabilities needed for hallucinations in natural language? 2. What are some ways the KG-based approach could be extended to study other types of hallucinations beyond factual knowledge verbatim in the training set?

Summary

The paper explores the relationship between LM hallucinations and their scale, focusing on eliminating hallucinations where correct answers are in the training set. To this end, the authors train a set of LMs on a KG-based dataset. The major conclusions include: 1) Larger and longer trained LMs hallucinate less on a fixed dataset, but increasing dataset size can increase hallucination rate. 2) Achieving a low hallucination rate requires LMs much larger than currently considered optimal trained for longer periods, with a trade-off between hallucination rate and other performance metrics. 3) Detectability of LM hallucinations decreases as LM size increases.

Rating

7

Confidence

2

Ethics flag

1

Reasons to accept

* The paper is generally well-organized and clearly written. * The paper explores an important topic with reasonable experiments and analysis. * The conclusions of the paper provide potentially useful knowledge regarding developing practical solutions to reduce LM hallucinations.

Reasons to reject

* My major concern is that the data used for training LMs in this paper is quite different from the data used for LM training in practice. While the paper has a short discussion about it in the limitations section, it would make the paper stronger if the authors could provide empirical evidence to justify the conclusions' generalizability.

Thanks for your response.

Thanks for your response. The replies for Reason 1 and 3 have partially solved my concerns, and I would like to keep my score for the current version. In addition, by "hallucination examples on the scale of LMs" I mean the examples by LMs on different scales (e.g., LM size, training epochs, datasets).