Statistical Knowledge Assessment for Large Language Models

Summary

This paper proposes a statistical approach called KaRR to assess the factual knowledge contained in Generative Language Models (GLMs). The authors use a large-scale assessment suite with 994,123 entities and 600 relations to evaluate 14 GLMs of various sizes. The results show that KaRR exhibits a strong correlation with human assessment and achieves a lower variance to varying prompts. The experiments also reveal interesting insights into the scaling law of GLMs and the impact of tuning on instruction-following data.

Strengths

1. The proposed statistical approach is novel and effective in assessing the factual knowledge contained in GLMs. The proposed method effectively considers different surface forms of subject, object, and relation. 2. The large-scale assessment suite used in the experiments is comprehensive and orders of magnitude larger than prior studies.1. Some of the paper's findings, such as scaling laws of knowledge and fine-tuning instructions to improve consistency, have been found in previous work. 2. Previous work has systematically assessed the consistency of the same facts under multiple prompts, but the facts considered in this paper are larger in scale and prompts are more diverse.

Weaknesses

1. Some of the paper's findings, such as scaling laws of knowledge and fine-tuning instructions to improve consistency, have been found in previous work. 2. Previous work has systematically assessed the consistency of the same facts under multiple prompts, but the facts considered in this paper are larger in scale and prompts are more diverse.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

The paper introduces an automatic evaluation metric to assess the amount of factual knowledge kept by large language models (LLMs). This proposed metric considers various surface forms of factual knowledge presentation, allowing for an evaluation that not only measures the accuracy of the models in terms of factual knowledge but also considers the prediction robustness of the knowledge. Later, the authors demonstrated that this metric has a higher correlation with human annotations than previously proposed metrics. Additionally, the paper includes several robustness analyses for the metric, such as examining the impact of the prompting format and its relationship to co-occurrence statistics. And it shows the new metric's effectiveness compared to previous metrics.

Strengths

1. The proposed metric is commendable as it goes beyond capturing mere accuracy and considers model performance consistency. This aspect is crucial in evaluating the factual knowledge of large language models, as it reflects their ability to provide correct information consistently. 2. The strong invariance results achieved by the automatic evaluation metric compared to other baselines are a notable strength. 3. The paper's comprehensive analysis of the proposed metric is great. By covering multiple aspects, the authors thoroughly evaluate the metric's performance. This level of analysis contributes to a better understanding of the metric's strengths, limitations, and overall effectiveness in assessing the factual knowledge of large language models.

Weaknesses

1. The human correlation score of 0.43 for the automatic evaluation metric may be considered relatively low. 2. The proposed metric is not interpretable. To improve the interpretability of the score, calculating an oracle score and then comparing the current score to the oracle score ratio could be a useful approach. This ratio can aid in better understanding the effectiveness of the metric and provide a clearer picture of its performance relative to the best achievable outcome.

Questions

1. In The KaRR scoring details section in Appendix, you mentioned the selection of a threshold based on human judgment alignment. It would be valuable to explore the impact of different threshold values on the metric's performance and investigate the generalizability of the chosen threshold on other sets of knowledge. Does certain threshold value also result in high correlation in other datasets? 2. Minor suggestion: compress Figure 5 by representing the average performance of the best and worst relations. This modification could improve the readability of the figure and make it easier to interpret the performance trends. 3. Adding the best and worst relations to the appendix would help understand the model performance. I would be curious to see it personally. 4. There is a missing number in the equation mentioned in line 43 of the appendix.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No, the limitation of the work is not discussed.

Summary

The paper proposes KaRR, a statistical approach to assess factual knowledge for generative language models based on graphical models. An assessment suite is also proposed for future research. Experiments are conducted with 14 popular large language models and comprehensive analyses are also conducted to reveal related properties.

Strengths

- This paper released a dataset for assessing factual knowledge in generative language models, which is large-scale (millions of entities and text aliases) and could be used in future works. - This paper proposed a statistical score KaRR based on graphical models and KaRR aligns well with human preferences as shown with human evaluation. - Lots of popular large language models are experimented with and the analyses are interesting, e.g., Table 2(b).

Weaknesses

- The knowledge assessment focuses on entity-aware knowledge, which could be a relatively limited knowledge form. The current LLMs are good at identifying entity-aware knowledge. The major hallucinations actually come from numbers, dates, etc. Not a strict weakness, just wondering if KaRR could be extended to these aspects. - There are in total 600 relations in the assessment suite. Does that mean the suite can only be employed for the specific 600 relations? What if there are new relations?

Questions

- The conclusion of "instruction tuning impairs knowledge reliability" should be considered more carefully (line 236). There are at least two factors contributing to the KaRR difference between Alpaca and Vicuna, data quality and tuning style. You do not know exactly whether data or tuning style is the major reason. - Minor comments: - line 228, repeated "KaRR score". - line 230, should be "with a 3.92 KaRR score difference" - GLM in Table 2 and GLM across the paper are a little bit confusing. - What are the distributions of relations in the dataset?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Refer to previous weaknesses and questions.

Summary

this paper proposes a statistical method to probe the knowledge in generative language models, which aims at connecting symbolic knowledge and GLM's text format generation. More specifically, the KaRR comprises two components with regard to specifying relation and subject entity. The authors also present the graphical model for model implementation on the text. The results show that the knowledge in GLMs follows the scaling law, but when the model is finetuned on instruction-following data, it may compromise the model's ability to consistently generate factually correct text.

Strengths

1. very important and interesting problem, assessing the knowledge stored in generative language models is challenging and worth studying 2. the proposed KaRR method shows strong robustness to prompt variance 3. Interesting findings: instruction-following data compromises the model’s capability to generate factually correct answer.

Weaknesses

1. This work only focuses on generating the object, I believe the author can assess the model's ability to generate the subject (by reversing the triples/facts) to gain a more comprehensive assessment of knowledge stored in GLMs. 2. The average alias count for each subject is approx 1.39, and for each object it is about 1.78, which I believe could not support the "diverse" claim. 3. The proposed KaRR’s design to measure reliability is not clear enough. 4. In line 38, the authors claim "Prior methods are designed for masked language models (MLMs) and are incapable of measuring GLMs.". However, to the best of my knowledge, there are several works about assessing the knowledge in GLMs, such as [1, 2]. [1] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. [2] Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022). Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10, 257-273.

Questions

1. I believe it would be interesting to see the model's knowledge across different domains, such as encyclopedia, biomedical, etc. 2. Since high-quality aliases are often difficult to obtain, it would be interesting to see the impact of alias, such as alias count, etc.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

The paper presents a framework for the quantitative assessment of the knowledge captured by large language models, which includes a proposed metric and a large set of relations. Given subject-relation-object triplets, a basic approach to quantify the model's knowledge would be to use the probability of the object entity being generated, given the subject and the relation. This naive approach has shortcomings, which the authors address in the following way: 1. Entities can have multiple synonyms, which are important to consider to accurately assess consistency. The authors therefore augment their proposed evaluation suite with a set of aliases for each entity, extracted from Wikidata, and adapt the metric to take these into account. 2. As this kind of probing often suffers from spurious correlations, the authors propose the "knowledge assessment risk ratio" metric, which also takes into account the expected generation probability when either the subject or the relation entity are not specified. The authors thoroughly evaluate the proposed approach on 14 generative model. They additionally measure the effectiveness of their approach compared to other pre-existing metrics, showing strong correlation with human judgement and robustness of the metric towards prompt variation.

Strengths

* This is a sound and extensively evaluated approach towards the automatic quantitative assessment of the knowledge learned by LLMs. This method displays strong correlation with human judgements. While it cannot replace human annotation entirely, it should be extremely useful to the community as a cheaper, faster automatic way of performing knowledge assessment, much like how BLEU can be used during model development to validate the performance of translation models. * The authors should be commended for calling out robustness (consistency of generation given similar prompts). This aspect is crucial for real-world applications of such models, and is often ignored in similar works.

Weaknesses

* The proposed framework is limited to assessing knowledge in a fairly simplistic way, via the prediction of entities in subject-relation-object triplets. It would have been interesting to hear more about the limitations of such an approach, taking into account e.g. slightly more complex type of queries (see e.g. arXiv:2305.01157). Very minor: * lines 141-144, 148, 202, 291, 313-315: straight quotes should be turned into curly quotes

Questions

* Do you think your proposed approach could be effective as a validation metric for the training of LLMs?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors did a decent job of addressing limitations, and I don't expect any potential negative societal impact from this work.

Thanks for the reply

I've read the author's response and will keep my score. I appreciate the explanation of Kendall's tau score and the additional analysis!

Further comments and discussions will be appreciated!

Dear Reviewer adWh, Thank you for your valuable time to review our work and for your constructive feedback. We posted our response to your comments a week ago, and we wonder if you could kindly share some of your thoughts so we can keep the discussion rolling to address your concern if there are any. In the previous response, 1. We clarified the scope of our "entity", which encompasses a wide range of categories, including dates and numbers as you mentioned. This enables our knowledge assessment to cover a wide range of knowledge. 2. We outlined the procedure for employing our method when new relations emerge. It is important to note that our method can be flexibly adapted to different relation types based on specific requirements. 3. To eliminate any misunderstanding, we explained the reasoning behind the conclusion on instruction-tuning and, as suggested, revised the claim to be more rigorous. 4. We addressed the distribution of relations in the dataset by providing a table on the proportion of related facts for the top-k relations. This table has been incorporated into Appendix 7 in the revised version of our paper. And we've modified the typos and writing problems you mentioned, thank you for your gracious help again. We would appreciate it if you could kindly take a look at both the revision and our response to your comments. We would really appreciate it if you are willing to increase your score. If you have any further questions, we are happy to discuss them! Best regards, Authors

Follow up to Reviewer 4epC

Dear Reviewer 4epC, We would like to thank you again for your reviews and your acknowledgment of our novelty. We have added replies to the weakness you mentioned and highlighted our other findings. Since the rebuttal deadline is approaching soon, we would appreciate it if you could let us know if our responses have addressed your concerns satisfactorily. If your concerns have not been resolved, could you please let us know about it so that we have the opportunity to respond before the deadline? We would really appreciate it if you are willing to increase your score. And we would be happy to have any follow-up discussions or address any additional concerns. Thanks very much! Looking forward to your reply. Paper9365 Autors

Further comments and discussions will be appreciated!

Dear Reviewer bA4R, We would like to thank you again for your detailed and constructive reviews. To better address your concerns regarding "the limitations of such an approach," we have incorporated discussions on two additional limitations in the latest version of our draft. Furthermore, we appreciate your reference to the related work on more complex query types. We have included a discussion on the paper mentioned[1] in both our Related Work and Limitation sections, as well as a discussion on the possible extensions of our approach for such queries (i.e., complex query decomposition and a compound metric for complex knowledge) in our future work discussion. For other questions, we have updated our draft and added replies to your comments. Overall, many thanks for your insightful points and suggestions. These comments really help improve our paper. We hope our answers have addressed your concerns. If you have any further questions, we are happy to address them. We would really appreciate it if you are willing to increase your score. Thanks very much! Best regards, Authors **Reference:** [1] Choudhary, Nurendra, and Chandan K. Reddy. "Complex Logical Reasoning over Knowledge Graphs using Large Language Models." arXiv preprint arXiv:2305.01157 (2023).

Thanks for the response.

I have read the author's responses and other reviewers' responses as well. I appreciate the further statistics and explanations for my questions. It would be great to include them in a future version of this paper.

Response to rebuttal

Many thanks to the authors for the responses to my questions. Having read the other reviews and rebuttals, I confirm my scores.