PRobELM: Plausibility Ranking Evaluation for Language Models

Summary

This paper introduces a new benchmark called PRobELM, which aims at testing the LLMs capacity of recognizing plausibility of events and scenarios. PRobELM is created from facts extracted from WikiData, using data points that were added at a later timestamp than the training data of the models under evaluation, and then a set of alternative, less plausible facts are generated for each of them and LLMs are asked to rank them from the most to the least plausible. The authors evaluate the new benchmark with 10 different LLMs of different families and sizes, using the perplexity scores for each set of scenarios, and found that plausibility inference is a challenging task even for models that excel in identifying factual accuracy. Even if large models generally perform better, model architecture seems to have a more important effect of the results. with the Pythia family achieving strong scores across metrics and settings.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

+ important and timely benchmark to evaluate LLMs capacity to discriminate the plausibility of events and scenarios. The dataset seems to be well conceived and several quality control measures have been implemented in the creation process + extensive evaluation with different models, prompt types and careful control of the data contamination issue, thanks to inclusion of Wikidata facts that the LLMs could not have been exposed to, given the timeframe of their training

Reasons to reject

None, for what I can see.

Questions to authors

- I was wondering if you could add an additional error analysis where you manipulate the surface form of the sentences corresponding to a set of scenarios (e.g. one way could be to try to paraphrase them via synonymy replacement, or using some syntactic transformations) and check to what extent the models' judgements are robust to such variations. - a recent preprint you might be interested in, evaluating LLMs from a psycholinguistic point of view: Kauf, C., Chersoni, E., Lenci, A., Fedorenko, E., & Ivanova, A. A. (2024). Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models. arXiv preprint arXiv:2403.14859. As an interesting point in common with your results, I see the fact that LLMs-derived log-likelihood can get you close to human-level for the distinction possible vs. impossible events, but they fall short of humans for more subtle distinctions between events with similar degrees of plausibility.

Summary

This paper proposes PRobELM, a new benchmark suite for plausibility ranking evaluation for language models. \ Plausibility ranking, as the research problem the paper studies, is important. The paper also provides a systematic approach to automatically curate test cases (authors name it as "scenario"). \ The evaluation section of the paper is its strongest part; it is very extensive, covering ten models and different configurations of the testing. \ The biggest weakness for me is that there appears to be a gap between the paper's realization of plausibility and the actual definition of plausibility (check weakness section). I am now giving a neutral score (round(5.5) = 6). I look forward to authors' responses and will raise my scores if questions clarified.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

Reason to accept: - The problem this paper tackles is important. There are many insightful intuitions in the paper, such as Bayesian inference as an approach to plausibility modeling. - The evaluation benchmark is promising, especially concerning the temporal evolution of world facts and a language model, which is trained before (or after) a fact has become established. It is particularly useful for end users to evaluate models on recent facts to assess their reliability in a specific (new) domain and scenario. - As mentioned in the summary, the evaluation of the paper is very comprehensive. It is beneficial to have three different evaluation settings (i.e., statement, text completion, and question answering). The consistency of models' rankings is also verified across different task settings and temporal changes.

Reasons to reject

Reason to reject: I have some reservations about the paper's realization of plausibility in the dataset. The definition of plausibility is "the quality of seeming likely to be true, or possible to believe" ([Cambridge Dictionary](https://dictionary.cambridge.org/dictionary/english/plausibility)). The paper's realization in Section 3.1 (most plausible scenario) adopts a Bayesian view, considering that a future event is the most plausible given the current world state. This current realization is more suited to temporal sequence prediction rather than plausibility prediction. This is because a future event could be very surprising (e.g., Marcelo might score a goal in the 90th minute of a football match, which is very surprising (NOT plausible) given all world states in the previous 89 minutes). Can the authors substantiate how the current implementation actually models plausibility, which can also be modeling surprising events? It is interesting to view all events $ (e_1, e_2, ..., e_n )$ in a Bayesian graph model. If I am not mistaken, the authors are now adopting a chaining view. Can the authors explain whether it is Markovian or not (i.e., future events depend only on the previous one step)? Will being Markovian or not make any difference to the empirical evaluation? I also encourage the authors to explore the graph structure of the Bayesian graph. Is it possible that a new plausibility evaluation case can be created by $P(e_3 | e_2, e_4)$? This means that an intermediate event ($e_3$ in this case) can also be tested as long as it is not shown in the training corpus and we are certain that observed events $(e_2, e_4)$ are shown. If such a graph structure can be realized, can we make the dataset larger?

Questions to authors

Questions for the authors: My major questions are raised in the Weakness section. Please help to address them. Other questions: - Can you better articulate or visualize the differences between your dataset and COPA? - How was the hyperparameter for the 11 less plausible events determined? - I am not very clear on the negative sampling part (Sec 3.2). Could you articulate or visualize this better? Are these samples centered around the event relations? If so, how many event relations are there in your dataset, and what is their distribution? Do different event relations undergo different processing? - I am keen to know whether your evaluation is Markovian. P(e_2 | e_1) is different from P(e_3 | e_2, e_1). I think it is interesting to find the difference between only looking at one previous observed event and looking at all observations.

Summary

This paper introduces a Plausibility Ranking Evaluation for Language Models, a benchmark designed to assess their ability to discern more plausible from less plausible scenarios through their parametric knowledge. In this benchmark, the LMs need to make predictions by either finishing a statement, completing the text, or completing the answer, and the authors use perplexity to rank all the candidates in the dataset to evaluate each LM's performance. Experiments are conducted on ten open source (relatively) small-size LMs to test the performance on this benchmark and show that factual accuracy (on another dataset) does not directly correlate with plausibility performance.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. Precisely evaluating plausibility is essential in many applications, such as knowledge discovery. 2. The authors conduct experiments on many LMs.

Reasons to reject

1. First, I have doubts about the ground truth ranking in this dataset. If we use entity co-occurrences to make the ranking plausible, then basically, it's all about statistics but not reasoning. Additionally, LMs themselves are trained with loss that are highly correlated with "token" co-occurrence, and hence, I don't think it's a substantial ground truth set with this construction setting. 2. Second, it would be more appreciated if the authors could provide solutions for generating all the "ranking candidates" in real applications. For example, if conducting knowledge discovery, how would this plausibility benchmark help us get and rank all plausible candidates? If we still rely on LMs to generate the results and use the perplexity to rank, what would be the difference between it and beam search? 3. I think deeper analysis should be conducted, especially on the observations the paper found -- is there any reason and insights on "higher performance on factual accuracy do not necessarily excel in plausibility" ; Any deeper analysis on the drop of 2017 performance drop?

Response from Reviewer

Thank you for the explanation. I get most of the rebuttal ideas. For the "Ground Truth Ranking" concern, I would expect a more thorough analysis to be presented in the later version of the paper. Another concern I have for this paper is that this paper is built for each LM and uses the corresponding cutoff time to generate the data. How do you expect this benchmark to be long-lasting for future research and development on LM? If this additional concern can be answered, I'll further raise my evaluation.

ACK your response

Hi authors, Thanks for your response. Thanks for the clarification for the first point. I would raise the score from 5.5 (neutral) to 6 as slightly positive for your paper. Based on your explanation, can we say that the plausible events given the current states must be seen in written corpora? If people haven't discovered the earth is round (and write them down in corpora), what will be the plausible event?