Using Natural Language Explanations to Rescale Human Judgments

Summary

This paper presents a method of using LLM to convert human Likert ratings to scalar values using the human explanations. They first collected the Inquisitive-board dataset. The dataset consists of news articles and questions written by a linguistics student, and answers from LLMs. Then they ask annotators to evaluate the generated answers on how much information (major/minor/none) from the original articles the answers are missing, provide an explanation and how many sentences are missing. They found some disagreement in the Likert scale (major/minor/complete) ratings They then provide some rubrics for converting the Likert scale ratings and explanations to a score in 0-100 for how much information is missing. They have 3 experts doing this rescaling for reference, and evaluate how well LLMs can do this rescaling. They found that the LLM rescaled values correlate with expert rescaled values and explanations provide fine-grained information beyond the Likert scale.

Rating

5

Confidence

5

Ethics flag

1

Reasons to accept

This paper provides an interesting method on rescaling Likert scale judgments and showed that the explanations provide insights into the discrete judgments. The methodology is sound and the approach is interesting.

Reasons to reject

The biggest question I’m left with after reading this paper is: what does one do with this method? The amount of missing information is a limited dimension of a generated answer. Can this method be extended to reconcile multiple and potentially conflicting dimensions? Since this paper is in the evaluation track, do the authors recommend using this method for future LLM evaluation? Does using the rescaled values as an evaluation metric leads to different models being preferred and are those models indeed better in some ways? Without clear takeaways/recommendations the impact of this paper is likely limited.

Questions to authors

I suspect the extent of information missing probably also correlate with the location of the missing sentence, or more broadly the information structure. Generally the information in the beginning of the article is probably more prominent and major than the information at the end of the article. The question being asked will also impact which information is prominent. Have you looked at this in your analysis?

Summary

The paper seeks a way to improve human evaluations of subjective tasks via large language models (LLMs). In particular, the authors exploit the natural language explanations to rescale ordinal annotations. The authors propose a method called Explanation-Based Rescaling (EBR) that utilizes natural language explanations provided by human annotators alongside their Likert scale ratings. EBR leverages an LLM to convert both the rating and explanation into a more fine-grained numeric score (0-100) guided by a post-hoc defined scoring rubric. The authors demonstrate the effectiveness of EBR on a document-grounded question answering task where LLMs achieve near-human performance. They show that EBR produces scores closer to reference scores generated by experts using the same rubric, without impacting inter-annotator agreement.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

S1. Utilizing natural language explanations to refine and quantify human judgments is novel. EBR allows for a more fine-grained evaluation of LLM outputs by mapping them to a 0-100 scale. This is particularly useful for tasks with subtle distinctions where traditional Likert scales might be too coarse. Furthermore, its flexibility (the fact that the scoring rubric can be designed post-annotation) can give room for better optimization. S2. The authors conduct thorough experiments and analysis on a newly collected dataset, demonstrating the effectiveness of EBR compared to various baselines and highlighting its ability to capture subtle nuances in human explanations.

Reasons to reject

W1. Achieving high agreement among human evaluators for subjective tasks is a significant challenge. As shown in Table 2, reaching a consensus among human annotators on missing majors seems to be particularly difficult. Given this, using the results of human annotators as references may not be the most effective method for evaluating the capabilities of LLMs on this task. While I don’t have a clear solution for this issue, it’s certainly a challenging area that requires further exploration. W2. It was unclear whether fine-grained labels are necessary and how they could be helpful in improving LLMs or for NLP overall. W3. Regarding the evaluation, the authors mention that the work aims to demonstrate the usefulness of using NLEs to enhance rescaling performances. To achieve this, the experiment should be designed to allow for a comparison between sets with and without explanations. The current setup does not directly evaluate this effect. W4. The evaluation is centered on a specific task of document-grounded question answering. Further investigation is needed to assess the generalizability of EBR to other tasks and domains. Specifically, in the presented task, the quality of the answers can be determined based on the missing sentences, which are relatively straightforward to quantify. However, in other scenarios, such clear criteria may not exist. W5. Lastly, the paper acknowledges that the creation of the rubric is currently a manual process. This could be time-consuming and subjective, potentially affecting the generalizability of the approach.

Summary

This submission proposes a method to rescale coarse numerical ratings to fine-grained ones. The envisioned scenario is that for a task like document-based question answering, annotators asked to rate answers have given coarse-grained ratings on a four-point Likert scale ("complete", "missing minor", "missing major", "missing all"), along with short free text justifications of their rating. Noting the inherent subjectivity of these ratings and justifications, the authors assume that a finer-grained numerical scale from 0 to 100 points is more appropriate. To rescale from coarse to fine ratings, the authors prompt LLMs with the rating, justification, and an instruction that describes how to assign scores. The evaluation of the rescaled scores focuses on the two middle labels ("missing minor" and "missing major") since the extreme points of the scale ("complete", "missing all") show little variation among raters. The evaluation is mainly performed by measuring agreement of LLM-rescaled scores with a rescaling performed by three experts on 145 instances. Agreement is measured via rank correlation (Kendall's Tau) and mean absolute error (MAE). In terms of rank correlation, the proposed method fails to outperform one of the simple baselines, namely the "missing sentence heuristic". In terms of MAE, the proposed method outpeforms all baselines, but unlike for rank correlation, there are no significance scores given, which is a concern considering the small sample size of 145 instances.

Rating

3

Confidence

4

Ethics flag

1

Reasons to accept

-

Reasons to reject

1. The proposed method is not motivated well, to the point that it is fundamentally unclear why and how it should address the research problem, for two reasons: 1.1 It is left unclear why a finer rating scale is better suited to capture the inherent subjectivity of the task at hand. The only motivation given is a reference to Kocmi & Federmann (2023), "which validated the effectiveness of a 0-100 scale compared to other options." What "effectiveness" means and if/how their results from machine translation apply to ratings of answers in QA tasks is unclear. If "subjectivity" in the task at hands means that different annotators have different views of what information is important in a document, then their ratings have no monotonic one-dimensional order and no amount of granularity can fix that. 1.2. According to the introduction, "subjectivity (and thus, inherent disagreement) in annotated data has surfaced as a key direction" and the authors make the observation that "intricacies in human judgments can be captured by natural language explanations provided during data annotation, which capture more nuanced subjectivity". So far so good, but now the authors claim that their "key idea is to make the ordinal label space (e.g., a coarse Likert scale) fine-grained, namely a 0-100 scale Kocmi & Federmann (2023), which then enables us to place the initial annotations in it by leveraging natural language explanations in a principled manner." In what manner the proposed approach is "principled" is not clear to me. The authors are possibly referring to the scoring rubric that is included in the prompt, but just because some instruction is included in the prompt does not mean that the LM adheres to it, and it is even less clear if the LM faithfully can execute complex instructions that require it to repeatedly subtract a variable number of points, with the amount of points varying on the amount of information that is missing from the answer to be rated. 2. The benefit of the finer-grained ratings is not shown. The evaluation stops at showing moderate agreement with expert-produced fine-grained ratings and stating that the rescaling does not impact inter-annotator agreement. The introduction claimed that the proposed method can "capture more nuanced subjectivity", but this is never directly shown.

Questions to authors

- While the intention is clear, the "Rescaling Prompt" shown in Figure 3 does not specify what the initial score is from which points should be deducted - It is also not clear how information is measured, so a phrase like "half of the correct information" is left wide open to subjective interpretation

Summary

This paper proposes to use LMs to rescale human quality judgements by using explanations provided by the human annotators. The authors motivate the work by stressing the importance and value of human annotation in today's NLP landscape for model adaptation and evaluation, especially as models become more capable and automatic metrics are no longer reliable for increasingly subjective or complex tasks. The proposed automatic rescaling method can be used for a few purposes: to calibrate humans who have different interpretations of numeric rankings, to take a coarse-grained rating and make it more fine-grained (such as on a 100-point scale, as the authors propose to do), and/or to apply or specify an arbitrary rubric or criterion *after* the annotations are collected. The authors first collect a dataset of 12.6k high-quality human annotations of labels with explanations (5 per instance). They then test their rescaling method on a document-grounded non-factoid QA task where the goal is to rate "completeness" of a model-generated answer. They find that, despite the subjectivity of the task, their method can 1) rescale judgements while retaining the same level of human agreement as the original data annotations, and 2) closes some of the gap with expert rescaling judgements made using the new rubric, indicating that the method is at least partially successful.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The paper is extremely clearly written and well-motivated. Related work coverage is comprehensive. - The research question is quite timely. The proposed rescaling is an interesting use-case of LMs that could resolve a number of issues in human annotation for subjective tasks. I think the paper will have a good audience at this conference. - The experiments are thorough and well-executed. Using a) LLMs to automate tasks and b) crowdsourcing can have a number of pitfalls if not assessed or evaluated rigorously, but the authors have done a good job in both of these areas, including elaborate quality control details in the Appendix.

Reasons to reject

- A few methodological details could be expounded on or are limited in scope. The main one is a limited choice of models investigated. I elaborated on these below in "questions/comments". Overall, I don't think that these issues affect the contribution and validity of the paper, just constrain its scope a bit. - Ultimately, the proposed metric is not extremely strong. The main results (Table 3) support the inclusion of a rubric, but the avg. EBR baseline with a rubric is quite competitive, which may be support for the fact that the 1-100 Likert Scale is too fine-grained (as I've also touched on in "questions/comments" below). I'd appreciate if the authors could elaborate on their hypotheses for why this may be or their arguments against using the average approach. - I assume the authors will publicly release their dataset with publication, but if not, that will severely limit the impact of the paper.

Questions to authors

Questions/comments: - Will you release your code & dataset upon publication? - The motivation for using such a granular scale (1-100) could be made stronger. It appears from the given example and reported human MAE that a 1-10 or 1-20 scale would be sufficient and that the super granular scale may invite noisy disagreement (for both humans and machines). Could you explain this choice a bit more? - Elaboration on "limited choice of models" mentioned above: Only one model (GPT4-0613) is studied for its ability to do the rescaling task, so its not clear how the correlation w/ human experts would change for other models and thus if this technique would be useful or valid for them, which is an important question for those doing non-API research. Similarly, only OpenAI API models (though 3 of them) are used for the initial question answering task. Related work: - there are some interesting papers in psychology literature on the Likert Scale and humans' varying interpretations or calibrations; may be good to cite. Typos/misc: - many of your citations are not in parentheses but should be (i.e., by using `\citep{}`). - please make Figs 2,4 larger given extra space in a camera-ready version. - abstract: what do you mean by "propose a method to rescale ordinal annotations *and explanations* using LLMs"? seems like it should say "using explanations" - Fig 1: would be good to also show the 3 expert scores here to get a sense of variance and that there is no 100% agreed ground-truth. - Seciton 6.2 last paragraph: here $M$ are the (unaveraged) individual reference scores, correct? It's a little confusing to double up on this $M$ notation. - Important to mention that being able to use an arbitrary scoring metric for rescaling *after* annotations are collected relies on the explanations containing enough detail for that particular aspect to be able to do so.