Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think

Summary

This paper analyses the robustness of multiple-choice QA evaluation of instruction-tuned LLMs. It compares two approaches: 1. Token probability: Ranking answer choices based on the model's predicted probability of an early token (varies by model). 2. Text-based: Extracting the answer choice from the generated text with a classifier. The paper compares three model families (Llama2, Mistral and Gemma) on the popular MMLU task. The authors argue that finetuning techniques have reduced the robustness when relying on token probabilities, and provide empirical evidence for this. They provide informative experiments, and show that token probability systematically underestimates model performance on MMLU tasks. They also provide evidence for improved robustness with their classifier.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

* The paper is very well written and well structured. * The included experiments provide strong empirical evidence for the author’s case to prefer text-based methods over token probabilities. * The findings are very relevant and important for LLM evaluation practices.

Reasons to reject

* Using a trained model for answer extraction raises concerns about the generalisability of the findings to models with significantly different instruction tuning / rlhf. Their discussion of Gemma doesn’t alleviate this, because they neither show how a model trained on Gemma data would behave, nor provide an error analysis that systematically compares the model answers. Something like this could have been achieved with a data ablation study without collecting more examples. * The paper is very similar to ​​https://arxiv.org/pdf/2402.14499 (that they cite throughout) in both findings and methods. The main additions are 1) including the Gemma model and 2) using the MMLU benchmark instead of just OpinionQA.

Questions to authors

* The fine-tuned Mistral model used for answer extraction could introduce its own biases, affecting the evaluation results. Have you considered measuring and mitigating this? * Have you considered evaluating the robustness of closed, but popular models in comparison to the open models studied in the paper? (Even small samples could be interesting.) * Do you have recommendations for finetuning methods based on your findings that could alleviate the robustness issues?

Summary

This paper studies the robustness of instruction-tuned LLMs in multiple-choice question (MCQ) evaluation. The authors found that compared to the traditional way, where LLMs are straightforwardly prompted to predict the option IDs as answers (using the next-token prediction probability), prompting the instruction-tuned LLMs with natural instructions to select the answers leads to better robustness. More empirical analyses consolidate the claim and findings.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* This paper focuses on an important problem, i.e., the selection bias of instruction-tuned LLMs in MCQ evaluation. This is somewhat paid less attention to in previous research. * The evaluation and analyses are thorough and sound. They involve comparison with baselines (PriDe), multiple benchmarks (MMLU and its multiple categories; OpinionQA), and multiple models. These efforts consolidate the findings in this paper and justify the effectiveness of properly prompting instruction-tuned LLMs in MCQ.

Reasons to reject

I do not see clear reasons to reject it, but I suggest the authors to polish the writing to make it more accessible. For instance, the term "mismatch" is quite unclear and confusing across abstract, intro, and Section 3.3. I think more formal definitions are needed. Also, its correlation with the robustness (Figure 2) can be better presented in scatter figures (e.g. the y-axis is a metric of robustness while the x-axis is the mismatch value).

Summary

When asking a LLM to answer a multiple-choice question with, say, four answer alternatives "a", "b", "c", and "d", it is a non-trivial problem to determine whether or not the model has given the correct response. This is because the model might generate variations of the answer, like "a", "Alternative a", "Answer: a", "The correct answer is a", etc. This might happen even if the prompt to the model explicitly asks for a single-letter answer. This paper examines various methods for determining the given answer, including first token probability (assuming that the model is well-behaved and answers with a single letter), and text-based evaluation, which amounts to training a classifier based on the text outputs from models responding to multiple-choice questions. The classes are the answer alternatives, e.g. "a", "b", "c", and "d". The investigations in the paper concern several models, datasets, and evaluation metrics. The authors have also studied the effect of perturbations, like changing the word order in the question, changing the order of the answer alternatives, adding extra answer alternatives, etc.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The paper is well-written, providing a good introduction to this research question. The experiments are systematically conducted, and the results are interesting, although not surprising

Reasons to reject

The contribution of the paper is not very big.

Summary

The paper tackles robustness of LLMs for MCQ answering by evaluating the actual text generated. Most work in MCQ predominantly uses first token probability, but prior work has shown that they have selection bias (Zheng et al, 2024). This paper perturbs the answer choices (editing the prompt, swapping order, and adding answers) and uses a fine-tuned model (Mistral-7b) to help evaluate text answers and measure discrepancy between first token and text answers.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

Experiments are clear and precise and imply that there is strong selection bias in first token probabilities, even when debiased with prior work on perturbations (3.2/3.5) and additional choices (3.5)

Reasons to reject

- Some of the experiments seem lack numbers with PriDe, which make the paper weaker; this could be a mis-reading and could be clarified in the next section. - Some of the terms could be defined better ( especially something like "robustness discrepancy" and"mismatch rate" which are in the main claims of the paper). - Experiments largely hinge on their fine-tuned Mistral model on human annotated texts that pick out the intended MC answer. Their results seem strong enough to be convincing (acc generally 98%+ excepting Gemma), but some more analysis of their fine-tuned model would be helpful, especially for the few cases the model was wrong, would help quell some worries. - The paper could be seen as less exciting with other related work like Lyu, 2024 which they cite. However, the paper goes further into depth about the discrepancy and offer a potential solution with a text-based model instead.

Questions to authors

- Are the results in Table 6, Figure 3, and Figure 4 solely without PriDe? If so, what are the numbers on those? - nitpicks: - Overall, the figures are very helpful to understanding the paper! But for some figures like Figure 2, it might be clearer to indicate a "better" direction, especially since metrics like accuracy and entropy are "better" in different directions. - It seems that PriDe is cited twice as a and b; is that intentional?

Author response

Hi reviewers, Please take a look at author's responses and other reviews of the paper. If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum. Overall, most of the reviewers are positive about this paper. Reviewer 6RFZ: - One of your concerns is that the generalizability of the results to other closed models or those trained with other instruction/rlhf training. Does the author response address your concerns? Reviewer 4zY5 also raised similar concerns. Other reviewers should also feel free to weigh in on the above points. Thanks!

Reply to Reviewer 6RFZ

Thank you for your comments and further questions! **Reliability** Thank you for your suggestions. We agree it is important to address that our classifier may not perform perfectly on models that are significantly different. If one wants to apply our classifier to their own models, we suggest inspecting the model's response first, and further finetuning the classifier if the response style is highly different from the training data, to further improve its robustness. Therefore, we will share our classifiers so that other people can continue improving them based on their results. We will include this in the final version of the paper. **Solution to Wang et al.** Sorry for causing any confusion. By "solution", we don't mean *"how to extract the text answer"*. The solution here means *"which one to choose when facing the choice between the first token and text answer evaluation".* Wang et al. showed the mismatch issue but didn't give a clear suggestion on which one is clearly better. **How much benefit do we get by doing text evaluation instead of first token probability evaluation?** Given the mismatch problem shown by Wang et al, we want to know which one should we choose by comparing them systematically in terms of robustness and accuracy. In both cases, the text answer evaluation shows superior properties. Therefore, we provide an answer to the mismatch problem raised by Wang et al. We hope this can address your concern adequately. We are happy to give further explanations if there is still any uncertainty. If you find the revisions and clarifications satisfactory, we would appreciate your consideration in re-evaluating the manuscript.