Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

Summary

This paper conducts an empirical study on the reliability and transitivity of the Elo rating system when evaluating Large Language Models. By conducting experiments on both simulated data and real-world scenarios, this paper suggests that certain parameter setting guidelines should be followed to ensure the stability of evaluation results when applying the Elo rating system to LLM assessments. The paper provides insights and guidance for future research and applications of the Elo rating system in various LLM evaluation contexts.

Strengths

- This paper provides a detailed experimental analysis of the practicality of the currently common Elo rating system used for evaluating the performance of Large Language Models (LLMs). - Through appropriate experimental setups, this paper explores the impact of various factors on the Elo rating system results, including the number of sequence arrangements, the number of games, the K value, and win rates. - By combining results from simulated experiments and real-world scenarios, this paper demonstrates good reliability in its findings.

Weaknesses

- This paper explores the experimental results with up to three players. It would be more meaningful if the conclusions could be further generalized to settings with more players. - In reality, $N_{perms}$ can be set very large without obvious computational cost to stable the evaluation results. Therefore, in multiple experiments, I would prefer to see results with settings like $N_{perms}$ = 10k, similar to those in Figure 2.

Questions

My main concerns are listed in Weaknesses

Rating

7

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

I think that this paper has no significant limitations apart from what is mentioned by the authors in Section 8.

Summary

Updated after response: I think this paper could provide some valuable contributions to the community, but has real presentational issues. The authors fixed some of them that had an easy fix (which I pointed out) during the rebuttal. However, I believe the presentational standards (and therefore the understandabiluty of the contribution) are not up to par for this conference, and I thus stand by my borderline reject recommendation. This paper studies the Elo rating system, recently popular in the context of ranking LLMs. Specifically they study two axioms that evaluation methods should adhere to: reliability and transitivity. They find that Elo ratings for LLMs are highly sensitive to the order of comparisons and the choice of hyperparameters, and that desirable properties such as transitivity are not always guaranteed. In their first experiment, the authors focus on order sensitivity. They conduct a controlled experiment with synthetic human feedback, in which they compare various model pairs with different probabilities of winning a pairwise comparison. They show that, with only order of comparisans (which, if I understand correctly, would be the actual scenario in which we deploy this?), if model A has a probability 40<P<60 to win a pairwise comparison with model B, even 500 games are not enough to converge to the conclusion that model A is better than model B. With 100 orders, the difference is clear after few games, though (semi) convergence happens only after around 80. For smaller differences (51%), even 100 orders do not reliably show the difference between models. For differences of P(a_win > 60) Elo results are stable (?) (I can't fully match this statement in the text with the figures). Next, the authors investigate the impact of K-factor (which, I learned from Wikipedia, is the maximum number of points a ranking can be updated after a match). They show that higher K-factors eacerbate instability for single orderings for models that have similar performances, but that faster convergence is observed for higher K-factors when models diverge more. Then, the authors investigate transitivity, using a set of three models with different respective win probabilities. They find that when there is a clear disparity between models, Elo ratings accurately reflect the expected hierarchy, but in more complex cases the rankings become less stable, challenging the assumption of transitivity. K-factor and number of orderings play a key roal in the stability of the rankings. The authors confirm several of their findings using real human feedback data. And wrap-up their paper with a list of recommendations for using Elo ratings.

Strengths

The topic studied in this paper (Elo ratings for LLMs) is important, and the experiments are extensive. The paper provides several interesting lessons for using Elo ratings for LLMs that I believe can (positively) influence how we compare models in rankings based on pair-wise comparisons.

Weaknesses

While I think this paper could potentially be very impactful and has many interesting results, in its current state it does not seem ready for publication. It took me excessively long to review, because the presentation is rather messy, and not clear at several points. Some of those things are easily fixable, e.g.: - The Elo rating system is briefly explained in an appendix, but given the importance of the concept it should be in the main text (if space is needed, I think equation 5 describing transitivity can safely be moved to the appendix). - Some explanation of the K-factor should also be included. It is sort of described, but not really explained or defined, not even in the appendix as far as I can see). - Maybe I missed it, but I don't think it is listed how Elo ratings are most commonly used for LLMs, is it with only a single ordering? - Table 1 is somewhat difficult to read, why not make a plot similar to Figure 1 but with multiple lines for the different models? - In the introduction -- impliations of our work -- it is listed that the work is particularly important in situations where model performances are closely matched, which is a common occurrence in many real-world scenarios, whereas in 4.2 it is listed that the result is relevant because skewed win-rates are common in real-world scenarios (or maybe that is not what is meant, if not I am not sure why it is mentioned that there are skewed win-rates in real world scenarios) More problematic are the unclarities concerning the results, for example: - It is not clear with what K-factor the main experiment (4.1) is conducted (or at least I couldn't find it) - in 4.1 it is stated that for P(A beats B) >= 0.6 Elo results are stable; I can't match this result with Figure 1 (middle column), where there seems to be quite some variation for P(a beats B) = 0.55 as well as P(a beats B) = 0.6 - The idea of comparing a single sequence with 100 different orderings is generally strong, but doing only one single sequence and one series of permutiations is not very informative in this case: what we actually want to know is how often a single sequence or an average over 100 sequences would give us the right results. Perhaps this single sequence was a very unlucky one among many that give us the correct result, or the 100 permutations were instead 100 lucky permutations. To really make the point, we'd need the variation between single sequences, and multiple selections of 100 permutations. (similar for Figure 4) - In 4.2 it is listed that faster convergence is observed for higher K-factors, but I don't understand the evidence for that. I don't think this is visible from Figure 3, and I couldn't find an appendix plot either.

Questions

- If I understand correctly, the number of permutations is the number of game orders, why not use a word that matches that closely, e.g. 'game orders' or 'orderings'?

Rating

4

Confidence

4

Soundness

2

Presentation

1

Contribution

3

Limitations

Yes

Summary

Evaluation plays an important role in LLM research. Previous works have assessed the performance using the Elo rating system, which is designed for ranking players in games. This paper shows that Elo rating does not always satisfy two critical axioms, reliability and transitivity. So the rankings of the models are not accurate. Transitivity is whether a consistently higher-rated model outperforms those with lower ratings in all scenarios. Reliability has two aspects, sensitivity to ordering and sensitivity to hyperparameters, how the Elo scores are affected by the order in which models are compared and the sensitivity to hyperparameter adjustments, particularly the K-factor. The authors first validate the properties of Elo using synthetic data generation via Bernoulli processes. Then extend the validation to include real-world human feedback. The findings are that the stability of Elo ratings is influenced by disparities in win rates, analogous to win probabilities in synthetic data, and by the choice of hyperparameters K-factor and Nperms. Finally the authors provide empirical guidelines for robust Elo-based Evaluation.

Strengths

1. The evaluation of LLMs plays an important role in LLM research. The reliability of the Elo rating is an interesting and meaningful research topic. 2. The authors highlight two Axioms for Elo-based evaluation, which is reasonable and useful. 3. This paper verifies the findings from both synthetic human feedback and real-world human feedback. 4. This paper provides several empirical guidelines for robust elo-based evaluation.

Weaknesses

1. The writing is not very friendly to the reader unfamiliar with the Elo rating. 2. This paper only conducts experiments on three LLMs.

Questions

1. In line 93 you mention the K factor, but it is hard to understand for the reader unfamiliar with the Elo rating. 2. How can the Elo rating be generalized to multiple-task evaluation? 3. Why not conduct experiments on more widely used API-based LLMs, such as ChatGPT, Gemini, and Claude-3?

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper has no potential negative societal impact of their work.

Summary

This paper examines how well the Elo rating system works for evaluating LLMs based on human feedback. The authors first check how stable and reliable Elo ratings are under different settings and point out key factors that affect these ratings. They study how changing the order of matches and adjusting settings like the K-factor can influence the consistency of Elo scores. Using both synthetic and real-world data, the paper shows that Elo ratings become more reliable with more match permutations and the right K-factor adjustments. The paper also discusses the transitivity of Elo scores, showing that this property doesn't always hold for models with similar performance levels. The authors suggest practical tips for more reliable Elo-based evaluations, like running many permutations for stable scores and adjusting the K-factor based on how different the models' performances are. They highlight the limitations of current Elo-based evaluation methods and propose future research directions, such as considering ties and using more complex data simulations to better mimic real human feedback scenarios.

Strengths

1. Despite Elo rating being a popular method for evaluating LLMs, there is a lack of analysis on its robustness. This paper provides a comprehensive investigation, helping us better understand and implement Elo rating for LLMs. 2. The authors explore the reliability and transitivity of Elo rating under different configurations. Their definitions and formulations are clear and reasonable. 3. Experiments are conducted on both synthetic data and real human feedback, offering valuable insights for future Elo rating applications.

Weaknesses

**Major Issue:** My biggest concern is the relatively small amount of real data used in the experiments. Only 500 prompts and three model pairs are included, with just one pair having similar performance (Dolly-v2-7b vs. Dolly-v2-12b). Given that Chatbot Arena currently includes dozens of LLMs and has released a large amount of preference data, the volume of real data in this study seems insufficient. **Minor Issues:** 1. As the authors mentioned, not considering ties appears to simplify the widely used "win/tie/lose" practice. 2. The empirical guidelines provided may have limited applicability since most papers on automatic evaluation of LLMs report win rates rather than Elo ratings. Also, running more than 100 permutations seems costly and lacks practicality.

Questions

1. If I want to automatically evaluate $k$ models to get their final Elo ratings, do you think the size of $k$ will affect the robustness of the Elo rating? 2. Why does Dolly-v2-7b have a slightly higher win rate than Dolly-v2-12b in Table 2?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The paper discusses its limitations in Section 8. My personal view on the limitations can be found in the weaknesses mentioned above.

Thank you for the Response

Thank you for the response and additional experiments. This addressed some of my concerns, so I’ve raised the score to 7. Good luck!

Thanks for the response

Thanks for the response. I will keep my original (positive) score.