Stronger Random Baselines for In-Context Learning

Summary

The primary contribution of this paper is to propose a stronger random baseline when accounting for validation set reuse. The authors argue that the standard random baseline (expected accuracy of guessing labels randomly) is not sufficient when we allow for multiple evaluations on validation set, and when accounting for that (multiple evaluations on validation set and picking the max), the authors derive the closed form expression of the stronger random baseline, and show that many published results outperform the weaker standard random baseline but not the newly proposed baseline. The authors also argue that this baseline predicts held-out performance better.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The new maximum random baseline is theoretically motivated and is intuitive to understand: if we allow multiple draws from the binomial distribution which the random classifier is doing, the expected value of the maximum accuracy will be higher than the accuracy if we only allow one draw. The baseline is built on this principle and I agree with the authors that this is also straightforward to implement and can be used as a drop-in replacement for the simple random baseline. - The revelation that many published results do not, in fact, outperform this simple baseline is a potentially impactful finding.

Reasons to reject

One of my biggest concerns is as follows: the standard machine learning practice is *not* to rely on validation accuracy as the result to report. To give an example, if we train an image classifier, the standard practice is to evaluate the validation set at the end of each epoch, and then choosing the argmax for testing *once*. The test accuracy is typically what we care about, and we are only allowed to evaluate on it exactly once -- putting into the context of this paper, this means we are only allow to draw one realization of the binomial distribution (Eq (1)) once rather than $t$ times at *test-time*, which will reduce the proposed baseline back to the standard random baseline. Thus, even though on validation set, which is designed to be evaluated multiple times, the proposed baseline will be much stronger, it will make no difference at test time if standard protocol is followed. The authors seem to suggest that many reported papers do not properly use this elementary train-validation-test split and either report the validation accuracy only or evaluate on test sets multiple times -- while it is certainly not the authors' fault that basic data splitting protocol is not adhered to, I think that providing a stronger validation baseline seems to be the wrong question to answer when the real solution, which is not very difficult to implement and is virtually in any machine learning 101 course, is to use standard data splitting and to report test performance and allow for a single evaluation on test set. As such, I felt that the contribution of this paper is somewhat conditional on a flawed practice of not properly using test set being widespread, and to my opinion using the test set properly as told in the textbook is a much better solution (even the authors have acknowledged so in Sec 6 *"While it is best practice to report test performance on a truly held-out set of examples"*). While I think it's unfair to harshly penalize the authors for violations of such basic rules by the others, I think it's necessary for this paper to strongly emphasize this caveat.

Summary

This paper introduces a novel random baseline for evaluating in-context learning performance. In contrast to traditional random baselines that rely on prompts achieving the highest validation set performance, the proposed method leverages multiple classifiers to generate more general and robust evaluation results. Specifically, it first identifies the best-performing random classifier and then reports the expected performance derived from a variety of prompts. The experiments demonstrate that this expected random baseline provides stronger and more reliable baselines compared to simple random baselines.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

- This paper addresses a critical issue in prompt engineering and in-context learning. - Its motivation is simple and straightforward. - The authors present extensive empirical results. - However, I could not understand how the empirical results support the claims.

Reasons to reject

- The usefulness of the proposed method is unclear. - It appears to simply replace a random baseline, offering no significant improvement. - Additionally, with a small sample size (number of classifiers), t-tests and p-value comparisons may not be appropriate statistical tools. - Finally, the computational cost seems significantly higher compared to a simple random baseline. The authors should address these concerns.

Questions to authors

Question 1: What do the classifiers mean exactly? - As you mentioned, the model selection in prompt engineering could be different such as backbone models, input prompts, and hyperparameters. Question 2: I would like to see the change of results according to the input prompt quality. Suggestion - I believe this concept could be developed further to create a general evaluation method or metric for prompt engineering.

Summary

This paper proposes a well-justified stronger random baseline for in-context learning (ICL), presenting the simplest case-study for the new random baseline: ICL in the context of single-label classification tasks. The idea behind the baseline is taking the expected best performance across multiple random classifiers, where the actual strength of the baseline naturally depends on the choice of the population/validation set, as well as on the number of the evaluated random classifiers. These dependencies are also analysed in the paper with some practical tips provided to the researchers and practitioners. The importance of having such a more elaborate baseline especially for ICL-style experiments is nicely motivated and the post-hoc analysis on BIG-Bench Lite reveals that the choice of a different random baseline can even have large impact on how one perceives performance of the actual method/system.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The main idea is nicely motivated and elaborated on from multiple perspectives: (a) practical need; (b) theoretical justification, and (c) empirical findings. - The analyses delve deep into the main problems of the standard random baseline for ICL and how different choices of random baselines can provide different experimental conclusions. - Some practical tips have been provided, and the paper provides a nice balance between more theoretical insights and its downstream/practical value. - The paper is very well written and the main concepts and ideas have been well described and I enjoyed reading the paper.

Reasons to reject

- The choice of tasks could have been made more comprehensive so that the reader also gets a perspective related to task complexity, number of labels, and some calibration issues and properties of ICL (e.g., see https://arxiv.org/pdf/2309.17249 for some work on calibration for ICL) - The paper leaves too many directions for future work which it should have ideally explored already. For instance, at least some directions should have been investigated among the ones listed in Section 6: 1. more models 2. more tasks 3. prompt variations 4. scoring strategies. 5. other evaluation metrics 6. classification tasks with different properties (single-label vs multi-label or a small number of labels vs a larger number of labels). Put simply, more experiments would have strengthened the main findings of the paper and possibly the wider adoption of the proposed stronger random baseline. - I am also unsure if the proposal for the stronger random baseline holds in a standard setup where we do selection/optimisation on validation set multiple times and then just port a single best model to the test only once. The random baseline on the test set is then again the standard random baseline? I might be missing something here, so I would definitely like to see this discussed.

Questions to authors

This is not a question, but rather a suggestion. It would be useful to maybe provide a table or a simple plot that shows how expected maximum accuracy changes with respect to its crucial parameters (n and t) so that the reader gets a better overview of these dependencies.

Summary

This work proposes an alternative approach to LLM in-context learning evaluation against a random baseline by considering the performance of the best random baseline across multiple runs. The proposed method is shown to reframe the performance of several state-of-the-art large models on a subset of BigBench tasks, serve as a better predictor of test set accuracy, and provide more precise information about multi-prompt evaluation results.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1. The problem of robust evaluation studied in the paper is highly important, especially in light of other advances in the field 2. The proposed method is simple to understand and implement, yet provides a theoretically grounded way of estimating maximum performance across classifiers 3. The set of experiments is sufficiently extensive to demonstrate the benefits of the method in several settings

Reasons to reject

1. I am slightly concerned about the potential impact of the method, because in a significant amount of research, language models are compared to each other and not to a random baseline. The maximum random baseline might indeed outperform the accuracy of state-of-the-art models on some tasks, but often the point of comparison is to measure progress in capabilities on a certain task and not to check if the model is meaningfully better than random choice. For instance, p-values against random baselines might see limited applicability because evaluations frequently involve multiple models and therefore should be viewed as multiple comparisons from a statistical perspective. 2. The approach appears to assume that the ground truth distribution of class labels is uniform: unless I misunderstand something, the probability of the random classifier being correct is not $1/m$ if a label is chosen at random. I wonder if the proposed approach can be extended to accommodate for such cases; at the very least, it would be worth mentioning the potential class imbalance when describing the method

Questions to authors

1. Have you empirically evaluated the impact of class imbalance on the performance of a baseline that samples classes uniformly?

After reading the response...

...and the responses to the other reviewers, I conditionally adjust my score, where the condition is to really integrate the new clarifications and motivations into the main paper. Many thanks for taking the time to provide a well-written response!