Summary
The authors provide a simulator for experiments with LLMs that aim to learn from human-feedback, in particular, human binary comparisons. This allows researchers to run exploratory experiments with, e.g., RLHF, quickly and cheaply, without having to collect human data. The main contribution is the open source library that includes various components needed for these type of experiments. In the paper, the authors evaluate their simulator; using end-to-end validation, validating subparts of the library, and showcasing how it can be used to produce findings match those found when using human feedback.
Strengths
Learning from human feedback is an important topic that has long been hard to study for any but the most well-resourced labs. Alpaca Farm will enable more research on this topic
Open source software that enables other researchers is always great.
The paper is well written.
Thorough validation.
The validation results, especially Figure 2, are impressive.
The finding in Figure 3 (reward model over-optimisation only happens with strong inter-rater disagreement) is intriguing (seems worth further study)
Weaknesses
From more to less relevant concerns. I will raise my score if these are addressed sufficiently in the rebuttal.
The paper could explore the limitations of Alpaca Farm more. I don't actually believe that Llama-7B with some some instruction fine-tuning and a bit of PPO is comparable to GPT-3.5; you probably don't believe this either. However, Table 1 shows a winrate of 55% of "PPO". This is probably because your evaluation instructions are relatively easy, only single turn, you don't factor in adversarial inputs, and so on. This is fine, but it should be discussed more in the paper. In general, the paper would be better if it elucidated the limitations of your simulator a bit more, as there clearly are some. Try to break it, show not only where it works, but where it stops working.
There has been recent work that shows that MTurk crowdworkers use AI a lot. https://arxiv.org/abs/2306.07899 . So, it's not wonder that MTurk evaluations agree with the evaluations from you GPT evaluator models. How do you expect this to affect your results?
Figure 4: this experiment would be better if you used humans to evaluate the winrate on the demo. The x-axis is the evaluation in Alpaca Farm. The y-axis should be the "realistic evaluation", which means realistic data AND human evaluation.
Why does Figure 2 have 10 data points? What are these data points? Comparing to Table 1, it seems like you included GPT-4 on Figure 2; this seems misleading, as this only measures the evaluation part of your simulator, and not the training part. You should show a Figure where you only include models that were actually trained on the simulated preferences.
I have some worries thinking about that the training signal (pairwise comparisons) and the evaluators (pairwise comparisons) come from the same API-models. This might lead to overfitting to these particular evaluators. However, in practicse, this doesn't seem to be an issue (see Figure 2). Of course, if you work with humans, you may also use the same humans for training signal and eval (although best practice would probably be to use different humans)
Paper could be shorter, a bit much repetition.
-------
EDIT: some of these points have been convincingly addressed in the rebuttal, so I raise my score
Questions
Why does Figure 2 have 10 data points? What are these data points? Comparing to Table 1, it seems like you included GPT-4 on Figure 2; this seems misleading, as this only measures the evaluation part of your simulator, and not the training part. You should show a Figure where you only include models that were actually trained on the simulated preferences.
There has been recent work that shows that MTurk crowdworkers use AI a lot. https://arxiv.org/abs/2306.07899 . So, it's not wonder that MTurk evaluations agree with the evaluations from you GPT evaluator models. How do you expect this to affect your results?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper could explore the limitations of Alpaca Farm more. I don't actually believe that Llama-7B with some some instruction fine-tuning and a bit of PPO is comparable to GPT-3.5; you probably don't believe this either. However, Table 1 shows a winrate of 55% of "PPO". This is probably because your evaluation instructions are relatively easy, only single turn, you don't factor in adversarial inputs, and so on. This is fine, but it should be discussed more in the paper. In general, the paper would be better if it elucidated the limitations of your simulator a bit more, as there clearly are some. Try to break it, show not only where it works, but where it stops working.