Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

Summary

This paper attempts to address a forward-looking and challenging question: "Can we limit human supervision to easier tasks, yet enable the model to excel in harder tasks?" (referred to as Easy-to-Hard Generalization). Based on the observation that "evaluation is easier than generation", the authors propose to train a verifier on easy tasks, then make use of its generalization ability to supervise the generator on hard tasks. To harness the complementary strengths of outcome reward models (ORMs) and process reward models (PRMs), the authors introduce the Outcome & Process Reward Model (OPRM), so as to better utilize Easy-to-Hard evaluators. Through extensive experiments, the authors verify that easier-level evaluators maintain their effectiveness on harder tasks. Further experiments explore the use of the easy-to-hard evaluator as a reward model in reinforcement learning and underscore the potential of using easy-to-hard evaluation to improve easy-to-hard generators.

Strengths

* The problem this paper aims to tackle is promising and challenging. * The proposed approach is intuitive and has strong motivation. * This paper is well-written and presents clear ideas. * Through extensive experiment, the authors validate that the proposed method of "training a verifier on simple tasks, then leveraging its generalization capability to guide the generator on complex tasks" is workable.

Weaknesses

* **The definition of difficult problems could be further refined.** The scenario considered in this paper is how to enhance the model's ability to perform difficult reasoning tasks when humans cannot provide effective supervisory signals. Given that the model's capabilities vary across the seven subsets in the MATH dataset on which the experiments are based, the definition of difficult problems may be biased. For instance, the LLM performs significantly better on the Algebra subset than on Geometry and Number Theory. Therefore, the improvement at levels 4-5 may primarily result from the performance enhancement in Algebra as it inherently has some evaluation ability for Algebra level 4-5 problems). Thus, it would be better to display the performance of the proposed method in different subsets of MATH, as well as the performance at levels 4 and 5 under different subsets, to prove that it can help the model solve "truly difficult" problems (such as Number Theory level 5).

Questions

See weaknesses. I will consider raising my score if the authors can address my concerns.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

I think the authors have addressed their limitations.

Summary

This paper addresses the issue that humans cannot always provide helpful demonstrations or supervision on tasks beyond their expertise. Based on the observation that evaluation is easier than generation, the authors propose "easy-to-hard generalization," training a verifier on easy tasks and leveraging its generalization ability to supervise the generator on hard tasks. Experimenting mainly in math reasoning tasks, they demonstrate that easy-to-hard generalization from evaluators can enable easy-to-hard generalization from generators.

Strengths

The paper introduces the Outcome & Process Reward Model (OPRM), which harnesses the complementary strengths of ORMs and PRMs. Experiments show that OPRM is more efficient. It also conducts a systematic experimental setup, testing various generators and evaluators, as well as optimization algorithms (BoN, Weighted Voting, and RL). It provides numerous experimental analyses in scalable alignment.

Weaknesses

While the paper is well-written and presents a solid analysis, several weaknesses need addressing: 1. What is the difference between "easy-to-hard" and "weak-to-strong"? You state that human supervision is available but not reliable in the weak-to-strong setting, but in the [OpenAI paper](https://cdn.openai.com/papers/weak-to-strong-generalization.pdf) it says, "We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization." In that study, they finetune GPT-4 with a GPT-2-level supervisor. Can a GPT-2-level supervisor be seen as a verifier on easy tasks? The novelty should be considered. 2. In Table 1, the Full ICL setting performs worse than the Easy-To-Hard ICL setting. How do you explain this? The intuition is that sampling from both easy and hard exemplars may help solve hard problems more effectively than just demonstrating easy exemplars. Although the quality of PRM800K is lower than MetaMATH, your explanation for why the Full ICL setting is worse than the Easy-To-Hard ICL setting is insufficient (line 198). 3. Can you report the average accuracy of verifying each step in a solution when you evaluate the evaluators? Section 3.5 does not explain why this is not reported. 4. When citing images and sections, consider adding automatic jumps for easier navigation.

Questions

1. What is the difference between "easy-to-hard" and "weak-to-strong"? 2. In Table 1, the Full ICL setting performs worse than the Easy-To-Hard ICL setting. How do you explain this? 3. In Table 1, the Easy-To-Hard SFT setting performs slightly worse than the Full SFT setting, which is expected. Do you think the evaluation of the Easy-To-Hard setting is influenced more by the format of problems or by the true grasp of the principles of solving hard tasks? Are there any better evaluation methods? This is an open question, and I would appreciate your thoughts on it.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The paper may lack innovation regarding the concept of "easy-to-hard generalization" and the method to achieve it. Many experimental conclusions appear to merely reproduce or support existing works and lack depth.

Summary

In this paper, the authors propose *easy-to-hard* generalization, which is to train a reward model on simpler tasks and then use it to evaluate the solutions for more difficult tasks. They have conducted in-depth studies on MATH, and also demonstrated effectiveness on the coding benchmark APPS. This work serves as a nice proof-of-concept study of "evaluation is easier than generation", and suggests a new way toward scalable alignment without human supervision.

Strengths

- The idea of *easy-to-hard* generalization is inspiring, and it's nice to see that the idea works out on the challenging MATH dataset. - The authors have conducted sufficient experiments and detailed analysis, which makes the paper quite worthy to be read and referred to.

Weaknesses

- The definition of "easy" v.s. "hard" is not very clear, and it seems that the proof-of-concept experiments rely on the structure of the benchmarks, as the MATH dataset has 5 divisions of difficulty and APPS has 3. However, when there are not explicit tags of difficulty in a benchmark, what is the authors' definition for "easy" and "hard" in terms of each data sample? - Following the above weakness, a natural question for the authors is to demonstrate the practical value of this work when compared with recent work that scales synthetic data for MATH (e.g., [1]). Can the massive synthetic data be viewed as a mixture of easy and hard problems + the corresponding solutions? If we treat such mixed data as the easy part and train RMs on it, can we expect similar easy-to-hard generalization? Which kinds of ability would be "unlocked"? [1] Improve Mathematical Reasoning in Language Models by Automated Process Supervision. https://arxiv.org/abs/2406.06592 - According to the figures shown in the paper, the "Weighted Voting w/ RM" method always yields better performance as N increases. By comparison, "Best-of-N w/ RM" and "Majority Voting" can plateau or even become worse when N increases from 512 to 1024. Does the weighted voting with RM guarantee the increasing performance, or is it just by accident? Would be great if there are formal explanations to this. - While scaling the sampling times N has seen improvements, are there certain problems whose correct solutions are never sampled by the LLM when letting N be very large? - More case studies would be beneficial to provide a more intuitive understanding of how the RM on easy problems generalize to hard ones. - The authors have conducted in-depth analysis about the comparisons between PRM, ORM, and OPRM on MATH. While I appreciate the experiments, I wonder what the effect of PRM/ORM w.r.t. easy-to-hard generalization is. For example, is it true that we should always adopt PRM when it is possible to get process-based supervision (Lines 219~221)? If this is true, would the results on APPS be better when we could access process-based supervision for code (for example, treat the interleaved comments in a code snippet as that)?

Questions

The idea of "evaluation is easier than generation" is appealing, and it seems that the idea draws inspiration from the assumption that P < NP. However, for some tasks it seems that evaluation shares a similar level of difficulty with generation. For example, let there be N integers: A_1, A_2, ..., A_N, and the task is to multiply them altogether: A_1 * A_2 * ... * A_N = ? . While doing multiplication is no easy task for LLMs, it seems that evaluating whether some potential answer is the result of A_1 * A_2 * ... * A_N is as difficult as generating the answer of the multiplication, since in either way, one should do it a serial manner (for i in 1,2..N). Any thoughts/evidence on this? And broadly speaking, are there cases when evaluation is as difficult as generation, or evaluation is even harder than generation? It would be great if the authors could shed light on this.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have adequately addressed the limitations.

Summary

- This paper studies the question of how a system can be improved when the performance of a system has surpassed human performance on a task. - As a testbed, the paper uses problems from the MATH dataset which have been sorted into 5 levels by difficulty. - The authors first train process-supervised reward models on level 1-3 problems on the MATH dataset. - They then use the reward models learned from easy problems to supervise policies on hard (levels 4-5) problems on MATH. - They find that the reward models substantially improve the performance of the policy on hard tasks when used as either reward models in RL or as re-ranking models during inference, despite being only trained on easy problems.

Strengths

- The extensive comparison of different training methods (ReST, DPO, PPO) is useful even outside of the context of the research question. - The methodological decision to compare both re-ranking and RL on hard problems is very sound and makes me more confident in the conclusion of the paper. - Although there have been several recent papers on easy-to-hard generalization that establish that easy-to-hard generalization is possible, I think the experimental setup here takes a different angle by showing that the *reward models* learned on easy tasks transfer to harder tasks *in multiple ways* and specifically that evaluators generalize better than generators to hard tasks.

Weaknesses

- The differences between the comparison categories in many cases are small, on the order of 1-2 percentage points. I also did not see any error bars or variance estimates (except maybe in Figure 4, though this is unclear). This makes assessment of the scientific validity of the results a bit more challenging. - The conclusions were demonstrated on only two tasks and both tasks were formal reasoning tasks. Would the conclusions transfer to natural language reasoning tasks?

Questions

- I found nearly all the tables in the paper hard to read / extract information from. - It isn't clearly indicated (or at least I couldn't tell) how many times each model was trained, were there multiple runs, etc. I see what looks like error bars on some of the plots, but no explanation of these is given.

Rating

8

Confidence

4

Soundness

3

Presentation

2

Contribution

4

Limitations

N/A