Iterative Reasoning Preference Optimization

Summary

This work proposes an iterative training algorithm that enhances a model's Chain-of-Thought (COT) capabilities in reasoning tasks by combining self-improvement with preference optimization. The algorithm employs ground truth labels as supervision signals to evaluate model-generated responses, which are then incorporated into subsequent training iterations. Experiments on the GSM8K, MATH, and ARC benchmarks show significant improvements compared to the baseline, with continuous enhancement observed over multiple iterations.

Strengths

This paper introduces preference optimization into the self-improvement of reasoning tasks, yielding excellent results.

Weaknesses

1. In the experiments, this work conducts experiments on the training sets of the GSM8K, MATH, and ARC benchmarks, and evaluates models on the corresponding test sets. All tests were completed on held-in data, with no performance results provided for the model on held-out tasks. 2. This paper emphasizes the importance of preference optimization, distinguishing it from other iterative training methods like Rest-EM [1], which should be considered as a baseline. [1] Singh, Avi, et al. "Beyond human data: Scaling self-training for problem-solving with language models." arXiv preprint arXiv:2312.06585 (2023).

Questions

Please refer to the weaknesses section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

--

Summary

The authors propose iterative RPO method for reasoning tasks. In particular, iteratively, the model at hand will be prompted to generate many CoT reasoning, and the ones that align with the true answers will be used as chosen, the other generated samples as rejected, for a DPO+NLL loss. The authors conduct experiments on GSM8K, ARC, and MATH to demonstrate the superiority of the proposed algorithm.

Strengths

The reasoning task at hand is important to the community, and the proposed algorithm shows promising improvements.

Weaknesses

The novelty between the proposed algorithm and self-rewarding language models seems to be marginal. It also occurs to me that more ablation studies should be conducted, see **questions** for detail.

Questions

1. My main concern is where the improvement comes from. In iterative RPO, there are (at least) three sources of improvements (a) more data, and (b) the model is updated iteratively, and (c) preference optimization is used in addition to SFT. - More data. While in table 1, iterative RPO with twice as much data is compared to STaR. It seems that the data is simply duplicated, but what if we use extra data generated by the model? Currently we are simply training more epochs. - What if we take the positive and negative samples generated by iterative RPO, and used it to train DPO or SFT? This tells us how much (b) or (c) helps, respectively. - Currently, is the DPO baseline trained on just (x, y) or (c, x, y)? 2. A less important question is the importance of NLL. My intuition on using NLL is that it regularizes the model from hacking the reward too much, but conceptually, choosing a larger $\beta$ does the same thing -- it enforces a larger KL regularization between the SFT model and the trained model. If you vary $\beta$, will figure 2 still hold?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Limitations have been discussed.

Summary

This paper proposes a novel method for preference optimization for reasoning tasks. Their method involves prompting models to generate the CoT reasoning steps and answers for a set of reasoning task inputs, then labeling samples as correct or incorrect using the ground truth outputs, and training the preference model. The authors use standard DPO loss with the addition of a negative sampling loss using samples for which the model generates incorrect outputs. They train models iteratively, using the previous generation as the base model to generate new outputs at each step. They observe that using this method they are able to improve significantly over DPO for three reasoning tasks.

Strengths

RPO demonstrates performance boosts on a variety of tasks, outperforming DPO and SFT. Improving model performance for reasoning tasks has shown to be a challenging task, and this method shows promise for improving model performance in this area.

Weaknesses

1. Although the authors present ablations on the number of data points for RPO, they do not present experiments with simply training the model for longer. This is true for the comparisons to DPO as well. Though iterative training likely is playing a role in the performance of the model, for the sake of careful analysis, it would be good to compare the performance of a model trained for multiple generations vs a model trained for only one generation but an equivalent number of steps. 2. I was not able to find details on the hyperparameters used for DPO or SFT or how optimized they were. In contrast, the hyperparameters for RPO are chosen carefully. While it's unlikely all performance boosts are due to this, it should be clear how hyperparameters were chosen for the sake of comparison and reproducibility. 3. Though the comparison of DPO with and without NLL shows that the loss for rejected samples decreases when NLL is applied, it does not demonstrate why this is important for model improvement. A more effective analysis would compare the mistakes made by the model with and without this loss (e.g. FNR/TNR).

Questions

1. What hyperparameters are used for DPO and SFT? Are they tuned with the same care as those for RPO? 2. Do you explore the reasoning steps generated by models? How often are they actually correct? 3. What is the cost of training models using this method?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors address limitations in the appendix.

Summary

The paper introduces a novel approach to improve the performance of language models on reasoning tasks through iterative preference optimization. It proposes an iterative method that generates multiple reasoning steps and final answers, constructs preference pairs based on the correctness of the answers, and then optimizes these pairs using a modified Direct Preference Optimization (DPO) loss combined with a negative log-likelihood (NLL) term. This iterative process results in progressively improved model performance. The approach demonstrates significant accuracy improvements on the GSM8K, MATH, and ARC-Challenge datasets using the Llama-2-70B-Chat model, outperforming other models that do not rely on additional datasets. Key contributions include the iterative application of DPO with an NLL term for reasoning tasks, comprehensive experimental validation, and performance gains without the need for additional human-annotated data.

Strengths

1. The paper is well-written, and easy to follow. 2. The idea is clean and the contribution is clear. 3. The evaluation and improvements are convincing and significant.

Weaknesses

I do not see significant weakness.

Questions

Although the authors have discussed other related iterative preference optimization methods (e.x. Iterative DPO, Self-rewarding, SPIN), why not include their results in the evaluation tables?

Rating

8

Confidence

2

Soundness

4

Presentation

4

Contribution

4

Limitations

See Questions.