V-STaR: Training Verifiers for Self-Taught Reasoners

Summary

This paper proposes V-STaR, an approach to boost small LMs' reasoning performance by combining two lines of existing work---1) Self-Taught Reasoner/Rejection Sampling Fine-tuning, where LMs are iteratively fine-tuned only on correct self-generated solutions and 2) Oversample-then-rerank, where multiple solutions are sampled for each test question and a verifier scores the correctness of each solution. Both in-domain and cross-domain evaluation results suggest the effectiveness of their approach across math reasoning and code generation tasks. The intuition behind the improvement is that the verifier training learns from mistakes in the incorrect generations discarded by the STaR/RFT training.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. A simple yet effective approach that combines the benefits of both STaR and verification to boost reasoning in fine-tuned smaller LMs. 2. The paper highlights the superiority of the DPO-based verifier over the previous ORM-based one, which is a novel discovery for me. 3. Improving reasoning in LLM is an important topic of widespread interest within the community. This paper will establish foundations for future work in this field.

Reasons to reject

1. While the improvements over the baseline are significant, the technical contribution is somewhat less exciting due to the straightforward combination of existing STaR and Verification techniques.

Summary

The paper presents an iterative approach to train LLMs to become better at math and code tasks by utilizing the models own generations. Concretely, the approach makes use of multiple samples generated by the model, keeping only the correct generations, and then training the model with this new subset. Additionally, the approach relies on a test time candidate selection model which is trained using the model generations using DPO, thereby further improving performance. The approach presents strong test accuracy improvement on popular math and code benchmarks.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- Simple method which works. Authors continually train generators by sampling trajectories and manually filtering the correct answers, thereby making the generator more tuned to output the correct answers. - Combined with a verifier which is trained on DPO, the VSTAR approach makes good use of the verifier during testing to further prune likely incorrect solutions. - Strong improvements on hard tasks such as GSM8k, MBPP (in-distribution) and MATH, HumanEval (out-of-distribution).

Reasons to reject

- It is not clear how many iterations ($T$) of training V-STAR is optimal. Authors use three iterations, but more study needs to be done on the impact of the number of iterations to find the ceiling of performance. - Ablations are needed to understand how many iterations ($T$) of data collections are needed for the verifier as well. As the generator gets better at each iteration, the verifier should get harder data, thereby improving its performance. - A small limitation of this paper is that the proposed approach is only verified against Llama 2 and CodeLLama. It is unclear whether this approach is general across several openly available models.

Questions to authors

_Questions_ - Fig 6 is missing the 95% CI done over 256 trials - it would be good to add them here. - Section 4.6 is confusing to me: why does the DPO trained verifier does poorly on generation? The model gets good at assigning log likelihoods to correct and incorrect trajectories, so wouldn't the generation reflect that too? - Adding the verifier in training not getting substantial gain is a bit obvious, right? The verifier initially is noisy as it doesn't do a good job at filtering the correct solutions. Did you try a schedule of training verifier with manually filtered data from generation, and then using it during training? _Suggestions_ - It would be interesting to measure the forgetting effect with more and more iterations. Does the model collapse when trained for increasing number of iterations? - Does any linguistic pattern of the solutions emerge with increasing iterations?

Summary

The paper modifies STaR (Zelikman et al., 2022) to make use of the sampled incorrect solutions by training an additional verifier that is used at test time to rank candidate solutions. To make use of both correct and incorrect solutions, the authors propose to train the verifier as a reward model using Direct Preference Optimization. During each iteration of STaR, the verifier is trained on a union of the collected samples so far from the policy model.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* Paper is well written with a clear motivation and contribution * Impressive performance boost on different reasoning tasks. * The direction of leveraging preference optimization approaches for learning from suboptimal solutions is interesting and exciting.

Reasons to reject

* Why use DPO? I feel the justification for using DPO is not clear, and no analysis is done to understand why it is performing better than standard ORM. * Lack of analysis on the trained verifier. It would be very interesting to see if the trained verifier was able to learn a process reward model (PRM) by only training using final answer labels.

Questions to authors

* How was the ORM trained? and what is the intuition behind why it performs very poorly compared to DPO? Khalifa et al., 2023 mentions that ORM can perform poorly if many of the correct solutions involve bad reasoning and the LM was able to reach these solutions arbitrarily. Do you have a similar observation? It would be interesting to draw connection with existing work. * Why train only for 3 iterations? What happens to the verifier performance if you train for more iterations? I'm surprised you do not have a plot showing verifier performance after each STaR iteration. **References:** Khalifa et al., "Discriminator-guided multi-step reasoning with language models." arXiv preprint arXiv:2305.14934 (2023).

Summary

The paper introduces a method that leverages the incorrect reasoning paths generated by LLM during SFT progression to train a Verifier. This Verifier, in turn, helps in selecting the correct answer during inference. The experimental results demonstrate a significant improvement in performance on both the math dataset and the code generation dataset, and the model also exhibits strong performance in transfer experiments.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. This paper pioneers the utilization of incorrect labels to train a verifier, showcasing its potential utility across various scenarios. 2. The experimental results reveal a significant improvement in performance for Llama2 across both the math and code datasets.

Reasons to reject

1. The effectiveness of the verifier remains underexplored. While the paper compares it with majority voting in their baseline, further investigation is warranted given the crucial role the verifier plays. For instance, it is imperative to determine whether the verifier outperforms self-evaluation when selecting answers [1] and during decoding [2] [3]. Failure to do so may leave the utility of the verifier incompletely assessed. 2. The transferability of the verifier remains unexplored. Demonstrating its ability to be transferred to other models would be valuable, as it would reduce the need to train independent verifiers for each model, thereby streamlining the process. 3. The method is tested on Llama2 whose reasoning ability is far from the sota model such as mistral and gemma, which may make the method less general. [1] Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs, EMNLP 2023 [2] Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Neurips 2023 [3] Self-Evaluation Guided Beam Search for Reasoning, Neurips 2023

Questions to authors

Please refer to questions.

Thanks for the response

Thanks for the detailed response and incorporating the suggestions. Adding the transferability results to the paper would be quite useful!

Reply to Rebuttal

Thank you for incorporating additional experiments, which partially address my concern. Taking into account the transferability of the verifier, which is beneficial for the scalability and applicability of the method, I have revised my score to 6. However, the self-evaluation experiments are not convincing. My initial thought was to not use the verifier that you trained at all, but to use the self-evaluation of the model itself as the verifier.