AutoPSV: Automated Process-Supervised Verifier

Summary

This paper introduces AutoCV, a novel method to create process reward models (PRMs). The approach involves training an an outcome-supervised verification model based on (S_{(1:t)}, y) pairs where S_{(1:t)} is the 1st~t-th steps in the response and y is the final correctness label of the entire response. The outcome-supervised verification model then generates process annotations by detecting confidence variations across reasoning steps. AutoCV combines outcome and process supervision, reducing the need for manual annotations or computationally expensive methods. Experiments across five datasets in mathematics and commonsense reasoning show a significant advantage of the PRM for best-of-N sampling. The method outperforms self-consistency baselines and outcome-supervised verifiers while requiring fewer tokens for annotations than existing approaches.

Strengths

- This work proposes a novel method to obtain process annotation for free from outcome supervision. The method is elegant and derived from a theoretical perspective. - Unlike previous work on PRM that only focuses on the math domain, this work shows effectiveness on commonsense reasoning. and natural language inference.

Weaknesses

Though the method is interesting, the experiments are not very convincing and the paper is not well presented. - I cannot get the point of section 4.2.1 since it's a bit confusing to me. Is there a reason why the method is specifically useful for "process calculation hallucination detection"? It seems to me that it's a general method to detect all faulty steps in a response, which is what a PRM is supposed to do. Also, I think the term "hallucination" may be overloaded, which originates from an open-ended generation. I suppose the setting in section 4.2.1 is simply "calculation error detection". - There is no baseline in the main experiment (Table 6&7). I'd suggest moving section 6.1 to the main exp, and results beyond the math domain as a separate subsection in section 5. - The setup part (sec 5.1) does not mention if the experiments are all in-domain. OOD evaluations are required to examine the generalizability. - From Table 8, AutoCV performs very comparable with MCTS-based methods with differences of <1% accuracy. The performance gain is marginal. - Table 9 may not be a fair comparison of efficiency between AutoCV and MCTS-based methods. Despite using fewer tokens, AutoCV requires training an ORM/OSV for annotation which could also be costly. Maybe the sum of GPU hours for training/inference would be fairer?

Questions

- It seems the verifiers are based on the Mistral series and in general its gain is less significant when the response generator is Qwen, especially on MATH where AutoCV best-of-5 underperforms self-consistency. Is it because the method lacks generalizability or does the method only work well when the response generator is weak?

Rating

7

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

See above

Summary

This paper proposes a new method (AutoCV) that bridges the gap in popular techniques for enhancing reasoning capabilities of LLMs. Prior work have proposed *verification models* (models that evaluate generated reasoning steps and rerank candidate responses) as a promising solution. Here the community has focused on two training paradigms for verification models: (1) outcome supervision (train on correctness of final answer; cheap but less effective) and (2) process supervision (train on labels for each reasoning step; expensive but effective). AutoCV attempts to bridge these two by first training an outcome-supervised model and then use it to annotate confidence for unannotated intermediate steps. Then, by calculating the relative confidence variation between steps, AutoCV generates process supervision data that can be used for training.

Strengths

AutoCV tackles clear limitations in prior approaches with some novel ideas: 1. Interesting use of how an outcome-supervised (OS) model (theoretically) implicitly learns process annotations: - OS replicates final answer’s correctness label across all steps, causing the model to implicitly learn values for partial paths. - Consequently, forces the optimal solution under MSE to be estimate of the probability of reaching the correct answer. 2. Overcomes clear efficiency limitations of previous automated process annotation approaches like MathShepherd and MiPS that use MCTS. The MCTS soln is to sample multiple complete traces at each reasoning step, and using the outcome to score the potential of this step (say k/N lead to correct answer).

Weaknesses

My concerns are mainly around the presentation, clarity, and significance of some results: **Presentation/Clarity** 1. Throughout the paper, there are several experiments using verifier at inference to rerank generations from a *response generator*. For instance, Section 4.1 and Table 3. However, there is no mention or analysis of the #samples being reranked? 2. Similarly, there is no mention (or analysis) of the aggregation function used for process-supervised models at inference time? 3. Section 4.1 and Table 4 attribute the performance disparity among OSVs (for Phi2 and Mistral) to training data quality. Particularly confusing, was why the training data was different at all for each model? From the description, it seems like the training data was generated from _a single_ (GSM8K fine-tuned) pretrained LLM. If not, how does using a stronger model to generate data affect performance? **Significance of results.** Table 6-7 describe the performance of a PSV finetuned on self-generated confidence variations of an OSV. From that, it seems like there is *very minimal gains* from using the PS data generated to enhance (finetune) the OSV model. Furthermore, when the PS data is used to retrain a new PSV, the performance even drops a few points (Table 10). 1. While one can argue that there is "minimal loss" in information from the OSV to a PSV, it concerns me that the quality of process labels might not be good. The generated labels add little-to-no value to overall performance? i.e., why use AutoCV to train a PSV at all if OSV is equally effective AND can provide process-annotations scores? 2. With minimal gains, the only advantage of AutoCV is that it is more sample efficient than MCTS based approaches for PSV training. Are there contributions beyond efficiency?

Questions

Note: Added most questions in the Weaknesses section. Additional: Could the authors comment on how this would work with Agent based reasoning traces? Particularly, the procedure to generate process supervision labels (described in Section 3.3) would not work because it makes the following assumption: `any step, including the final result, following a step containing an error is considered incorrect`. This does not hold true for agentic approaches that use environment feedback to correct their mistakes (think code generation w/ execution feedback).

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The authors propose AutoCV, a method for solving multi-step reasoning tasks with chain-of-thought prompting that involving automatically labeling each step of the multi-step process based on its likelihood of leading to the correct outcome, and combining an outcome supervised classifier (OSV) and process supervised classified (PSV) to come up with a more effective method. Experiments on 2 math and 3 reasoning datasets show that OSV + PSV outperforms self-consistency as well as just OSV across the board, albeit by a small margin in several cases.

Strengths

The paper is carefully written and provides good background for readers (like myself) not very familiar with the approaches considered. The idea of using the difference in the confidence of an OSV model at step t and at t+1 is interesting; it's perhaps a bit surprising that it actually works well, as it's not often easy to learn how good an early intermediate step is towards achieving the final goal. The findings in section 4 seem to lay out a good justification for the design choices behind AutoCV, although I found the discussion in 4.2 (hallucination in math reasoning) harder to follow. The performance on 5 benchmark datasets, as noted above, is a strength of the paper, even though in many cases the absolute improvement is small.

Weaknesses

Not having familiarity with the area, I wasn't able to judge the novelty of the work and the substance (i.e., whether there is enough new material to warrant publication). It seems to me that f_theta() being a good indicator of the probability of reaching the correct final outcome is something known from reference [22]. The new part here is looking at the delta of f_theta() between time steps t and t+1, as done in Eq (3). In terms of naming, I found the use of the phrase "confidence variation" not intuitive for referring to the change in the confidence from t steps to t+1. To me, confidence variation suggests how the confidence changes as one varies some parameter or some sampling, not as one takes another step. In a similar vein, I found "AutoCV" to not be an informative name for the system -- it doesn't connect well with the task of multi-step reasoning or with chain-of-thought prompting. That said, this is a subjective choice and it's up to the authors to decide if they want to look for a more intuitive name. As noted above, I found the discussion in section 4.2 somewhat confusing. Lastly, as also noted above, while OSV + PSV generally shows consistent gains over the OSV, the gains are often very small. This somewhat lowers the value of the proposed technique, but consistent gains remain a net positive.

Questions

Please see weaknesses above.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Reviewer Response

Thanks for your clarification. Most of my concerns have been addressed, but I think AutoCV still needs ground truth for verifier training. Yes, the process annotation stage does not require so, but the number of annotated data for verifier training may have an effect on the final result. Anyway, I am pretty sold by the approach now and will raise my score. I would love to further increase the score if the authors can show the effects of verifier capability on the final process reward annotation.

Re: Rebuttal by Authors

Thank you for the detailed and informative response! I am happy to see your willingness to reconsider the system name and for your efforts in explaining section 4.2 better. Also happy to hear that the small-looking gains are statistically significant; please make sure to include that in case it's not already mentioned. I remain in favor of accepting this paper.