Response to Reviewer 7aGY
Dear Reviewer 7aGY,
Thanks for your comprehensive and detailed suggestions for our work! We really value your comments on the training overheads and clarification of experimental details. We hope our responses below could address your concerns:
**1. On Training Overheads**
> Weakness 1: SPaR introduces additianal training overheads.
Thank you for your insightful comment! As the ultimate goal is to enhance the model's performance in instruction following, inference-time scaling is actually more efficient than traditional pretraining solutions or creating high-quality data with human annotators.
We have calculated the average number of expanded tree nodes in our framework and included these details in Appendix C. Specifically, with the LLaMA3-8B model, the average number of expanded tree nodes is around 3.8, which is within an acceptable range of added computation.
Moreover, to effectively understand the costs and gains, we have conducted an additional series of experiments controlling the maximum number of response generations (inference times) during the training data construction process. We perform these experiments on two models, LLaMA3-8B and Mistral-7B, and report the average performance on IFEval. Here are the results:
| Model | Method | 5 | 10 | 15 | 20 |
|---|-|--|--|--|--|
| LLaMA3-8B | Self-rewarding | 79.3 | 79.2 | 79.2 | 78.9 |
| | SPaR | 79.0 | 79.5 | 80.1 | 80.3 |
| Mistral-7B | Self-rewarding | 66.5 | 65.5 | 66.5 | 66.2 |
| | SPaR | 67.0 | 68.3 | 70.0 | 70.8 |
The results indicate that SPaR surpasses the Self-rewarding method when additional computational resources are allocated for inference. This highlights the advantage of our method in scaling inference-time costs to achieve superior performance.
**2. Clarification of Experimental Details**
> Weakness 2: Some experimental details are not shown in paper.
Thank you for your valuable suggestions! We have included the average number of search nodes in our experiments in Appendix C. For LLaMA3-8B, the number of average expanded nodes is 4.3, 3.7, and 3.4 across different iterations, demonstrating a decreasing trend as the model becomes better. The performance of LLaMA3-70B at each iteration, previously omitted due to space constraints, has now been added to Appendix D.1 Table 9.
> Question 1: line 527, GPT-4-Turbo or GPT-4o-mini?
The model is GPT-4-Turbo. SPaR-trained LLaMA3-8B-Instruct outperforms the GPT-4-Turbo on the IFEval benchmark. The performance of GPT-4-Turbo is derived from the original benchmark [1].
[1] Zhou, Jeffrey, et al. "Instruction-following evaluation for large language models." arXiv preprint arXiv:2311.07911 (2023).
> Question2: Can you compare the training cost of SPAR, Self-Rewarding and Meta-Rewarding?
We would like to clarify that the training costs for all three methods are nearly identical, as we have controlled the number of training samples to ensure fairness.
The inference times required for data construction vary among these methods. For instance, in the case of LLaMA3-8B, the average number of responses generated by Self-Rewarding and Meta-Rewarding methods is 5, whereas for SPaR, it is approximately 8.8. This increase is within an acceptable range. As mentioned in our response to Weakness 1, we have conducted experiments to compare the costs and gains.
**3. On Iterative Improvement**
> Question3: Does more iteration bring higher performance?
Thank you for your insightful comment! Additional iterations can generally improve the model's performance but with diminishing returns.
For instance, in our experiments with LLaMA3-8B, extending to a fourth iteration showed smaller improvements of an average of 0.3.
This can be caused by model capacity limitations or challenges in iterative DPO training.
Furthermore, most self-training methods, such as Self-rewarding and SELF, typically use three iterations. We follow this practice to make it easier to compare our results with these methods.