Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

Summary

The paper introduces Chain-of-Preference Optimization (CPO) to improve reasoning in LLMs. CPO utilizes preference data generated during ToT to fine-tune models. This approach improves reasoning without increasing inference complexity.

Strengths

1. The idea of CPO is interesting. 2. CPO achieves enhanced performance without increasing inference complexity. 3. The paper is well-written and easy to follow.

Weaknesses

1. While CPO is effective across several tasks, its performance on more complex reasoning benchmarks, such as the MATH benchmark, remains unexplored. 2. Fine-tuning with reasoning paths generated by ToT may reduce the diversity of generated paths. A systematic investigation into how the generated reasoning paths change before and after fine-tuning is needed. 3. The effectiveness of CPO is heavily dependent on the quality of reasoning paths generated by ToT. If ToT produces suboptimal paths, the preference data might not be as beneficial. 4. The paper does not adequately address the applicability of CPO to more advanced models, such as LLaMA3. 5. While CPO enhances performance, the paper lacks a deep analysis of the interpretability of the optimized reasoning paths. Understanding why certain paths are preferred could provide additional insights and improve trust in the model’s decisions.

Questions

NA

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper introduces Chain of Preference Optimization (CPO), a method to enhance mathematical reasoning in large language models by feeding step-level pairs rather than response-level ones into DPO objectives. CPO leverages non-optimal reasoning thoughts from tree-search processes to construct paired preference data at each reasoning step, where both responses share the same content until a certain step and then start varying. Experiments across seven datasets show CPO significantly improves performance compared to Chain-of-Thought approaches, achieving comparable results to Tree-of-Thought methods while being substantially faster during inference. The authors provide a comprehensive analysis of CPO's components and demonstrate its effectiveness in utilizing both preferred and dispreferred thoughts in the reasoning process.

Strengths

- The experiments demonstrate that CPO is effective across different tasks including question answering, fact verification, and arithmetic. - This paper conducts an extensive analysis of several factors that can influence the model performance. The insights on scaling effects, data mixture, and dispreferred information can be useful. - The paper is well-written and easy to follow.

Weaknesses

- The major concern is that despite being effective, the method is basically a new use of the DPO algorithm with different inputs (converting response-level pairs to chain-level) and thus it's fundamentally still DPO. - In section 6, the authors have explained why chain-level optimization is important. However, according to the explanation, token-level DPO is more natural in locating errors and avoiding LCP gradient cancellation. Then according to your theory, why not directly use token-level optimization, which is exactly what PPO does, but an intermediate chain-level? I suppose there exists an efficiency-effectiveness trade-off in these settings.

Questions

See weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

See above

Summary

This paper presents a method called Chain of Preference Optimization (CPO) that fine-tunes large language models (LLMs) using the tree-of-thought (ToT) method to improve the performance of chain-of-thought (CoT) decoding. CPO aligns each step of CoT reasoning paths with those of ToT by leveraging preference information in the tree-search process, resulting in similar or better performance than CoT alone while avoiding the significant inference burden. Experimental results demonstrate its effectiveness.

Strengths

Innovation: The method of constructing preference dataset with ToT has a certain novelty. Writing quality: Your paper is well-written, with clear and precise language and a smooth flow of ideas. The structure is reasonable, and the logic is sound, making it very enjoyable to read. Experimental analysis: This paper has made a full experimental demonstration.

Weaknesses

1. The experiment lacked a stronger baseline comparison. In my opinion, the author's method belongs to the self-imporvement methods, so it should be compared with such typical methods, such as the ReST and self-rewarding mentioned by the author in related work. 2. The ToT approach to dataset construction appears to be computationally expensive. 3. the author does not seem to provide a task-specific prompt templates (although the general guideline prompt is mentioned in line 141). This is not necessary, but I think the quality of building a preference dataset can be seriously affected by the prompt template of evaluation.

Questions

1. Why use DPO instead of other preference optimization algorithms? 2. Explain why the experiment lacks different self-imporvement baseline algorithms. 3. There may be a large number of negative samples in the preference dataset constructed by ToT making the dataset unbalanced. May I ask whether this will have a negative impact on the training stage?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations of the study and the possible negative social impact have been well documented by the authors.

Summary

The paper presents a novel method to enhance the performance of LLMs in complex problem-solving tasks using Chain of Preference Optimization (CPO). The authors propose fine-tuning LLMs using the search tree constructed by tree-of-thought (ToT), allowing CoT to achieve similar or better results without the heavy tree-searching during inference. The CPO method aligns each step of the CoT reasoning paths with those of ToT, leveraging the inherent preference information in the tree-search process. The authors provide experimental evidence of the effectiveness of CPO, showing significant improvements in LLM performance in tasks such as question answering, fact verification, and arithmetic reasoning.

Strengths

The paper presents a novel method, Chain of Preference Optimization (CPO), that significantly enhances the CoT reasoning ability of LLMs without increasing the inference load. It offers a convincing comparison with the ToT method, demonstrating that CPO substantially reduces inference time, as evidenced by the decrease in latency from 1749s/ins to 38s/ins on Llama2-7B. Furthermore, the paper shows that CPO not only matches but surpasses the ToT baseline in terms of accuracy across multiple datasets. This indicates that CPO effectively fine-tunes LLMs to generate more optimal reasoning paths, thereby improving their performance in complex problem-solving tasks. The research is significant as it provides a practical solution to the challenge of balancing inference complexity and reasoning quality in LLMs.

Weaknesses

The paper presents a promising approach to fine-tuning LLMs using CPO. However, there is a significant gap in the evaluation of the potential impacts of this fine-tuning process on other abilities of the pretrained LLMs. This omission could leave readers questioning the broader implications and versatility of the proposed method. Future work should consider a comprehensive evaluation of the fine-tuning process to fully understand its effects on the LLMs' overall performance.

Questions

LN35, what is the advantage of CPO comared to existing MCTS methods? And why? LN117, have you considered online RL that regenerates the chain-of-preferences using the updated LLM? Table1, I'm wondering how does the method perform on common reasoning tasks like GSM8K and ProofWriter?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper presents Chain-of-preference optimization (CPO) a self-supervised learning extension of Tree of Thought (ToT). Rather than use ToT during test time, which takes exponentially longer than end-to-end sampling, this paper proposes to use ToT at training time to annotate data for DPO fine-tuning and then use the DPO-tuned model end-to-end at test time. Notably, the DPO annotations are performed by the untuned model itself being prompted to label inferences as either useful or not. This differs from other approaches that are trained on only full successful reasoning paths. For certain QA and reasoning tasks, the paper presents experimental evidence that CPO yields stronger models than either (1) fine-tuning the same LM on positive examples (TS-SFT) as done in previous papers or (2) running regular chain-of-thought prompting.

Strengths

1. The approach is an intuitive and straightforward improvement to multi-hop reasoning that that is much faster than tree-based inference procedures. 2. The approach is appealing as it doesn't rely on correct labels nor external LLMs other than the LLM being fine-tuned on its own preference data. 3. The methodology is well explained and easily reproducible.

Weaknesses

1. While the authors run evaluation over several datasets, they only consider 300 questions per dataset which is quite small. A stronger evaluation would use the whole test sets. 2. Details about the baseline implementation are unclear. The authors refer to both "TS-SFT" and "TS-LLM" (l.206) which in the original paper refer to different things-- SFT refers to the model fine-tuned on training examples with reasoning traces pulled from both a gold-annotated dataset and/or reasoning traces from the model that led to correct answers. However, TS-LLM in the original paper refers to the result of an iterative refinement process. 3. The choice of datasets is somewhat odd. E.g. HotpotQA is meant to be evaluated against support documents, which might explain the very low scores. They also compare against approaches (ToT, TS-SFT) that were not evaluated on any of the QA or Fact Verification tasks under consideration. This paper did _not_ consider the Game of 24 or GSM datasets, which both ToT and TS-SFT did evaluate. Missing references: 1. Khalifa et al 2023: [GRACE: Discriminator-Guided Chain-of-Thought Reasoning](https://aclanthology.org/2023.findings-emnlp.1022.pdf) 2. Li et al 2023: [Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions](https://openreview.net/pdf?id=4D4TSJE6-K)

Questions

* What metrics did you use for the QA datasets? many of them use more than exact match, e.g. HotpotQA uses F1. * The discussion around ablations in 5.3 is unclear. Are the trends consistent across different models and datasets? * It is odd to have only considered up to 200 instances for constructing preference pairs. Why is this experiment limited to such a small maximum? The difference between e.g. 160 and 200 seems much less important than the difference between 200 and 1000 or 1000 and 5000. * How many total preference pairs do you end up training on? It would be helpful to include this number in the main body of the paper. * How stable is this approach to different prompts for state evaluation? Did you experiment with other prompts?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

yes

Thanks for the responses

Thank you to the authors for their careful rebuttal. I still have concerns about the evaluation given the authors have been selective about which results were shown in the rebuttal PDF. > we have included evaluations on the entire test sets in Table A This is greatly appreciated, though this only includes 3 of the 7 considered datasets in the paper. Were the results on the other 4 datasets not positive findings? Running on those datasets is clearly within scope because you included SVAMP and Fever in Table C. > added performance for the LLaMa2-7B base model on the full GSM8k test set in Table B Thank you as well for this result. Table B is a good start but these numbers do not constitute a full results that could be added to the paper. Why is it missing the other baselines? I appreciate the improvements by about a point over CoT. However, TS-SFT (and ToT) consistently improve upon CoT. For the other datasets you were outperforming CoT by 3-7 points pretty consistently, but for GSM it is only 1 point-- are the TS-SFT and ToT baselines somewhere inside this 1 point spectrum (raising questions about statistical significance), or are they stronger than CPO? Moreover, others have found (e.g. in the [Gemma report](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) that Llama-2-7B matches your 14.6% performance on GSM using just few-shot prompting. This suggests that CPO is not providing any benefits. > [Ablations] Thank you for these results, they have somewhat addressed my concern. Which datasets are the Figure C plots computed over?