Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding

Summary

This paper proposes a method to apply Monte-Carlo Tree Search (MCTS) decoding over PPO-trained policy and value models for controlled text generation task. The key idea is to use the value model produced from PPO training as the evaluation function in MTCS decoding. Empirical evaluations are conducted on 4 text generation tasks. The proposed method outperforms a baseline of direct decoding with PPO policy model, as well as a best-of-n decoding baseline. Further ablation experiments are analyzed, including the effectiveness of the value model vs. reward model, the effectiveness of the MCTS vs. decoding with stepwise value, MCTS vs. more PPO, and so on.

Rating

6

Confidence

2

Ethics flag

1

Reasons to accept

1. In general the idea of this paper is clear and straightforward. Empirical evaluations demonstrate the effectiveness of the proposed method over several different controlled text generation tasks. 2. The proposed method is an inference-time algorithm, which means no further SFT, RLHF needed. It should be easy to apply to any LLM after RLHF.

Reasons to reject

One concern I have is the efficiency of the proposed method. I like the discussion of the inference time overhead in Sec. 5. But I also think it is not adequate. I would love to see some empirical numbers like wall-clock time. I think this should be further explained.

Questions to authors

1. I think some writings can be further polished for better fluency and consistency. For example, in the explanation of Eq 3 in page 5, $\bar{V}(s)$ does not appear in the equation. 2. Table 1: any idea why PPO+best-of-n is worse than PPO (51.47 vs. 52.44) in % of desired sentiment for positive sentiment?

Summary

Quality: 1. The paper presents a novel method (PPO-MCTS) for controlled text generation by combining Monte-Carlo Tree Search (MCTS) with Proximal Policy Optimization (PPO) models. 2. The method is theoretically well-motivated and empirically evaluated on multiple tasks, showing significant improvements over baselines. 3. The experiments are comprehensive, and the results are analyzed in-depth with relevant ablation studies. Clarity: 1. The paper is well-written and structured, making it easy to follow the proposed method and the experiments. 2. The background and notations are clearly explained, aiding in understanding the technical details. 3. The figures and tables effectively illustrate the key ideas and results. Originality: 1. The idea of using the value model from PPO for guided decoding with MCTS is novel and has not been explored before. 2. The paper introduces a critical modification to the MCTS algorithm (initializing the Q of children actions from the V of their parent node) to improve exploration. 3. The work contributes to the under-explored area of utilizing value models for inference-time improvements. Significance: 1. The proposed method achieves substantial improvements in generating preferable text across multiple tasks, including sentiment steering, toxicity reduction, knowledge introspection, and creating helpful and harmless chatbots. 2. The results demonstrate the promise of search algorithms like MCTS even on top of aligned language models from PPO. 3. The work highlights the valuable potentials of the value models, which are often discarded in practice, and recommends saving them for enhanced inference.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. The paper proposes a novel and theoretically well-motivated method (PPO-MCTS) that combines MCTS with PPO models for controlled text generation, which is a significant contribution to the field of language modeling and controlled generation. 2. The extensive experiments and analysis on multiple tasks provide strong empirical evidence for the effectiveness of the proposed method, demonstrating its practical value and potential for real-world applications. 3. The paper explores an under-explored area of utilizing value models for inference-time improvements, unlocking the valuable potentials of these models, which are often discarded in practice. 4. The work is well-written, clear, and easy to follow, making it accessible to a broad audience in the computational linguistics and natural language processing communities. 5. The proposed method addresses an important problem in controlled text generation, which is a critical area of research with potential impacts on various applications, such as dialogue systems, content generation, and language model alignment.

Reasons to reject

1. One potential concern is the computational overhead introduced by the PPO-MCTS method during inference time. 2. While the authors provide some discussion and mitigation strategies, a more in-depth analysis and comparison of the computational costs and trade-offs with other methods could further strengthen the paper. 3. The font of figures is too small.

Summary

This paper proposes using the value model to guide text generation from PPO policy, particularly in the form of Monte-Carlo Tree Search (MCTS). The experimental results demonstrate the effectiveness of value-guided MCTS in controlled text generation tasks. The idea is simple, but it is a good contribution to reminding people to use value models that have not been utilized in practice. The paper is clearly written and well-organized.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

* Guiding with the value model is simple but has been overlooked in practice. It is a good contribution to bring this direction to attention. * The experiments on four text generation datasets and the well-designed ablation studies demonstrate the effectiveness of the proposed decoding method. * The paper is clearly written and well-organized.

Reasons to reject

* The advantage of MCTS itself is not verified enough. The ablation study "Do we need MCTS?" partially shows the effectiveness of MCTS but the baseline, which decodes based only on values, seems to be weak. Why not additionally use $p_{\theta}(x_t|w, x_{<t})$, as in existing controllable text decoding methods, e.g., FUDGE [1]? In addition, I would like to see the value-model guiding combined with other search algorithms, such as beam search. * DPO, which is widely used these days, does not accompany a value model. This situation may reduce the utility of this method. [1] FUDGE: Controlled Text Generation With Future Discriminators. 2021. NAACL.

Questions to authors

* Why is there no human evaluation conducted in helpful and harmless chatbots? * I understand that using only the value model is the benefit, but did you try MCTS based on the reward models used for the evaluation in each dataset? If a user has a clear idea of the reward he or she wants to maximize, it may be most effective to generate sentences using that reward model rather than a value model, as in existing methods. * Is there any way to apply the proposed decoding in DPO as well?

Summary

The PPO objective during the RLHF training simultaneously learn two models: a policy model and a value model. Often, the learned value models are ignored and only the policy model is used during inference. The paper proposes a novel guided decoding algorithm called PPO-MCTS which integrates the value model to work with the policy model during the inference time using Monte-Carlo Tree Search (MCTS). The paper is very well written and nice to read. The proposed method is thoroughly evaluated on four different controlled generation benchmarks. The results demonstrate the advantages of PPO-MCTS compared to PPO-only approaches, using both automatic and human evals.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The proposed approach is simple, yet very effective. The chosen text generation benchmarks are very relevant for the study and the results clearly show the advantages of PPO-MCTS compared to PPO-only approaches. The LLM community will find the paper very interesting. The paper is well written. The authors have done a great job of clearly motivating the proposed approach.

Reasons to reject

There are a couple of questions (see below) that were left unaddressed. It would be great to have the authors respond to them and also improve the paper accordingly. * I did not understand why the PPO-MCTS approach was not directly compared with prior guided decoding approaches using either hand-crafted rules or separate discriminative models as value functions. I feel the value functions specifically tailored for the generative LM (as used here) will work better than prior guided approaches, but it would be important to show this comparison. * It would be interesting to see the cost comparison between PPO and PPO-MCTS during inferences. Is it feasible to use PPO-MCTS when generating a long sequence? * Additionally, the paper would benefit from comparison to DPO (https://arxiv.org/abs/2305.18290) or Slic (https://arxiv.org/abs/2210.00045) trained models. But these comparisons might be okay to skip in this paper.

Questions to authors

Please see my three points/comments raised in 'Reasons to Reject'. It would be great to have you respond to them. I would be happy to reconsider my scores.