Deductive Beam Search: Decoding Deducible Rationale for Chain-of-Thought Reasoning

Summary

This paper propose a new prompting method to improve the reasoning capability of llms. Specifically, instead of using greedy search, this paper proposed to conduce beam search based on the step level with the help of an additional NLI model to provide the beam ranking signal. Experiments on several datasets show that this method can achieve marginal improvement over the majority voting baseline.

Rating

4

Confidence

5

Ethics flag

1

Reasons to accept

This paper combines a sound search algorithm (i.e., beam search) and the CoT process to improve the model's complex reasoning capability.

Reasons to reject

1. Combining advanced search algorithms into the reasoning process of large language models is a hot research topic in the community. There are several existing works that applies beam search, tree search, or MCTS into the reasoning process. This paper is missing the comparison of these baselines. I could think of a key difference between this work and other baselines is that the proposed deductive verifier is task-independent while other methods either use signal from the model itself (e.g., PPL) or a task-specific scorer. This is an important feature. However, from the results in table 2, it seems like the proposed verifier is not good enough since the improvement over majority voting is quite marginal. I would suggest the authors to pay more attention to the general verifier and further improve it. 2. Some of the claims in the paper are over-claims. For example, Many existing works are indeed working on addressing the middle errors hence the first claim in the abstract might not be accurate. Also, from the view of a higher level, all the search algorithms (including self-consistency, beam search, ordinary tree search, MCTS) are all trying to alleviate the error caused by the greedy decoding. The key difference is the signal used to conduct the sampling. Hence, one should directly compare these methods instead of adding them together. For the extreme case, one should be able to combine all of these methods together but that is not helpful in a real system since it is too heavy.

Questions to authors

1. What is the generalization capability of the proposed verifier. Can you test on a more general benchmark such as big-bench hard? 2. Can you elaborate more on the implementation of the self-consistency baseline including all the important hyper-parameters?

Summary

This paper proposes Deductive Beam Search (DBS), which integrates chain-of-thought (CoT) and deductive reasoning with step-wise beam search for LLMs. To train such verifier, the authors propose labor-free data construction methods by replacing numbers in the equations with randomly generated ones and randomly selecting reasoning steps. The experiments show the effectiveness of the proposed method and the paper is quite easy to understand.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

* The proposed method combines deductive reasoning with beam search exploration * Propose a labor-free data construction method for training the verifier, which generates grounding errors and logic errors.

Reasons to reject

* Lack of further analysis of different reasoning steps

Summary

In this paper, the authors propose Deductive Beam Search (DBS) to alleviate accumulated errors in LLM CoT reasoning. Specifically, it integrates the step-wise beam search method into LLM CoT and designs a deductive verifier to verify intermediate reasoning step. Experiments results show that DBS improves the base performance of LLMs on various task and can detect diverse and subtle reasoning errors and robustness on different model scales, proving its superiority over previous methods.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. The integration of step-wise beam search and a deductive verifier with CoT reasoning is a novel and effective approach to mitigating accumulative errors in multi-step reasoning. 2. Comprehensive experiments on diverse reasoning tasks and LLM scales demonstrate the broad applicability and effectiveness of DBS. 3. The verifier analysis provides good insights into its error detection capabilities, robustness, and impact on the deducibility of generated reasoning paths.

Reasons to reject

1. Paper does not provide a detailed analysis of how different beam search hyperparameters impact this trade-off and the final performance. 2. The computational efficiency of DBS compared to baselines is not discussed in depth. Given the additional overhead of beam search and verification, an analysis of the inference time would be informative. 3. The paper focuses on the deductive reasoning paradigm, but many real-world reasoning tasks also involve inductive and abductive reasoning. Discussing DBS's potential limitations or extensions to these other forms of reasoning could broaden its impact. 4. There are significant differences among various types of reasoning tasks. DBS is more effective in Arithmetic Reasoning task. The paper lacks analysis and discussion in this regard.

Questions to authors

The idea in DBS is straightforward, and the results show that it is a promising approach. I have some questions regarding the DBS method and experiments: 1. The paper only using two relatively weak LLMs to evaluates DBS on eight datasets. Considering the submission time, the state-of-the-art accuracy should have already surpassed the results reported in the paper, which may undermine the persuasiveness of the work. 2. DBS categorizes the errors in the reasoning process into several types, but these do not cover all the errors encountered in real-world scenarios (which is also mentioned in the paper). If the reasoning process is deductive reasoning, DBS aligns with expectations. However, many real-world reasoning tasks also involve inductive and abductive reasoning. Can you discussing DBS's potential limitations or extensions on these other forms of reasoning? 3. What is the model's strategy for handling scenario where multiple errors coexist simultaneously in one intermediate process ?