Response to reviewer rMGU
We thank the reviewer for the detailed and helpful feedback! We are happy that you find our method novel and effective! We address your concerns in the following texts:
**W1: the prompting method is overused**
Thanks for raising this important point. We would like to clarify that “problem” and “question” are different in our paper. Specifically, “problem” = “conditions” + “question”. For example, a problem “I have two apples, and he has three apples. How many apples do we have in total?” = conditions [“I have two apples”, “he has three apples”] + the question “How many apples do we have in total?”. Therefore, to determine whether the reconstructed problem is identical to the original one, we first decompose problems into “conditions” (multiple conditions per problem) and “questions” (one question per problem), and then conduct “condition comparison” and “question comparison”. In short, “problem comparison” = “condition comparison” + “question comparison”. The final feedback is formulated based on the results of “problem comparison”.
We believe there are some misunderstandings between caused by Fig 20. We apologize for the confusion and make a clearer Figure 20. To rectify solutions, we do take into account “question comparison” results. However, in Fig 20, the “question comparison” results tell us that the question is correctly understood by models. Therefore, the final feedback in Fig 20 does not contain the result “question comparison” as the model did not make mistakes here. The final feedback will include the results of “question comparison” if mistakes are found in this process.
We apologize for the confusion and add clarifications accordingly in the paper.
**W2: the flaw of the "dialog system"**
Our revision step only uses the history of original wrong answer, which is the same as Self-Refine[1]. The core of RCoT is to let LLMs check the answer and rectify it by themselves, so it is necessary to provide the previous answer. It’s worth noting that reconstruction, decomposition, comparison and revision are independent. Each stage only utilizes previous results but not conditions on previous prompts. We argue that “stateless” or not should not be the “flaw” of the method or experiment design, since all methods (RCoT, CoT, Self-Consistency, Self-Refine) only utilize LLM itself without external intervention. RCoT and Self-Refine enables LLM to act as humans that can check and revise answers by themselves to achieve the correct answers.
[1] Madaan, Aman, et al. "Self-refine: Iterative refinement with self-feedback." arXiv preprint arXiv:2303.17651 (2023).
**W3: the gain of providing the reason seems to be moderate**
We hope our clarification in W1 could help address the understanding of Table 2. The “w/o reason” means we do not tell the LLM what exact mistakes it made, but only tell the LLM that it made mistakes. That is to say, as long as one error is found during “problem comparison”, we will tell the model it makes mistakes and needs to rectify its answers. Otherwise, we tell the model it does not make mistakes and the current answer is the final answer. The “w/o judgment + reason” has nothing to do with RCoT; it just tells models to double-check answers every time. Therefore, “w/o judgment + reason” denotes the necessity of giving coarse feedback (i.e., Wrong or Correct), whereas “w/o reason” further denotes that fine-grained feedback formulated from the results of “problem decomposition” can enhance the LLM’s capability of rectifying responses.
We would also like to point out that RCoT performs better than Self-Consistency in 6 datasets and only underperforms Self-Consistency in one dataset in Table 4.