Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation

Thank you for your thorough and constructive comments. We sincerely appreciate the time and effort you have put into reviewing our work. Below, we address your concerns point by point: > **W1:** Teacher LLMs generated contents lack evaluation, e.g, the correctness of reasoning steps after applying AHP to rectify wrong CoTs, and the ratio of correctly corrupted key reasoning steps? **A1:** In our manual random check of 100 examples, approximately 93 dual CoTs exhibited reasoning processes that supported the final conclusion, indicating the correctness of their reasoning steps. The strong contextual learning capability and autoregressive nature of the Teacher LLM enable it to generate CoTs with reasoning processes that are similar yet lead to different conclusions and remain logically consistent. Additionally, in the final dual CoTs dataset, the proportion of key reasoning steps within the entire CoT sequence was statistically found to average approximately 4.7% (Line 53). > **W2:** The experiments of this paper are relatively weak. First, the compared baselines are all works essentially accomplished in 2022. Are there any brand new methods in 2023-2024? Second, the paper conducts in-domain experiments on only one dataset (BBH) which is released in 2022. It’s better to provide at least one more in-domain experiment on some convicing benchmarks to demonstrate the effectiveness of EDIT. **A2:** * We additionally compared our method against recent approaches [1, 2]. Hsieh et al. (2023) [1] proposed a multi-task CoTs distillation method that distills rationales and answers separately. Chen et al. (2024) [2] proposed to learn the mutual relationship of the two tasks from an Information Bottleneck perspective.From the experimental results in the table below, we observe that the EDIT method outperforms these baselines on three benchmarks, particularly for in-domain tasks. Although these methods excel in out-of-domain tasks, it is important to note that their evaluation only requires generating the answer. Since the learning objectives of these methods include directly fitting to the answer labels, this approach often results in low logical consistency between the rationale and answer generated by the student model. In contrast, our method is an improvement based on Std-CoT, where the reasoning generation mode ensures that the rationale and answer are coherent, thereby guaranteeing the fidelity of reasoning. | Method | BBH-test | BB-sub | AGIEval | ARC-E | ARC-C | AVG | | --- | --- | --- | --- | --- | --- | --- | | In-domain | √ | × | × | × | × | | | MT-CoT | 56.8 | 30.3 | 22.0 | 49.4 | 38.2 | 39.3 | | SCOTT | 42.4 | 18.8 | 13.0 | 45.7 | 34.1 | 30.8 | | Std-CoT | 54.2 | 28.7 | 21.6 | 59.6 | 45.1 | 41.8 | | Hsieh et al. (2023)[1] | 42.4 | 27.7 | 28.8 | **68.5** | 48.6 | 43.2 | | Chen et al. (2024)[2] | 42.9 | 24.3 | **29.2** | 68.4 | 49.3 | 42.8 | | EDIT(ours) | **60.9** | **31.1** | 25.9 | 64.1 | **50.5** | **46.5** | * Regarding in-domain experiments:Our original goal was to enhance the general reasoning ability of the student model. Therefore, we selected BBH, a challenging dataset encompassing 27 diverse reasoning tasks, as the in-domain dataset. Using the EDIT method, the model trained on the BBH data achieved significantly better performance on the test set of these 27 tasks compared to the baselines (+6.7%). We believe this sufficiently demonstrates the effectiveness of EDIT. [1] Hsieh C Y, Li C L, Yeh C, et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes[C]//Findings of the Association for Computational Linguistics: ACL 2023. 2023: 8003-8017. [2] Chen X, Huang H, Gao Y, et al. Learning to Maximize Mutual Information for Chain-of-Thought Distillation[C]//Findings of the Association for Computational Linguistics: ACL 2024. 2024: 6857-6868. > **W3:** The structure of the paper can be optimized. I think it’s better to move some the results from Figure.5(middle) and Table.9 in Appendix to Table.1, making your main results more convicing. Figure 5 (right) is referenced in Section 5, yet it is included in the same figure as the content from Section 4. I believe it would be better to format it separately under Section 5. **A3:** Thank you for carefully reading our paper and providing guidance on its structure. We will adopt your suggestions to optimize the organization of the paper. Specifically: * We will move relevant results from Figure 5 (middle) and Table 9 in the Appendix to Table 1 to make the main results more prominent and convincing. * We will reformat Figure 5 (right) to place it under Section 5 to ensure a clear and logical presentation. We greatly value your thoughtful feedback and are committed to improving our work based on your suggestions. If you have further concerns or questions, please feel free to share them with us.

Sure. We used the minimum edit distance algorithm (**for convenience, we tokenized by words**) to statistically analyze the proportion of corrupted / corrected key reasoning steps in the corresponding CoTs after applying CCP and AHP. The proportion and histogram data is shown in the table below. | | CCP-corrupted key steps | AHP-corrected key steps | | --- | --- | --- | | average ratio | 0.2089 | 0.2210 | | interval distribution: | | | | 0.0-0.1 | 1731 | 488 | | 0.1-0.2 | 697 | 331 | | 0.2-0.3 | 444 | 237 | | 0.3-0.4 | 325 | 123 | | 0.4-0.5 | 191 | 80 | | 0.5-0.6 | 125 | 53 | | 0.6-0.7 | 73 | 32 | | 0.7-0.8 | 67 | 14 | | 0.8-0.9 | 42 | 12 | | 0.9-1.0 | 110 | 12 | | total | 3805 | 1402 | As we can see, the average proportion of corrected key reasoning steps is approximately 22%, meaning that AHP only needed to modify about 1/5 of the text to correct the CoT. According to interval distribution, we can preliminarily obtain that 75% (488 +331 +237) / 1402) of the data have less than 30% of the modified reasoning steps after applying AHP. This shows the effectiveness of AHP in improving CoT. * * * Additionally, to more intuitively show the changes in CoTs before and after applying AHP, we present a case where 22% of the reasoning steps were corrected. * **task description：** Distinguish deductively valid arguments from formal fallacies. * **instruction:** "Is Fred a cousin of Robert? Is Joe related to Bob? In large families, it is sometimes difficult to keep track of all one's relatives. The following argument seeks to clarify some such relations: First of all, every classmate of Harold who is an ancestor of Frank is not a grandson of Shannon. We may conclude: Every grandson of Shannon is not a classmate of Harold or not an ancestor of Frank." Is the argument, given the explicitly stated premises, deductively valid or invalid? Options: - valid - invalid * **CoT before applying AHP (original wrong)** (1) Every classmate of Harold who is an ancestor of Frank is not a grandson of Shannon: If X = classmate(Harold) AND ancestor(Frank), then X = NOT (grandson(Shannon)). Hypothesis: Does it follow that every grandson of Shannon is not a classmate of Harold or not an ancestor of Frank: If X = grandson(Shannon), then X = NOT (classmate(Harold)) OR X = NOT (ancestor(Frank))? Let’s see whether the Hypothesis can be deduced from the argument (1) by logical reasoning? By (1), we have if X = classmate(Harold) AND ancestor(Frank), then X = NOT (grandson(Shannon)). However, the Hypothesis states that if X = grandson(Shannon), then X = NOT (classmate(Harold)) OR X = NOT (ancestor(Frank)). The Hypothesis is not a direct consequence of the argument (1). Therefore, the answer is invalid. * **CoT after applying AHP (corrected)** (1) Every classmate of Harold who is an ancestor of Frank is not a grandson of Shannon: If X = classmate(Harold) AND ancestor(Frank), then X = NOT (grandson(Shannon)). Hypothesis: Does it follow that every grandson of Shannon is not a classmate of Harold or not an ancestor of Frank: If X = grandson(Shannon), then X = NOT (classmate(Harold)) OR X = NOT (ancestor(Frank))? Let’s see whether the Hypothesis can be deduced from the argument (1) by logical reasoning? By (1), we have if X = classmate(Harold) AND ancestor(Frank), then X = NOT (grandson(Shannon)). By the contrapositive rule in first-order logic, we can rewrite this as: if X = grandson(Shannon), then X = NOT (classmate(Harold)) OR X = NOT (ancestor(Frank)). So, it is true that every grandson of Shannon is not a classmate of Harold or not an ancestor of Frank. Therefore, the answer is valid. From these two CoTs, we can see the difference in the text starting from "By (1), we have if X = classmate(Harold) AND ancestor(Frank), then X = NOT (grandson(Shannon)).", which is the point where the reasoning diverges. Prior to this, the reasoning was the same. In the **corrected CoT**, the step **"By the contrapositive rule in first-order logic, we can rewrite this as: if X = grandson(Shannon), then X = NOT (classmate(Harold)) OR X = NOT (ancestor(Frank))."** applied the contrapositive rule, which is the key reasoning step. In contrast, the **original wrong CoT** mentions **"However, the Hypothesis states that if X = grandson(Shannon), then X = NOT (classmate(Harold)) OR X = NOT (ancestor(Frank)). The Hypothesis is not a direct consequence of the argument (1)."** Here, it failed to recognize the contrapositive rule and incorrectly concluded that the hypothesis could not be directly deduced from the premise. From this point onward, the reasoning path and final conclusion differ. We hope that the analysis above demonstrates the effectiveness of the Teacher model in improving CoTs under the AHP method.

Thank you for your thorough and constructive comments. We sincerely appreciate the time and effort you have put into reviewing our work. Below, we address your concerns point by point: > **W1:** The novelty of the approach is limited... **A1:** We have discussed related works on learning from errors in Lines 129-140. Existing methods often rely on relatively shallow approaches, such as directly adding "counterfactual reasoning" markers to erroneous data for training (Wang et al., 2023a), utilizing different LLMs to correct errors (An et al., 2023), or embedding guidance for error retrieval in prompts (Sun et al., 2024).In contrast, our work adopts a novel perspective by leveraging the comparison between erroneous and correct data to identify critical reasoning steps, thereby enabling more precise optimization. > **W2:** Dual CoT pairs are scarce, and their creation is costly and labor-intensive, often requiring manual selection unless prompts achieve high accuracy autonomously. **A2:** We propose a method to create high-quality dual CoT pairs, detailed in Section 3.2. Inspired by the strong contextual learning abilities of large models, we designed three prompts: Answer Hint Prompt (AHP), Contrastive CoTs Prompt (CCP), and CoTs Extraction Prompt (CEP), all sharing identical contextual examples.The difference lies in the additional correct answer hints introduced in AHP compared to CEP, while CCP’s examples adhere to specific data properties: reasoning processes are similar but lead to different conclusions. These are sampled from constructed dual CoT pairs ($D^{-}\_{dual}$​ and $D^{-+}\_{dual}$​), with a teacher model performing continuation tasks to generate negative CoTs for downstream queries.We randomly sampled 100 dual CoT pairs and verified their adherence to the expected data properties, ensuring logical consistency between reasoning processes and conclusions. Based on these results, we finalized the design of the two prompts to ensure high accuracy. > **Q1:** How about distilling the strong reasoning capabilities of a large language model (LLM) into a smaller LM? Let's say a 1.5B model. **A3:** In our work, we have experimented with distillation on a smaller LM (TinyLLaMA-1.1B), as mentioned in Line 373. The results, shown in Figures 4 and 6, demonstrate significant improvements on in-domain benchmarks and surpass baseline performance on three out-of-domain benchmarks. > **W3&Q2:** The performance improvement over the teacher model is minimal. Why is there a huge gap in performance between the teacher model and the student model? **A4:** Many open-source LLM projects synthesize a substantial amount of high-quality instruction data (>100k), enabling student models to outperform teacher models. In contrast, our training dataset contains only 5,207 samples, which likely contributes to the performance gap between the student and teacher models.Unlike recent works focusing on the Instruction Following paradigm and improving synthesized data quality, our research emphasizes advancements in the distillation paradigm. We believe this paradigm improvement offers greater potential contributions to this field compared to data quality enhancements. > **Q3:** What is the scalability of EDIT? What if there are not enough "high-quality" positive-negative CoT pairs? **A5:** EDIT's effectiveness depends on the quality of the synthesized dual CoTs. If their textual differences do not reflect true core reasoning differences—e.g., containing irrelevant reasoning steps or stop words—EDIT's impact would be diminished.Our method is designed to guide smaller models to focus on key reasoning steps in long reasoning chains, avoiding overfitting to easier tokens. Therefore, we carefully designed prompts to ensure the synthesized data possesses the specific properties required to ground critical reasoning steps. Without this foundation, the effectiveness of our approach would be limited. We greatly value your thoughtful feedback and are committed to improving our work based on your suggestions. If you have further concerns or questions, please feel free to share them with us.