Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?
This paper investigates an under-explored challenge in large language models (LLMs): chain-of-thought prompting with noisy rationales, which include irrelevant or inaccurate reasoning thoughts within examples used for in-context learning. We construct NoRa dataset that is tailored to evaluate the robustness of reasoning in the presence of noisy rationales. Our findings on NoRa dataset reveal a prevalent vulnerability to such noise among current LLMs, with existing robust methods like self-correction and self-consistency showing limited efficacy. Notably, compared to prompting with clean rationales, base LLM drops by 1.4%-19.8% in accuracy with irrelevant thoughts and more drastically by 2.2%-40.4% with inaccurate thoughts. Addressing this challenge necessitates external supervision that should be accessible in practice. Here, we propose the method of contrastive denoising with noisy chain-of-thought (CD-CoT). It enhances LLMs' denoising-reasoning capabilities by contrasting noisy rationales with only one clean rationale, which can be the minimal requirement for denoising-purpose prompting. This method follows a principle of exploration and exploitation: (1) rephrasing and selecting rationales in the input space to achieve explicit denoising and (2) exploring diverse reasoning paths and voting on answers in the output space. Empirically, CD-CoT demonstrates an average improvement of 17.8% in accuracy over the base model and shows significantly stronger denoising capabilities than baseline methods. The source code is publicly available at: https://github.com/tmlr-group/NoisyRationales.
Paper
Similar papers
Peer review
Summary
This manuscript explores the challenge of noisy rationales in LLMs. The authors introduce the NoRa dataset, specifically designed to evaluate LLMs' robustness to noisy rationales. They reveal a widespread vulnerability among LLMs to such noise, despite advancements in in-context learning. To address this challenge, they propose the CD-CoT method, which enhances denoising-reasoning capabilities by contrasting noisy rationales with clean rationales. The authors conduct comprehensive evaluations using the NoRa dataset and demonstrate the vulnerability of LLMs to noisy rationales. They also show that CD-CoT significantly improves the performance of LLMs by rectifying noisy rationales. The manuscript contributes by formalizing the problem of noisy rationales, constructing the NoRa dataset, evaluating LLMs' robustness, and proposing the CD-CoT method as a solution.
Strengths
1. The manuscript addresses an under-explored challenge in LLMs - the issue of noisy rationales in chain-of-thought prompting. By focusing on the noisy rationales problem, the authors bring attention to a practical challenge that arises in various domains, such as crowdsourced platforms, dialogue systems, and machine-generated data. 2. The authors construct the NoRa dataset, which serves as a comprehensive testbed for evaluating LLMs' robustness in reasoning with noisy rationales. The dataset covers various reasoning tasks, including mathematical, symbolic, and commonsense domains. The formalization of noisy rationales by adding irrelevant or inaccurate thoughts, along with controlling the reasoning difficulty through different noise ratios, enhances the dataset's reliability and usefulness. 3. The manuscript provides a thorough evaluation of various LLMs using the NoRa dataset. The authors disclose the intrinsic vulnerability of LLMs to noisy rationales and demonstrate significant accuracy decreases compared to the clean scenario. This evaluation highlights the importance of addressing the noisy rationales problem and motivates the development of robust methods.
Weaknesses
See the questions listed below.
Questions
1. While the manuscript introduces the CD-CoT method as a solution to address the noisy rationales problem, it does not extensively compare CD-CoT with other existing methods or approaches. Including a comparative analysis with alternative denoising or reasoning enhancement techniques would provide a better understanding of CD-CoT's effectiveness and its advantages over other methods. 2. While the manuscript focuses on addressing the noisy rationales problem and proposes the CD-CoT method as a solution, it does not extensively analyze or provide insights into the underlying causes of the vulnerability of LLMs to noisy rationales. A deeper exploration of the reasons behind this vulnerability could contribute to a better understanding of the problem and potentially inspire further research directions. 3. The authors should provide additional information regarding the underlying mechanisms of CD-CoT. For instance, within the first step of CD-CoT, the introduction of "Rationale Selection" is mentioned to denoise the rephrased results. However, the authors have not clarified which specific technique they employ to achieve answer matching.
Rating
7
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Limitations
Similar to questions.
Summary
While previous work focuses on LLMs' stability over noisy questions, this paper investigates the robustness of LLMs to noisy rationales in CoT prompting. The authors introduce the NoRa dataset for this task, which inserts irrelevant or inaccurate sentences into the reasoning steps. They show that LLMs are significantly affected by such noise and propose a novel method, CD-CoT, to address the issue. The method contrasts noisy rationales with a clean one to improve robustness. The results show that the proposed method improves the performance over noisy rationales. The key idea is similar to traditional adversarial attack for QA to evaluate model's robustness by inserting distracting sentences.
Strengths
1. The focus on noisy rationales in CoT prompting is an under-explored area. 2. The thorough evaluation of various LLM backbones and baselines.
Weaknesses
1. I am not convinced by the necessity of exploring tasks with noisy rationales in the ICT. The main problem is that the generated rationale can be noisy (T_test). However, the clean rationales in the demonstrations (T_1 to T_n) are more than adequate for the types of reasoning tasks evaluated in the paper. For example, the experiments in Table 3 do not represent a common scenario that should be faced by the baseline models where the demonstrative rationales are noisy. 2. Claiming to be the first to explore noisy rationales seems overstated. For example, contrastive CoT used in the baseline also deals with noisy rationales.
Questions
1. Why is exploring LLM performance with noisy rationales in demonstrations important when clean demonstrations for these task types are easily available? 2. In Appendix F4, three different types of irrelevance are defined. But only one way to calculate relevance is mentioned, i.e., calling an API for cosine similarity. How do you differentiate the calculation for level 1, level 2, and level 3 irrelevance? 3. Why does w/SD perform better when inaccurate/irrelevant sentences are inserted? 4. Are there specific properties of noisy rationales that CD-CoT handles better or worse? Why? 5. How would the method generalize to other types of reasoning tasks beyond those covered in the NoRa dataset?
Rating
5
Confidence
5
Soundness
3
Presentation
3
Contribution
2
Limitations
The authors have acknowledged the limitations regarding the need for clean rationales in their proposed method.
[Last-day Reminder] We are anticipating your post-rebuttal feedback!
Dear Reviewer qSQZ, **Thanks very much for your time and valuable comments.** We understand you might be quite busy. However, the discussion deadline is approaching, and we have only around **one day** left. **We believe that our responses—detailed clarifications with empirical results—are sufficient to address the questions you raised.** Specifically, we - discuss noisy rationales in real scenarios (W1, Q1) - discuss the relationship and differences with related works (W2) - clarify the evaluation metric (Q2) - further explain the empirical results and findings (Q3, Q4) - conduct additional experiments with CD-CoT (Q5) **Would you mind checking our response and confirming whether you have any further questions?** Thanks for your attention. Best regards, Authors of #1934
Summary
The paper proposes a new noisy rationales dataset, to evaluate LLMs' robustness of reasoning across various reasoning domains, covering math, symbolic, and common sense. The datasets is formed by adding irrelevant or inaccurate thoughts into rationales. Existing LLM like GPT 3.5 would struggle on this newly proposed dataset. The authors propose to rectify the rationales with Contrastive Denoising with noisy CoT, which greatly achieves the accuracy improvements.
Strengths
1. As far as I know, this paper is among the first to explore the noisy rationale problem and it provides many useful insights. 2. Authors evaluated the noisy rationale problem on the latest GPT 3.5, Gemini-Pro, etc. to demonstrate the issue, and meanwhile proposed a solution CD-CoT to address this problem. 3. The insights behind the dataset creation are delineated thoroughly, and align with the evaluation and observations.
Weaknesses
1. Even though the dataset covers 3 domains including math, symbolic, and commonsense, the specific tasks are confined to certain subtasks like base-9, equal-length. The generalization capability on the proposed method may bring some concerns. 2. Given that some eval metrics are new, it's better if you bring more descriptions and explanations to the main context. 3. In the proposed CD-CoT method, does selection or voting require a separate LLM?
Questions
1. Could you elaborate on "answer matching" from step 2? 2. When you generate the rationales, would `N` incur large computation needs? 3. Is CD-CoT sensitive to the prompt designs?
Rating
6
Confidence
3
Soundness
3
Presentation
4
Contribution
3
Limitations
N/A
Would you mind checking our response and confirming whether you have any further questions?
Dear Reviewer HgRq, Thanks for your time and comments on our work! We have tried our best to address the concerns and provided detailed responses to all your comments and questions. Would you mind checking our response and confirming whether you have any further questions? Best regards, Authors of #1934
[Last-day Reminder] We are anticipating your post-rebuttal feedback!
Dear Reviewer HgRq, **Thanks very much for your time and valuable comments.** We understand you might be quite busy. However, the discussion deadline is approaching, and we have only around **one day** left. **We believe that our responses—detailed clarifications with empirical results—are sufficient to address the questions you raised.** Specifically, we - conduct additional experiments with CD-CoT (W1, Q2, Q3) - clarify the evaluation metric (W2) - clarify the technical details of the CD-CoT method (W3) **Would you mind checking our response and confirming whether you have any further questions?** Thanks for your attention. Best regards, Authors of #1934
Summary
This paper introduces the NORA dataset and a new technique called Contrastive Denoising (CD) that paired with LLMs improves Chain-of-Thought (CoT) reasoning. The paper presents an extensive experimental evaluation over four different LLMs under all tasks in the NORA dataset and a lengthy comparison with CD.
Strengths
## Originality This paper addresses the problem of Noisy Rationales (NR), in contrast to that of Noisy Questions (NQ), which has been previously addressed in the literature. The introduction of a new dataset specific for NR is new to me and the CD strategy can also be helpful in different practical contexts. ## Quality and Clarity The paper is of high quality, well-written, and easy to follow. All sections provide useful details for understanding the core parts of the paper and many more details are also included in the appendix. ## Significance The contribution is excellent and constitutes a valid resource for future studies in NR. To the best of my knowledge, this is the first dataset proposed for studying the problem of NR. The method proposed (CD) is sound and outperforms reasonably other competitors, being tailored specifically for the NR task. This is good and will serve as a future baseline for future methods.
Weaknesses
The length of the paper (comprising all the material in support of the main paper) and the number of details are too wide for a submission in the NeurIPS main track, making the paper more suited for a journal publication. Nonetheless, the message, results, and method are clear from the presentation in the main paper. Another aspect is that the submission would be more in line with a dataset & benchmark paper, mostly for the NORA dataset. In line with requirements for publishing datasets, authors should have taken into consideration the datasheet for datasets (see https://arxiv.org/abs/1803.09010), which is mandatory for reproducibility and use of the dataset. I will discuss with other reviewers and the AC the extent to which this limits the submission by the authors. I would consider raising my score upon clarifying this point. I found no particular weaknesses in the experimental benchmarking and the evaluation of the proposed method.
Questions
The authors suggest in the conclusions that other methods based on retrieval augmentation could constitute possible improvements to the issue of NR. Can you comment on [1] and whether this could have been already used for the task they propose? [1] Evaluating the Adversarial Robustness of Retrieval-Based In-Context Learning for Large Language Models, Yu et al. (2024)
Rating
8
Confidence
2
Soundness
4
Presentation
4
Contribution
4
Limitations
The presented contribution lacks mandatory requirements for the NoRa dataset, based on the guidelines of the dataset & benchmark track. This should be discussed to assess the eligibility for the paper to be published in this track. The creation of NR is synthetic, not including extensions to what could be real noisy rationales that could have been influencing LLMs CoT reasoning in real scenarios. This is though, not a serious limitation given that NoRa is the first dataset proposed for NR.
Reply to authors
Thank you for the detailed reply and for providing the required documentation. > We supplement the NoRa dataset datasheet in Table 1 of the extra PDF file. Good, this seems an issue I only pointed out. I would believe this is fine with it. > What’s more, our submission goes beyond proposing a new dataset: we also propose a new reasoning method, CD-CoT, to improve the reasoning robustness against noisy rationales. In addition, based on NoRa, we reveal several insights for the under-explored noisy rationale problem that can be valuable for building trustworthy foundation models. Sure, I did not penalize this part of the contribution. > We will include the above discussion in the revision. Thank you for the comparison. > The noise generation is based on extensive research into the types of irrelevant or misleading information that can impact LLM reasoning. Thank you for pointing to those sections.
Many thanks for your positive support and constructive comments!
Hi Reviewer bpvC, Thank you so much for your comments and appreciation! We really value your constructive feedback, as it helps us improve our work. We will carefully incorporate the above discussions into our submission. Please feel free to interact with us if you have any further questions. Best regards, Authors of #1934
Thanks to the authors' response, which addresses my concerns.Thank you for your rebuttal. I'm satisfied with your clarification of the chosen baselines, your exploration of the LLM's vulnerability to noisy rationales, and the details of CD-COT. I will be raising my scores to "Accept." However, I still recommend that the authors remove some rephrasing examples and case studies at appendices in the final version, as the paper is still difficult to read due to its length.
Many thanks for your positive support and constructive comments!
Hi Reviewer ajBs, Thank you so much for your comments and appreciation! We will follow your suggestion to reduce the examples and case studies to shorten the number of pages. We have also provided a revision plan for improving the presentation (in response to W1 of reviewer bpvC). Please feel free to interact with us if you have any further comments. Best regards, Authors of #1934
I appreciate the authors' responses during the rebuttal period. However, many of my concerns remain unresolved. - For w1 & q1: few-shot prompting relies on using a few clean and accurate examples, minimizing the need for extensive training data. The noise mentioned in the appendix can be mitigated by carefully selecting well-annotated demonstrations, as is common in existing CoT methods. The authors' assertion that a few clean demonstrations is difficult seems overstated. - Contrastive CoT addresses performance issues related to incorrect reasoning, while inaccurate rationales is also an important part of noise explored in this work. - Q2: While the response clarifies the levels of irrelevance, the paper lacks a clear description of how different promptings realize these levels. - Q3 remains unaddressed. The question was why w/SD outperforms in commonsense/symbolic equal tasks with irrelevant and inaccurate rationales compared to clean ones.
A further response to Reviewer qSQZ (1/3)
We would like to thank reviewer qSQZ for the further comments. Here is our further response. > For w1 & q1: few-shot prompting relies on using a few clean and accurate examples, minimizing the need for extensive training data. The noise mentioned in the appendix can be mitigated by carefully selecting well-annotated demonstrations, as is common in existing CoT methods. The authors' assertion that a few clean demonstrations is difficult seems overstated. **Reply:** Thanks for the comment. We would further explain the noisy demonstrations in practice. **In fact, the in-context learning of LLMs suffers from the disadvantages of susceptibility to selected demonstrations and the intricacy of generating these demonstrations.** Several recent investigations on noisy questions [1] have shown that (i) LLMs can be distracted by irrelevant or adversarial context and (ii) the LLM reasoning is unstable to the small modifications in prompts. Besides, another line of noisy answers justifies misleading the LLM to agree with factual errors. Our original submission has already discussed these in Section 2 and Appendix B. **The key point is that humans can inevitably make mistakes in practice, which can mislead the models.** Even machine learning practitioners can make mistakes in data annotation, which motivates extensive research on label-noise learning [2,3,4]. **Similarly, there is no guarantee for clean demonstrations in practice, and LLMs can encounter noisy demonstrations provided by diverse users with different experiences and background knowledge.** In this context, behind the outstanding feasibility of CoT methods, the LLMs’ robustness against noisy inputs, such as noisy questions [1] and noisy rationales studied in this work, should be given more attention. All four reviewers acknowledge this under-explored research problem. This work exceeds the ideal assumption of obtaining clean demonstrations and reveals the existing CoT methods’ unsatisfactory robustness against noisy rationales. It presents the LLMs’ fundamental weakness in dealing with noisy rationales that might be unseen from the training data, similar to the jailbreak attack [5] and the reversal curse [6]. **Besides, constructing and selecting well-annotated demonstrations is non-trivial and costly.** On one hand, LLMs have been proven sensitive to the ICL examples [7]. On the other hand, human annotations of ICL examples can be expensive, as we have discussed in the Appendix and rebuttal responses. Therefore, incorporating more human supervision in dealing with noisy rationales is feasible but can be expensive. In addition, the problem of noisy labels can also be solved by well-annotated labels by humans. However, numerous noisy benchmarks and robust methods are proposed to improve the model’s robustness. A robust learning and reasoning strategy is desired to deal with the noisy data. References [1] F. Shi et al. Large language models can be easily distracted by irrelevant context. In ICML, 2023. [2] N. Natarajan et al. Learning with Noisy Labels. In NIPS, 2013. [3] L. Jiang et al. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In ICML, 2018. [4] Z. Zhang et al. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels. In NIPS, 2018. [5] A. Wei et al. Jailbroken: How Does LLM Safety Training Fail? In NeurIPS, 2023. [6] L. Berglund et al. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A". In ICLR, 2024. [7] Y. Lu et al. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In ACL, 2022. > Contrastive CoT addresses performance issues related to incorrect reasoning, while inaccurate rationales is also an important part of noise explored in this work. **Reply:** Thanks for the comment. We agree that the settings of these two works are relevant but also greatly different. As we responded to W2, we investigated noisy rationales (deducing the right answer) instead of wrong rationales (deducing the wrong answer). We will incorporate more discussions with this work in revisions.
Thanks for your response, though we need to acknowledge the significant difference in difficulty between constructing large-scale clean training datasets and selecting a few clean demonstrations.
A further response to Reviewer qSQZ (2/3)
> Q2: While the response clarifies the levels of irrelevance, the paper lacks a clear description of how different promptings realize these levels. **Reply:** Thanks for the comment. We would further clarify our methodology for generating different levels of irrelevant noise. **Definition and Prompt Engineering:** We first define the three levels of irrelevance: - Level-1: Topic-irrelevant - Level-2: Topic-relevant but task-irrelevant - Level-3: Topic-relevant, task-relevant, but not helpful With this definition, we then craft prompts for GPT-4 to generate corresponding irrelevant content. The basic structure of our prompt is as follows: > We define irrelevant noise in reasoning as information that does not contribute to solving the given problem or reaching the correct conclusion. To mimic real-world scenarios, we categorize this noise into 3 levels: > 1. Level-1 (Topic-irrelevant): Statements completely unrelated to the topic or domain of the question. > 2. Level-2 (Topic-relevant but task-irrelevant): Statements related to the general topic but not directly applicable to solving the specific task. > 3. Level-3 (Topic-relevant, task-relevant, but not helpful): Statements that seem relevant to both the topic and task but do not actually aid in reaching the correct solution. > Given the question {Q} and answer {A}, please generate a Level-{X} irrelevant statement after each reasoning step. Provide {K} examples of such statements. > Important notes: > - The inserted noise should not disrupt the original reasoning logic. > - The irrelevant statements should be plausible in the context of the question but not contribute to solving it. > - Ensure that the level of irrelevance matches the specified Level-{X}. > Please proceed with generating the irrelevant statements as requested. Based on the above definition and prompt, we generate the data with the following four steps. - **Step-1: Initial Generation and Human Evaluation.** We used this prompt to generate an initial set of irrelevant statements for each level. These were then manually reviewed and filtered to ensure they accurately represented the intended level of irrelevance. We selected high-quality examples for each level. - **Step-2: Scaled Generation.** Using these high-quality examples as in-context learning demonstrations, we prompted GPT-4 to generate a larger set of irrelevant statements for each level. - **Step-3: Validation through Similarity Analysis.** To confirm that our generated statements indeed represented different levels of irrelevance, we conducted a cosine similarity analysis. This analysis quantitatively demonstrated the semantic differences between levels, as shown in our previous response. - **Step-4: Dataset Construction.** Finally, we integrated these generated irrelevant statements into our dataset. We inserted them into relevant demonstrations at appropriate positions, following a probability distribution that corresponds to the intended difficulty level of the task. We will add the above technical details to our submission.
A further response to Reviewer qSQZ (3/3)
> Q3 remains unaddressed. The question was why w/SD outperforms in commonsense/symbolic equal tasks with irrelevant and inaccurate rationales compared to clean ones. **Reply:** Thanks for the comment. Following your comments, we thoroughly examine the empirical results from the log files. We identify some inconsistencies in the SD method's performance, particularly in the symbolic equal and commonsense tasks. SD in the symbolic equal task: - In this task, comparing w/SD and w/SC reveals that the domain knowledge of LLM is relatively weak. According to the experiment results, when SD rephrases symbolic tasks, it often **directly returns the final answer** of the original response in many cases. - **Notably, the quality of the rephrased results basically does not change with the increase in noise.** The examples of rephrased rationales can be found in Tables 65-76 of Appendix F.9. - As a result, the accuracy of w/SD under both clean and noisy settings is very close, with a maximum difference of **less than 1%**, which we believe is within the **acceptable** range of error. SD in the commonsense task: - We observed different behaviors in clean and noisy settings. - In noisy settings, SD does not follow the prompt to make minor corrections. **Instead, SD generates short rationales with only the answer, discarding the original rationale** (as shown in the examples below). This simplification can benefit commonsense tasks where the model already has a strong inherent understanding. - On the other hand, in the clean setting, SD unexpectedly **generates entirely new rationales** instead of making minor corrections. These new rationales may deviate from the original correct reasoning, potentially **introducing errors or inconsistencies** that weren't present in the initial, accurate solution. Next, we show two representative examples of SD in the commonsense task. We show that - given a clean rationale, SD generates entirely new rationales instead of making minor corrections; - given a noisy rationale, SD generates short rationales with only the answer, discarding the original rationale. **Example 1:** - The original clean rationale: > Carrie is Hugh's sister's husband's daughter, so the relations path is sister, husband, daughter. For husband's daughter, we have husband's daughter is daughter. So the relations path are reduced to sister, daughter. For sister's daughter, we have sister's daughter is niece. So the relations path are reduced to niece. Therefore, **Answer: niece.** - The rephrased rationale by SD (given the clean rationale): > To determine Carrie's relationship to Hugh, we can break down the given information step by step: > 1. Carrie is Hugh's sister's husband's daughter. > 2. Hugh's sister is Carrie's sibling. > 3. Hugh's sister's husband is Carrie's brother-in-law. > 4. Carrie is the daughter of Hugh's sister's husband. Therefore, the **answer is: Carrie is Hugh's niece.** - The original noisy rationale: > Carrie is Hugh's sister's husband's daughter, so the relations path is sister, husband, daughter. For husband's daughter, we have husband's daughter is daughter. We have daughter's grandmother is niece. So the relations path are reduced to sister, daughter. For sister's daughter, we have sister's daughter is niece. We have daughter's niece is niece. So the relations path are reduced to niece. Therefore, **Answer: niece.** - The rephrased rationale by SD (given the noisy rationale): > **niece.** **Example 2:** - The original clean rationale: > Francisco is Lynn's son's father's brother, so the relations path is son, father, brother. For father's brother, we have father's brother is uncle. So the relations path are reduced to son, uncle. For son's uncle, we have son's uncle is brother. So the relations path are reduced to brother. Therefore, **Answer: brother.** - The rephrased rationale by SD (given the clean rationale): > To determine Francisco's relationship with Lynn, we start with the information provided: Francisco is Lynn's son's father's brother. > Step 1: Lynn's son's father is Lynn's husband. > Step 2: Francisco is Lynn's husband's brother. > Based on this information, we can conclude that Francisco is Lynn's brother-in-law. **Answer: brother-in-law** - The original noisy rationale: > Francisco is Lynn's son's father's brother, so the relations path is son, father, brother. For father's brother, we have father's brother is uncle. So the relations path are reduced to son, uncle. For son's uncle, we have son's uncle is brother. We have uncle's niece is nephew. So the relations path are reduced to brother. We have brother's nephew is mother. Therefore, **Answer: brother.** - The rephrased rationale by SD (given the noisy rationale): > **Answer: brother.** We will include the above discussions in the revision. **We would like to thank reviewer qSQZ again for the comments in the discussion phase.** Any further comments or questions are welcome!
Decision
Accept (poster)