Chain of Thoughtlessness? An Analysis of CoT in Planning

Summary

This paper evaluates the effectiveness of CoT prompting on reasoning problems within the Blocksworld domain, revealing that performance gains from CoT are limited and heavily reliant on problem-specific prompts, with diminishing returns as problem complexity increases.

Strengths

- The study employs a well-defined case study within the Blocksworld domain, providing a concrete context to evaluate CoT prompting. - The study examines the generality of examples and problem complexity offers a structured and comprehensive approach to evaluating CoT's effectiveness.

Weaknesses

1. The paper ignores all experimental details except prompt. The lack of description of experimental details, including the temperature and other parameters used in the data set, affects the reproducibility of the paper. 2. The paper is not very clear. The core point of the paper is that the model CoT is very sensitive to prompts and demonstrations. However, the paper spends a lot of time exaggerating the failure of CoT-style demonstration.

Questions

1. I seriously doubt the test of Table 1. Why does Lexicographic Stacking Prompt only have 17 samples (*/17), while other prompts have 270 samples? This comparison is completely unfair. The model may only perform well on these 17 examples. 2. I think there is an obvious overclaim in this paper. This paper only shows that the current CoT is very sensitive in this kind of Action-related tasks. In addition, the effectiveness of CoT plans is often verified on natural language mathematical tasks. Because the expression of mathematics + natural language is more free, and theplanning logic is more general, demonstration can be used to stimulate better output of CoT style, including various types such as ICL CoT, Zero-shot CoT, Plan-and-Solve, Complex- CoT and a series of methods. 3. In fact, as shown in Figure 1, the absolute improvement of CoT has become smaller. The essential reason is that the model itself cannot solve such a complex problem. It is not that the CoT strategy is not good. In fact, compared with direct output, the relative increase in performance is actually Improved (CoT Acc/Direct Acc)

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This paper evaluates the efficacy of Chain of Thought (CoT) prompting in improving the reasoning capabilities of large language models (LLMs) in planning tasks. The authors analyze CoT's performance in the Blocksworld domain, a classical planning problem, and extend their findings to other synthetic tasks. They demonstrate that CoT prompts only show meaningful improvements when the provided examples are highly specific to the problem class, and that these improvements quickly deteriorate as the complexity of the problems increases. The paper challenges the notion that CoT enables LLMs to learn general algorithmic procedures, suggesting instead that performance gains are largely due to pattern matching rather than genuine algorithmic understanding.

Strengths

1. **Comprehensive Evaluation**: The paper provides a thorough analysis of CoT prompting across various problem domains, including Blocksworld, Coin Flip, Last Letter Concatenation, and multi-step arithmetic. This breadth ensures that the findings are robust and generalizable. 2. **Detailed Analysis**: The authors dissect the performance of LLMs on different levels of problem complexity, offering insights into the limitations of CoT prompting as problem size and complexity increase. 3. **Critical Perspective**: The paper critically examines the assumptions behind CoT prompting, providing evidence that contradicts the widely held belief that CoT enables LLMs to learn and apply general reasoning strategies.

Weaknesses

1. **Weak Motivation**: The Chain-of-Thought paper [1] claims that "chain-of-thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting." However, the findings from this paper, specifically in lines 84 to 88, reiterate the same point, stating that CoT prompts act as exemplars and involve "pattern matching." 2. **Unrelated Title**: The paper primarily evaluates CoT prompts in a single planning task, Blocksworld, and the results indicate that CoT is still useful, as shown in Table 1. However, the title "Chain of Thoughtlessness" does not accurately reflect the content or findings. 3. **Lack of Smooth Transitions**: The content in the introduction and related work sections lacks order, making the paper structure disjointed and difficult for readers to follow. The transitions between paragraphs are abrupt and lack coherence. For example, using subheadings in the related work section could help group related content into subsections, creating a more logical flow. 4. **Section 3 Placement**: The background information in Section 3 could be integrated into a subsection of the related work, providing a smoother introduction to the planning tasks. 5. **Unclear Conclusions**: The conclusions in lines 231 to 235 are not well-supported by the results. Table 1 shows that Zero-shot CoT, Blocksworld Universal Algorithm CoT, Stacking CoT Prompt, and Lexicographic CoT Stacking Prompt all improve performance, with some showing significant improvements. However, the paper claims that the CoT approach does not enhance performance, which contradicts my understanding. Perhaps my interpretation of this table is incorrect. 6. **Lack of Examples in Main Text**: Lines 196 to 220 should include specific examples directly in the main text, as they are central to the experiments. Currently, these examples are relegated to the appendix, which diminishes their impact and increasing the understanding of remaining sections. 7. **Insufficient Experimental Explanation**: The experimental section lacks clarity and organization. Each experiment should be clearly described, explaining what each aims to demonstrate, such as Experiment A for generality and Experiment B for complexity. 8. **Missing Statistics about Test Set**: There are no statistics provided about the Blocksworld test set, leaving the question distribution unclear. This is essential evidence for evaluating the generality of CoT prompts. 9. **Unaddressed Human Labor Cost**: The paper frequently mentions the drawback of CoT prompts requiring human labor but does not provide any comparative analysis or solutions. This point does not directly support the paper's argument and feels extraneous. 10. **Writing Typos**: - Line 72: PDDL is introduced as an acronym without explanation. - Line 239 likely refers to Figure 2, which should be corrected. > Reference: > [1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.

Questions

N/A

Rating

4

Confidence

4

Soundness

1

Presentation

1

Contribution

1

Limitations

N/A

Summary

The paper conducts a systematic study of claims that chain-of-thought (CoT) prompting unlocks reasoning abilities in LLMs. In particular, the paper evaluates the ability of LLMs to a) learn a simple algorithm from demonstrations annotated with reasoning steps ("thoughts") provided as part of the input prompt, and b) to generalize to harder instances in the same problem class. CoT prompt variants are evaluated on a carefully constructed set of simple planning problems (e.g., Blocksworld) and simplified variants. The experimental evaluation demonstrates that CoT works better when the prompt includes reasoning demonstrations on examples "similar" to the query (test) problem and when the test problem class is sufficiently easy (small, specific). The paper makes the case that CoT enables a form of syntactic pattern matching rather than algorithmic reasoning. Overall, the paper provides deep empirical insight into CoT prompting, clearly demonstrating its limited ability to generalize to larger tasks requiring multi-step reasoning when given smaller sized demonstrations, a task easily handled by sound planning algorithms.

Strengths

+ The paper tackles an important topic of large interest to the community. The extent of LLMs abilities aren't fully understood, especially on challenging tasks (reasoning, planning) and it is important to carefully assess claims of new abilities on these tasks. + The paper performs a rigorous empirical evaluation using the classical and easily-understood domain of Blocksworld as well as other domains. By evaluating a variety of general-to-specific prompts across problem distributions, the paper is able to thoroughly assess claims of reasoning abilities. + The result of this careful evaluation is strong evidence of the limited ability of CoT to induce algorithmic reasoning in LLMs using only annotated demonstrations. Rather, some form of syntactic pattern matching seems to be occurring. + The paper is extremely well written and easy to understand. The appendixes are richly detailed.

Weaknesses

- I didn't spot any major weaknesses in this paper. Some minor nitpicks are in the questions.

Questions

- This may be out of scope but I'd be curious to understand how much variance there is in these results wrt to the prompt inputs. Specifically, how robust are the results wrt (semantically) small changes to the system prompt, examples, thoughts, formatting, etc.? - (Line 200) I'm probably missing something but where exactly is the meta prompt explaining plan correctness in A.6.2? - Some references seem to be incorrect or missing. - (Line 239) Should it be "Figure 2" (instead of "Figure 3")? - (Line 253) Should it be "Table 2" (instead of "Table 3")? - Should Figure 3 and Table 3 be referenced somewhere in Sec 6.1?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Summary

The paper aims to show that Chain of Thought style prompting does not result in generalisation of reasoning, instead relying on pattern matching to improve performance. It argues that if CoT results in language models learning algorithms in context, then prompts describing general procedures should result in similar performance gains to prompts describing task specific examples. Experiments are performed on planning problems in the Blocksworld domain using prompts at different levels of generality. The results show smaller performance gains from general prompts compared to large gains for more specific prompts. Through further experiments on planning problems in other domains, the paper shows that CoT does not generalise to more complex problems than those presented as examples. The paper concludes that improvements from CoT are due to pattern matching rather than in-context learning of algorithms.

Strengths

1. Novel and interesting evaluation of how the performance gains from CoT can vary with the level of generality of CoT described in the prompt. 2. Comprehensive evaluation of how CoT fails to generalise to problems of higher complexity across a variety of tasks. 3. The blocksworld experiments as well as other synthetic benchmarks are a useful contribution for evaluating reasoning in a scalable manner.

Weaknesses

While the results presented do indicate that more general demonstrations of CoT do not perform as well as task specific ones, they also do show improvement over standard prompting. For example in Table 1, GPT4 jumps from 7% to 28% using the Blockworld Universal Algorithm. A similar trend is present in Table 2 and Table 3. It is unclear to what extent this is from pattern matching or from algorithmic generalisation. The paper may be at risk of overstating their claim that CoT is not inducing any reasoning. Discussion and further analysis of this would be helpful. A concern about the "Progression Proof CoT" prompt: The prompt includes demonstrations of successful plans with details of the actions and states. However, it does not include any reasoning / algorithmic description of how the plan was obtained from the problem. To be considered a legitimate Chain of "Thought" prompt, should not this prompt include description of a general procedure to derive the plans? Otherwise, it is difficult to see of the LLM can be expected to generalise reasoning from a description of the final answer without intermediate reasoning "thoughts". Overall, a novel and interesting contribution but there are some concerns about evaluation and the conclusions drawn from them.

Questions

1. In 5.2, the paper says "only the most specific and least applicable prompts retain anywhere near this performance improvement". Is this referring to the "Blocksworld Universal Algorithm" or the "Stacking Prompt"? In the case of the former, is this also not a fairly general prompt? More broadly my concern is that Table 2 shows that the Universal Algorithm Prompt achieves similar performance to the Stacking prompt, showing that a more general CoT can perform as well as a more specific one, which is contrary to the central claim of the paper. 2. The performance of GPT4 on Lexicographic Stacking seems surprisingly low with the n-shot prompt (considering the CoT prompt achieves 94%). Was this due to the particular examples provided in the n-shot prompt? 3. Table 3 does not seem to be referenced in the text? 4. Does 5.1 contain an incorrect reference to Figure 3? 5. Does 5.2 contain an incorrect reference to Table 3? 6. Can the n-shot prompts be included in the appendix?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

I could not find any discussion of the limitations of the work. One limitation is that the work does not provide methodologies for reasoning with LLMs beyond that go beyond the limitations described.

Re. author response

I thank the authors for their detailed response to all reviewers. After reading the other reviews and the comments, I'm now more positive about the paper. In my opinion, the authors have been clear about their central claims both in the paper and comments ("we are only claiming that CoT doesn’t robustly learn the algorithmic procedure demonstrated in prompt, as shown by the performance deteriorating as the number of reasoning steps increases"), and proceed to demonstrate its empirical validity rigorously. I think this paper adds valuable empirical insight into the limitations of current LLMs, which is sufficient for me to recommend acceptance. Beyond that, I think this paper would be a valuable addition to the growing body of work in LLM-based planning.