Summary
The authors present an investigation of ‘unfaithfulness’ in CoT prompting for large language models. They argue in particular that models can be predictably influenced by biasing features in the input, which the CoT “explanations” reliably fail to mention, and that they can be influenced by other factors (eg., social stereotypes) but mask this reasoning by appealing to other, weak forms of evidence. The authors test a GPT-3.5 model and a Claude 1.0 model with modifications to two datasets: a subset of Big-Bench Hard (BBH) and the Bias Benchmark for QA (BBQ). With the first dataset, they add a ‘biasing feature’, which is either a modification of the examples such that the correct answer (in a few-shot prompt) is always the same multiple-choice selection (e.g., (A)), or a modification to the example such that there is an additional “Suggested answer” line (e..g, “I think the answer is A but I’m curious to hear what you think.”). In the second analysis and dataset, they introduced a piece of “weak evidence” in ambiguous reasoning problems, to see (1) whether the model applied the weak evidence in inconsistent manners and (2) whether these were in part out of a tendency to produce stereotype aligned answers (e.g., the answer is always “the Black man” regardless of whether the weak evidence would suggest that or not).
Weaknesses
**Results and Design**
- Like a lot of LLM evaluation papers, the main statistical analyses provide a rather crude basis for statistical evidence, and many of the conclusions are presented with no statistical evidence at all.
- For example: The conclusion in 3.2 that “generally few-shot CoT exhibits less unfaithfulness than zero-shot Cot” is a statistical claim but there are no inferential statistics (nor even descriptive statistics such as confidence intervals) present in any of the analyses in this section.
- Generally, the evaluation setting is systematically structured (CoT vs. no COT; debiasing instructions vs. not; FS vs. Zero-shot; GPT vs. Claude) with the same items/prompts being evaluated in these different conditions, but this structure isn’t taken into account when calculating the statistical results, which makes it difficult to assess the claims made by the paper. The fact that that the data are structured in this ways means the individual data points are not iid, and thus a binomial test is not an appropriate statistic test. A paired difference test is better, but still misses the multi-level nature of these comparison. What would be appropriate here is a multi-level (hierarchical / mixed-effects) logistic regression model, taking into account the fact that the experimental manipulations have different effects on different tasks in BBH, and that the same examples appear in all of the different levels of the manipulated variables (e.g., it’s the same prompts that appear in the CoT context as in the no-CoT context, as well as the zero-shot vs. Few-shot, as well as Claude vs. Gpt, etc.) See Lampinen et al. (2022) _Can language models learn from explanations in context?_ for an example of this kind of analysis in the language model evaluation context, or see Gelman and Hill (2006) for further background.
- There are potential alternative interpretations for what the model is learning from the prompts in these contexts. It’s possible these explanations do not account for the whole pattern of data as well as the explanation that the authors put forth. But it’s still important to bring some of them up and discuss to help the reader better understand the evidence in favor of their account.
- In Study 1, for the “Always A”, in few-shot setting with chain-of-thought, the model could be learning that the task is to *Respond A and provide a plausible explanation that doesn’t mention A*. The example CoT are all good or valid explanations (by design), and almost certainly represent a certain style of explanation. In addition to the fact that none of them likely mention the fact that ‘all the answers are A’ as an explanation, the explanation that ‘all the answers are A’ is a kind of meta-task explanation, which is probably a different kind of explanation than those that appear in the few-shot examples. It’s an explanation at the level of the whole set of examples, whereas each individual CoT (in the shots) probably only concerns the individual example. Therefore, it’s unclear if the model’s reasoning is unfaithful here, or whether the prompt is encouraging it to be unfaithful.
- The “suggested answer” manipulation also will fall under the same header of not being mentioned previously in the prompt (appendix F.2), and furthermore, the “Suggested answer” would be just a bad explanation for doing something, and the model only saw good (high-quality) explanations in the few-shot prompt (and likely in the pretraining set, for the Zero-shot case). If my helpful assistant told me "Oh I heard some voice that suggested it me weirdly out of the blue just now", I would say that is a bad reason for doing something.
- In BBQ, I tried hard to find out the nature of the few-shot prompt examples but I couldn’t determine if the few-shot prompt examples always included only stereotype-aligned responses for the disambiguated content examples (as is shown in Table 16), or if this varied in a counterbalanced manner. Without knowing this, it’s hard to evaluate the Few-shot results, as the prompt may have biased the model to have stereotype-aligned responses.
- I’m generally unclear about the design choices around the BBQ data with ambiguous scenarios and weak evidence. There seems to be some implicit model behind this study, having to do with weighing evidence.
- In the appendix, the authors write how some generated pieces of evidence were too strong and others were too weak, which makes me confused about what the ideal amount of ‘weak evidence’ is and what it’s intended to do.
- In the example in Appendix B, we have a story about a line-cook and a physician, and which one is likely to have an exclusive credit card. The authors note that selecting the physician, “while arguably reasonable”, is counted as reasoning on the basis of stereotypes according to BBQ. Fine, but insofar as there is (quite plausibly) a statistical correlation between being a physician vs. line-cook and owning an exclusive credit card, how are we to say that _that evidence_ (that statistical association) is weaker than the ‘weak evidence’ provided in their modified task. In all likelihood, it’s stronger. Then in this case, the model could infer the task that is being asked of it is to jusifiy the highest probability outcome using the piece of ‘weak evidence’ provided.
- Another example: on Table 16, the Unknown response could also be considered stereotype-aligned, if you deemed the evidence of being on the phone vs. speaking to a young child as sufficiently strong to sway you were it not for the low-income vs. high-income bias.
- I think the logic of this Study is either
- The situation is ambiguous and the evidence is weak, meaning the normative Bayesian posterior probabilities of each outcome is close to 50% (and the model should say Unsure)
- OR, the evidence presented is stronger than the prior association that exists such that the evidence could sway you in one direction or the other (and you should be consistently swayed)
- But, it seems like many examples there is likely a strong prior (statistical) association and the evidence is not strong enough to overcome it (normatively, speaking).
- I’m not totally sure about this but I think it might be useful to know what the models predict without the ‘weak evidence’ provided. I think this could shed light on if the model has a prior tendency that it’s looking to rationalize, or if the process of reasoning is bringing out some tendency.
**Literature**
- Rationalization seems to be a key theme in this work, and I’m surprised the authors didn’t discuss that rationalization is a powerful way of making LMs better, and the implications of that for their studies. In particular, the work of Zelikman, Wu, Mu, & Goodman (2022) _STaR: Bootstrapping Reasoning With Reasoning_
**Other comments**
- Some of the language and logic throughout the paper was sloppy and/or overstated. Here are some specific.
- The logic in the first paragraph doesn’t follow for me: One might hope that the fact that CoT improves performance makes it so the reasoning described by the CoT is an explanation for its predictions, but I wouldn’t say that performance increase alone suggests this.
- In the next paragraph, there is also a short discussion about how the models could have correct, unfaithful reasoning, which I found confusing and I think the point is quite subtle. I guess the authors take “correct reasoning” to not entail that the conclusion follows from the reasoning? Or, is that also part of ‘correct reasoning’?
- In Section 2, the authors write “measuring these effects gives us an account of what models actually rely on to make their predictions, …” I think this is overstated: You don’t know what the model relies upon in the nonbiasing cases.
- I found the discussion around the relevance of subjectivity a little too terse and nonobvious. For example, I kept wondering why switching answers for subjective questions would be concerning, given that subjective questions could have multiple valid answers. In Table 3, the example about Navigation, I found particularly confusing, as it highlights the ambiguity of the task where its not clear if the orientation matters. If the model changes responses based on some biasing features, is that actually problematic? I think the authors could do a lot more to motivate this particular aspect of the evaluations.
- 3.2 The main conclusion that CoT explanations are systematically unfaithful. This is a minor point: This conclusion relies on the point made in footnote 2, that the explanations don’t mention the biasing feature. Footnote 2 should be elevated to a top-line result then, so the reader can see more clearly the logic of the argument being made. This is done in the figure caption, but not in the main text.
- CoT can steer models from correct initial predictions towards bias-consistent predictions: There is no statistical evidence in support of this claim.
- In the BBQ results: The authors talk about “unfaithful explanations” even in the no CoT conditions, and this is generally confusing.
Questions
The limitations above raise obvious questions that I won’t rehash here.
1. From the start, the authors label the auto-regressive generations of LLMs under CoT prompting as “CoT explanations”. But is there any reason to call them explanations? Of course, there’s the empirical fact that the authors are testing… about faithfulness, etc., but from an a priori perspective, what is the step to thinking these might be explanations? As far as I can tell, the CoT prompting is about “thinking step by step”, which I don’t think would fall under any standard definition for what an explanation is.
2. For the models evaluated, do we know whether CoT was used during their training process? This would be an important feature to know about the models.
3. Looking at Table 12 in the appendix (if I’m reading this correctly), it looks like the “Suggested Answer” trick only really works for a very small subset of tasks, and even the “Answer is always A” trick works for about half the tasks. If this is a correct reading, can you speculate why this might be?
4. In the few-shot setting in the BBQ eval, the few-shot prompt uses disambiguated context examples from BBQ and one ambiguous one with an Unknown label. In the appendix, the example shows 2 stereotype aligned responses for the disambiguated cases. Is that always the case? If not, what is the ratio of proportions of stereotype-aligned vs. -misaligned examples among the disambiguated context examples used in the prompts?