Contrastive Decoding Improves Reasoning in Large Language Models

Thank you for your detailed review and feedback! We have updated the paper to address some of your concerns, and are open to further discussion and changes as need be. ### Responses to Weaknesses: - **Consistency of results:** We have highlighted that contrastive decoding yields consistent improvements when coupled with self-consistency on all measured tasks, providing at least one generation paradigm in which it is always shown to help. Further, we include analysis for the generation tasks on which contrastive decoding fails to improve performance -- for instance, the smaller expert models perform worse than chance on AQuA and therefore do not have a clear advantage over the amateur model to contrastively exploit, and the expert’s overall performance on the MATH benchmark is quite low. We also now include more analysis in Section 3.5 about the challenges of CD on ARC-Easy (inconsistent scaling behavior) and MMLU (factual-heavy). - **Error analysis:** We now include example generations with and without CD in the appendix of the paper. Most differences are difficult to attribute to CD for reasonable values of beta, and so we rely on the differences in automatic metrics (e.g. n-gram copying) as well as the manual error analysis in Table 5 to give insight into contrastive decoding’s strengths and weaknesses. - **Chain-of-Thought Limitation:** We now clarify in both Section 3.3 and 4.2 that contrastive decoding only works when the expert is significantly better than the amateur at a given task. Previous work from Li et al find benefits to open-ended text generation, and our experiments show that non-chain-of-thought performance does not improve as much as chain-of-thought performance with scale, suggesting a lack of a sufficient contrastive gap to exploit. In other words, solving these tasks effectively with an untuned model requires chain-of-thought generations both with and without contrastive decoding. - **LLaMA Models:** Contrastive decoding, and contrastive-decoding-like strategies like coherence boosting (Malkin et al 2022) and DoLa (Chuang et al 2023) have been shown to improve performance on open-ended task generation. Most of these works use one to two families of models; we include the FLAN-T5 study to show that our method works out-of-the-box on a family of models without tuning any additional hyperparameters. Further, the original Contrastive Decoding work found improved open-ended generation on OPT and GPT-2, suggesting that the efficacy of scale-based Contrastive Decoding is not unique to the LLaMA family of models. ### Responses to Questions - We have now included example generations in the appendix for qualitative analysis. It is important to note that CD does not significantly harm performance on the MATH benchmark (a maximum of -0.4% absolute performance), even though it does not bring the same benefit it does to other arithmetic reasoning tasks. - We have refactored 2.1 to give more background on the original method, and in doing so emphasize the range of possible values for alpha. - In this context, GSM8K “EM” is the exact match accuracy on GSM8K results. For clarity, we have changed the label to “Accuracy (GSM8K)”. - Similarly, we have changed the caption to read “Accuracy on the GSM8K arithmetic reasoning benchmark.” - We include the n-gram F1 scores for both greedy decoding (blue bars) and contrastive decoding (orange bars), with greedy decoding as the baseline in Figure 6. - As $\beta$ goes to $\infty$, the ratio term dominates the the prior term, assuming there is at least one item in the feasible set which the expert assigns higher probability than the amateur (creating a ratio greater than one which the beta exponent would amplify infinitely above the rest).

Thank you for your detailed review and feedback! We have updated the paper to address some of your concerns, and are open to further discussion and changes as need be. ### Response to Weaknesses **Novelty** While the central method of our paper draws on Li et al 2022, we believe that our reformulation of the contrastive objective and application to reasoning tasks constitutes a novel contribution that fills a clear research gap. Contrastive decoding was originally proposed as an alternative to sampling methods in order to improve open-ended text generation in language models. Given a single generation, most decoding methods that produce more diverse text (e.g. temperature sampling, nucleus sampling) harm performance on reasoning tasks. In showing that contrastive decoding improves reasoning on performance tasks, our findings shift a common understanding of decoding methods: that there is a hard tradeoff between diversity and reasoning ability. Beyond our findings, our reformulation constitutes a novel benefit to research on contrastive decoding in the future. The application of too-large amateurs under the original, non-$\beta$ formulation of CD has led other papers to wrongly conclude that contrastive decoding harms reasoning. (DoLa; Chuang et al 2023) The $\beta$-parameter formulation allows better fine-grained control over CD, allowing a continuous reduction to standard greedy decoding in a way that the original formulation does not and making it more straightforward to apply to most problems (with a reasonable search space of [0, 1] rather than [1, $\infty$]). Besides this reformulation, we combine self-consistency with contrastive decoding for the first time, finding greater benefits than either alone. Negative prompting and contrast against a partially trained amateur are both novel methods that improve results as well. Thus, we believe that our paper presents several novel methods to study, in addition to a key reformulation of a previous work as well as counterintuitive findings that fill a clear research gap. **Presentation:** Thank you for this feedback -- for clarity, we have expanded section 2.1 with more background on the original method prior to our reformulation. ### Response to Questions - Yes; distilling the contrastive objective into a single language model is an ongoing work. It remains out of scope for the purposes of this paper, which focuses on merits of the training-free, inference-time intervention form of contrastive decoding as it pertains to reasoning tasks. Again, thank you for your feedback to help us make the paper clearer. Please let us know if this helps to address your concerns, or if there remain sections of the paper that remain unclear, and we would be happy to update the paper accordingly.

Thank you for your detailed review and feedback! We have updated the paper to address some of your concerns, and are open to further discussion and changes as need be. ### Response to Novelty While the central method of our paper draws on Li et al 2022, we believe that our reformulation of the contrastive objective and application to reasoning tasks constitutes a novel contribution that fills a clear research gap. Given a single generation, most decoding methods that produce more diverse text (e.g. sampling) harm reasoning. In showing that CD improves reasoning, our findings contradict this supposed tradeoff. Beyond our findings, our reformulation provides a novel benefit. The application of too-large amateurs under the original, non-$\beta$ formulation of CD has led other papers to wrongly conclude that CD harms reasoning. (DoLa; Chuang et al 2023) The $\beta$-parameter formulation allows more fine-grained control over CD, allowing a continuous reduction to standard greedy decoding in a way that the original formulation does not and making it more straightforward to apply to most problems (with a reasonable search space of [0, 1] rather than [0.5, $\infty$]). Besides this reformulation, we combine self-consistency with CD for the first time, finding greater benefits than either alone. Negative prompting and contrast against a partially trained amateur are also both novel methods that we find to improve results. Thus, we believe that our paper presents several novel approaches to study, a key reformulation of a previous work and surprising findings that fill a clear research gap. ### Responses to Other Strengths and Weaknesses - **Interpretability:** Our formulation replaces two opaque hyperparameters (amateur & expert temperatures) with one that cleanly corresponds to contrastive strength, which is orthogonal to the choice of output temperature. It also allows a continuous reduction to greedy decoding (by sending $\beta$ to zero). - **Coherence boosting:** Thank you for drawing our attention to this paper. Our paper now references coherence boosting and several other CD-inspired papers (CID, CAD). CB benchmarks on some shared ranking tasks as our study in Table 4, although we do not optimize beta on every task in the same way so as to avoid overfitting to each benchmark. Our central results, on generation tasks for reasoning, remain unique. - **Inconsistent results:** It is difficult to extract a trend from StrategyQA and CSQA, both of which are noisy and at times hurt or helped by CD. CSQA in particular behaves oddly in some ways; for example, maj @ 20 only boosts performance 0.9%. CD is also weak at factual recall, which is somewhat entangled with commonsense reasoning and could help explain the gap between performance on mathematical and commonsense tasks. Still, the aggregate trend with CD is relatively neutral for commonsense and positive for mathematical reasoning. ### Responses to Questions - **Coherence boosting:** This is a good suggestion for a baseline, but the hyperparameter selection of $\alpha$ and the context window $k$ is too costly to optimize and then benchmark effectively during the rebuttal period; note that coherence boosting is much more costly in FLOPs than CD as well, and so it isn't a completely fair comparison from an efficiency standpoint. - **Smaller Models:** We are also interested in studying smaller models. Cross-family models are possible, although it requires aligning different tokenizer vocabularies. - **Non-CoT Results:** There is not as large of an expert-amateur gap without CoT; further, one-step QA is a poor benchmark for a decoding method. CD’s main benefits (fewer skipped steps, fewer semantic errors) are not helpful without explicit reasoning steps. We notice a similar result for factual recall, which CD harms. - **FLoPs:** **Importantly, we use a single $\beta$ value (0.5) across all tasks, after optimizing it only on GSM8K.** Thus our results do not rely on additional compute to search for an task-optimal $\beta$. This deflates our results; we could likely achieve better per-task performance by more carefully fitting $\beta$. However, our results show that a user of CD could find benefits across many tasks with one choice of $(\alpha, \beta)$. - **Figure 1:** Thank you for the suggestion. We have added per-task averages to Figure 1. Again, thank you for your feedback -- we hope that this addresses some concerns you had with the paper. Please let us know if there is anything that remains unclear and we would be happy to update the paper accordingly.