Learning to Reason and Memorize with Self-Notes

Summary

The paper presents a general prompting approach for LLMs: instead of generating intermediate "thoughts" after processing the prompt as Chain-of-Thought (CoT), this paper proposes adding "Self-Notes" **while reading the prompt**. That is, while the initial reading of the prompt, the model can generate intermediate Self-Notes that defer the reading of the rest of the prompt until the end of generating the Self-Note. The approach is evaluated across multiple reasoning benchmarks, with supervised models, semi-supervised models, unsupervised models (which I'd call "unlabeled" models, because they *are* supervised on an augmented version of the supervised input), and few-shot prompted models.

Strengths

## Strengths * The idea to generate Self-Notes *while processing the input*, rather than at the end, is simple and novel. It is inspiring to see how such a simple idea has not been used before. * The paper evaluates multiple datasets and tasks: Toy-Story (a synthetic multi-hop QA), algorithmic & boolean (a synthetic source code simulation), Chess Piecetype & Chess Move (a synthetic chess simulation). * The paper evaluates multiple supervision scenarios: supervised models, semi-supervised models, unsupervised models (which I'd call "unlabeled" models, because they *are* supervised on an augmented version of the supervised input), and few-shot prompted models. * The Related Work places the paper well in the related literature. * The paper is well-written and easy to follow

Weaknesses

## Weaknesses * The main weakness is the relatively small models that were used: * In supervised and semi-supervised settings, the base model for both the baseline and the model that Self-Notes is applied on is a **GPT-2-base** (there is actually no `gpt2-base` model on Huggingface. Do the authors mean `https://huggingface.co/gpt2`? If so, this is the **smallest** version of GPT-2, with 124M parameters). While providing a good initial indication, I am not sure that the conclusions for GPT-2 will hold for larger models (because of "emergent abilities" and generally very different behavior of GPT-2-sized models compared to larger models). Even if the authors have access only to a single GPU, I think that there are larger models that can be fine-tuned on a single GPU. * The few-shot prompting experiments were performed on **GPT-J**, which is a bit disappointing - these days, it is relatively easy for the broad community to experiment with large models via prompting API, and new few-shot prompting approaches such as Self-Notes can be easily evaluated on much stronger models such as GPT-3/4 / PaLM / Claude / Codex. * For the GSM-8K benchmark, the few-shot prompting experiments were performed using **GPT-3** (since not mentioned otherwise, I am assuming `text-davinci-001`). Since these experiments could be easily evaluated using `text-davinci-002`, `text-davinci-003`, `code-davinci-002` by just replacing the model name, I am left with the conclusion is that Self-Notes does not work with these stronger models. While these experiments are extensive, I believe that they do not confirm the strong claims of the paper, which hurts its soundness.

Questions

## Questions 1. Wouldn't the approach be stronger if the question was provided **first**, before the context input? Currently, "The model can use notes as a form of working memory by **writing information that might be useful in the future**" (Lines 76-77). But how would the model know what information might be useful in the future, if it hadn't processed the question yet? 2. I would rename the "unsupervised Self-Notes" to "Unlabeled Self-Notes" or something in this spirit, because the model **is** supervised, but as far as I understand, on augmentation of the dataset, rather than on a manually-labeled version of the dataset. 3. Imagine that you had the ability to run GPT-3/4 locally. Isn't expecting the model to generate self-notes much slower than standard prompting? in standard prompting, the model reads the entire prompt in a single forward pass, where all prompt tokens can be processed in parallel. In Self-Notes, don't we must feed the prompt tokens one-by-one, to check if the model has predicted the next token to be "start-of-note"? (and in that case, stop feeding the original prompt, and feed the generated note instead?) ## Summary Overall, the idea presented in this paper is really nice and novel. I also appreciate the authors' empirical efforts across various benchmarks and supervision scenarios. Unfortunately, the idea seems not to work with models newer than the first version of GPT-3 and supervised models that are larger than `gpt2` (small), and thus I gave low "soundness" and "contribution" scores, and overall a "weak accept" score. I would have given a higher score if the paper had demonstrated that the same idea can be useful to improve stronger models such as `text-davinci-002`, `text-davinci-003`, `code-davinci-002`, `gpt-3.5-turbo`, `gpt-4`, and supervised models that are larger than `gpt2` (even `gpt2-large` / `gpt2-xl` can be fine-tuned on a single GPU, I believe). I hope that the authors would strengthen the evaluation for the next version by evaluating with stronger base models. Although I vote for acceptance, I will understand if the other reviewers will argue for rejection because of these evaluation limitations.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Limitations

N/A

Summary

The authors introduce a method called "Self-Notes" that allows the model to think and write down its thoughts during the reasoning process. Unlike other approaches, this method enables the model to deviate from the input context, integrate previous reasoning steps, and enhance its memory with useful information. The experiments show that the Self-Notes method outperforms chain-of-thought and scratchpad methods by interleaving the input text with the model's notes.

Strengths

1. The authors propose a method called "Self-Notes," which involves jotting down thoughts during the reasoning process. 2. The experiments compare their method with different algorithms and demonstrate the advantages of their approach in supervised, semi-supervised, and unsupervised learning settings.

Weaknesses

1. The "Supervised Self-Notes" experiments were conducted using GPT-2. However, it is recommended to use larger open-source models like LLaMA for these experiments. 2. It is recommended to conduct "Semi-supervised Self-Notes" experiments for all tasks shown in the paper. 3. It is preferable to use the same evaluation set for all three types of experiments. For instance, Tables 4 and 5 utilize math word problems for evaluation, which are not included in Tables 2 and 3.

Questions

1. The performance of GPT-3 is significantly low in table 5. What could be the reason behind this? Have you considered using GPT-3.5 for the experiments instead? 2. Is it possible to generalize the "Self-Notes" data from one task to different tasks?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The authors have discussed the limitation and broad impact in the paper.

Summary

One very general and beneficial method for improving the outputs of LMs is called *chain-of-thought* (CoT) reasoning, by which an LM is trained or prompted to first output its step-by-step reasoning before outputting the answer to a problem. This paper proposes a major extension of CoT wherein the LM is trained or prompted to insert its reasoning (as *Self-Notes*) at any relevant points in the token stream, rather than waiting until the end of the prompt, which can be quite long when the problem description is long and complex and contains example solutions. Self-Notes also serve as a form of in-context memory, summarizing key inferences made by the model along the way. The new method is evaluated against LMs with and without CoT, on a wide range of datasets, and employing four different learning paradigms: Supervised, semi-supervised, unsupervised, and few-shot prompted. The experiments clearly demonstrate significant performance gains due to Self-Notes in most settings, particularly in generalizing to cases not seen in training.

Strengths

This work is sound and compelling. The presentation is especially clear and easy to follow. The results are particularly interesting given the current importance of LLMs, and the field’s focus on finding ways to improve LLM reasoning even further.

Weaknesses

Most of the experiments were performed using GPT-2, which seriously lags behind the current generation of LLMs in terms of output quality. A few experiments in this work involved GPT-3, but the field has changed dramatically since the introduction of GPT-3.5 and especially GPT-4, which is particularly good at leveraging few-shot prompts. This work’s contribution could be much greater if the experiments included GPT-4. Section 4.4 mentions that GPT-3 was called through the public API. Given this constrained access, Self-Notes were intentionally limited to appear only at end-of-sentence positions. The same could be done when using GPT-4 through the public API. But the small margin of improvement over CoT shown in Table 5 (using few-shot prompting) raises the question of how effective Self-Notes would be when limited to end-of-sentence positions using GPT-4. All of the examples from the datasets in this work share a certain underlying structure: step-by-step revelation of partial information, in a way that benefits from progressive inference and ongoing memory to track the evolving state. Such tasks seem ideally suited to the insertion of Self-Notes into the token stream. By contrast, the benefit of Self-Notes would be more surprising (and valuable) in the broader set of reasoning tasks that don’t share this special structure. For instance, consider the problem from section 8.2 in [2], where the LLM is tasked to modify exactly one number in an algebraic equation in order to make the right hand side equal a particular value. A chain of reasoning is required here, but the facts in the problem statement are not arranged in an obvious way to provide convenient insertion points for Self-Notes. The significance of Self-Notes would be greater if their benefit were shown to extend to other classes of important problems.

Questions

What are the numeric results shown in Table 4 & Table 5? Accuracies?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No concerns.

Summary

This paper introduces a variation to the chain of thought and scratchpad techniques that can easily be applied to pre-trained transformers. While reading a passage, the model can insert "self notes" at any point in the input sequence. These self-notes allow the model to perform chain-of-thought reasoning, by annotating the input sequence with additional information about characters, variables, etc. Self-notes differ from standard COT because the model can create multiple notes, and place those notes at appropriate locations in the input sequence. The authors compare several different ways of training a model to insert notes: supervised, semi-supervised, unsupervised, and prompting. The "unsupervised" method is the most interesting, because it requires the least amount of human-generated annotations, although it does require that the model be fine-tuned on a QA dataset. IMO, it's not quite "unsupervised", simply a clever way of using existing QA datasets to train a model to "ask itself questions". All three methods produce improvements over SOTA on a variety of tasks.

Strengths

This paper was very clearly written. I especially applaud the authors for concisely illustrating their main contributions in a figure that appears right before the opening introductory paragraph. The self-note mechanism is a fairly simple extension to prior work on scratchpad and chain-of-thought reasoning, but I fully expect it to be high-impact. The technique is easy to understand, easy to implement, works with pre-trained models, and results in large and obvious improvements on reasoning tasks. The experiments are well-designed; the authors compare multiple training methods on multiple tasks, and the chosen tasks are appropriate.

Weaknesses

The main limitation of this work is (1) the requirement for human annotations in the supervised and semi-supervised cases, or (2) the requirement for a Q/A dataset in the unsupervised case. The size of the models are relatively small (GPT2), and the tasks are fairly simple, and partially synthetic. Additional work may be needed to scale this technique up, especially since existing QA datasets are quite small compared to the training corpus for modern LLMs. However, I feel that some combination of unsupervised training and prompting could probably overcome the scaling challenges.

Questions

Have you considered how this technique could be scaled to LLM size? What potential problems do you expect to encounter?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors do not discuss societal impacts. They do briefly mention scaling as an important area for future work, but do not go into any detail about the limitations of their training techniques.

Discussion period

Dear authors, Thank you for your response. I appreciate your efforts and I like the elegant and simple idea proposed in this paper, but I am left with the conclusion that Self-Notes does not work with stronger models than `GPT2-small` and `GPT-3`. >"we observe that the GPT-2 result trends are consistent with the few-shot prompting experiments on the larger models" I do not agree with this claim, since no experiments were presented to support it, and there is much evidence that it is quite the opposite (e.g. ["Emergent Abilities", Wei et al., 2022](https://arxiv.org/pdf/2206.07682.pdf)). >We did evaluate using a GPT-3 model (text-davinci-001). Note that for cost as well as model openness reasons, we didn’t evaluate our models on a whole array of models accessible via a paid API. While I am a strong supporter of openness as well, I cannot accept these arguments: 1. According to OpenAI [[pricing #1]](https://openai.com/pricing#language-models) [[pricing #2]](https://platform.openai.com/docs/deprecations/), **`text-davinci-001` is not cheaper than `text-davinci-002`, `text-davinci-003`, `gpt-3.5-turbo`**. The authors could easily run experiments with these models for the same cost, with a change of a single string. 2. `code-davinci-002` is **free** and still accessible. 3. There are various open-source models, across a variety of sizes, that the authors could have experimented with such as LLama, LLama-2, Vicuna, Falcon, etc. 4. Models such as `gpt2-large` / `gpt2-xl` can be fine-tuned on a single GPU, and the authors persist to use only `gpt2-small` with no convincing justification. The authors argue that "there is nothing inherently limiting about our proposed method as we scale the size of the model up", which is true except for the fact that they did not demonstrate it, and the returns may diminish with larger models. The authors' refusal to consider experiments with newer and larger models leads me to no conclusion other than that Self-Notes does not work with stronger models than `GPT2-small` and `GPT-3`. I hope that the authors will convince me otherwise by the end of this discussion period, or acknowledged that Self-Notes's benefits are limited to smaller models. In the meantime, I am reducing my score.

Discussion period

Dear Authors, Thank you for your response. My primary concern regarding the first weakness mentioned is unresolved in the rebuttal. If you do not want to use GPT-3.5, I'd recommend considering test your method with LLaMa and LLaMA-2. To this end, I've adjusted my score from **5** to **4**. If this concern can be addressed during the discussion period, I am willing to retain my initial score of **5**.

Response to rebuttal

Thank you for the clarifications!

GPT-3.5 and GPT-4

Thank you for providing this explanation regarding other models. Since many readers will wonder why the work does not include GPT-3.5 and GPT-4, could you add a comment to the paper explaining the issues involved, perhaps in the Conclusion?

Author Rebuttal by Authors: Llama2-70B Results

Thank you for the responses. We conducted additional few-shot prompting experiments using the very recent state-of-the-art open source model, Llama2-70B (Touvron et al., 2023), as per the request of the reviewers. For these additional experiments, we used the Algorithmic task, MultiArith, and GSM8K. For GSM8K and MultiArith, we use the 8-shot arithmetic reasoning prompt from Wei et al., 2023. The CoT and Vanilla prompts are the same as from Wei et al., and the Self-Notes prompt adapts the CoT prompt by putting some of the reasoning steps inside the context. For Algorithmic, we use a 5-shot prompt of randomly selected examples from our training set, ranging from 6-33 statements. We test on the 2-100 statement test set from our paper. In summary, we find that Self-Notes outperforms Chain-of-Thought (CoT) and the Vanilla baseline on all 3 tasks. The detailed Llama2-70B results are shown in the table below. We report accuracy (%) for each method. | Dataset | Vanilla | CoT | Self-Notes | | --------- | :----: | :----: | :----: | | Algorithmic | 23.3 | 37.4 | __40.3__ | | MultiArith | 33.3 | 94.5 | __96.2__ | | GSM8K | 16.5 | 55.9 | __59.9__ |

Response to rebuttal

Thank you for including the experiments. I have no concerns regarding your work now. By the way, for the final version, I recommend testing the complete set using all llama2 models.