Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models

Summary

The authors propose a new framework to optimize prompting for various tasks. The frameworks consists of a buffer, where high-level problem-solving templates are stored. A query is first processed by extracting its key aspects.Templates are selected using embeddings of the templates and the extracted key aspects query. The high-level problem-solving templates is then instantiated with the extracted key aspects of the query and its inference is executed. Additionally a buffer manager is introduced to allow updating of the thought template buffer with new, "specialized", thought templates.

Strengths

The paper combines the idea of RAG with a library of templates for solving tasks in order to make automatic prompt engineering more flexible and by using a cache (the library) more efficient, since it avoids more complex exploration. The framework and its components are sufficiently outlined and their description is clear. The writing is easy to follow. The evaluation shows clear advantages over existing approaches, while being executed with modern LLMs.

Weaknesses

Claims from the checklist: * point 6: It is not clear, what embedding model is used. * point 8: No sufficient information are provided for the compute resources, especially the local (?) Mixtral execution. Not clear, how well their approach scales with the number of different thought templates (and not just specialized one). Additionally it is not clear, how well their approach works, if the query (something completely different) does not match any of thought templates in the buffer, as at least some of thought templates seem to specifically designed for the respective evaluation tasks. This is also one of the weaker points, since the authors claim that their approach requires less manually labor from the user than other approaches. I would have appreciate some examples for the tasks in the appendix, so that it is not necessary to look them up in the respective code/papers. The language can be polished a little bit more. For example: * lines 29/30: "which makes it impractical to manually design them task by task" - it is just cumbersome to manually design them, but not impossible * lines 69/70: "the retrieval-augmented LLM first queries an external database with billion-level tokens [23] for retrieving a subset of the text" - I think that is just how that particular paper implemented RAG, but it is not a general description of the idea behind RAG. * typos: * "enahnced" (line 78) * Llama-70B (line 277) references could also use a little but of work, for example: * reference [27]: no place of publication * references are slightly inconsistent: * for example one time the abbreviation ICLR is used, otherwise not * sometimes the number prefix (for example Eleventh) for conferences is used, sometimes not

Questions

* What embedding model is used for the thought buffer? * Why it is too computationally expensive to provide error bars for the existing results? It should be possible to compute them locally at least for some of the tasks. * Figure 3: * What is logarithmic time? * Why is the inference time of BoT not at least twice that of the Expert baseline, since it needs at least two LLM interactions, where as the Expert baseline only requires one (if I understood that correctly)?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are reasonably explained.

Summary

This paper proposes a novel reasoning procedure for LLM, named "buffer of thoughts". The core idea is that for each task, first extract a template describing how the task should be solved, then store all these templates in a "buffer". As the LLM receives a new query, a retrieval procedure is applied to extract the most relevant task template out of the buffer. The template is then instantiated for this specific query, which becomes a concrete guideline that instructs the model how to solve the new query. The paper demonstrates that this procedure outperforms baselines like tree-of-thought and graph-of-thought significantly on some reasoning tasks.

Strengths

The novelty of the proposed method "buffer of thought" is a good contribution of the paper which may significantly improve LM's reasoning ability for some tasks.

Weaknesses

Generalizability and robustness of the proposed approach needs to be further verified, as the introduction of a hierarchical reasoning procedure (abstract template + instantiation) may create more noise and make it less reliable for novel tasks.

Questions

1. The success of the proposed approach critically depends on the quality of the automatically induced template. While the paper showed empirical performance on downstream tasks, it remains unclear how good the templates themselves are. It would be great to make a comparison between the automatically generated task templates with manually prepared templates for some tasks. 2. It remains unclear to me how the template for a specific task category (say maths) can potentially be revised and improved, as the model sees more and more examples from that task. Although the paper described briefly how the buffer can be dynamically updated (line 182), I don't quite understand how the template is updated automatically for higher accuracy as the model receives more examples (this seems to be a verbalized gradient descent procedure, but no quality improvement is guaranteed). 3. What are the embedding models you used for retrieval purposes? Also it would be great to give some analysis on the impact of retrieval quality on downstream task performance, as finding the right template is critical. 4. In Eq 5, it seems whether the template should be replaced should depend upon the template quality, instead of its similarity with the embedding vector. 5. How quickly can this approach adapt to an unseen task? Would one or two examples from new task be enough to achieve reasonable accuracy? It would be great to give some analysis to the adaptability and continual learning ability of the proposed approach.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper properly addressed limitations and impact of the work.

Summary

This paper proposes a new approach called Buffer of Thoughts (BoT) to improve LLMs reasoning abilities. BoT addresses this by creating a "meta-buffer" that stores general problem-solving “thought” templates across different tasks. When a new query is given as input, BoT retrieves a relevant thought from the meta-buffer and tailors it to the specific situation. The method also includes a "buffer-manager" to keep the meta-buffer up-to-date and effective as the LLM encounters new challenges for scalability and stability purposes. The authors perform experiments on different reasoning tasks that showed significant improvements with BoT compared to previous methods.

Strengths

- The framework is clearly explained and the paper is easy to follow - The empirical evaluation is thorough - The improvement for some tasks is notable. - The idea of a library of thoughts templates is sound and novel

Weaknesses

- BoT seems to work well for problems with existing templates in the meta-buffer. However, entirely new or unique problems might not have a relevant template for BoT to adapt, potentially hindering its ability to solve them effectively. - What happens in the case of unsolvable queries? Does the method allows for uncertainty based abstention?

Questions

see weaknesses.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

limitations have been discussed in the paper.

Summary

The paper introduces "Buffer of Thoughts" (BoT), a novel framework designed to improve the reasoning abilities of large language models (LLMs) by incorporating a 'thought-augmented' approach. This framework uses a component called 'meta-buffer' to store high-level thoughts—concise, distilled reasoning strategies from various problem-solving instances—which can be dynamically retrieved and adapted to new tasks to facilitate efficient reasoning. This approach significantly enhances the accuracy, efficiency, and robustness of LLMs across multiple reasoning-intensive tasks. 1. Novel Framework: BoT innovates a thought-augmented reasoning mechanism that leverages previous problem-solving insights to aid new reasoning tasks, reducing the need for generating reasoning paths from scratch. 2. Meta-Buffer: Introduces a storage system that holds distilled high-level thoughts, allowing for rapid adaptation to different tasks by retrieving and instantiating these thoughts as needed. 3. Buffer Manager: A dynamic management system that updates the meta-buffer based on newly encountered tasks, continuously improving the system’s reasoning capacity. 4. Empirical Validation: The paper reports extensive testing across 10 complex tasks, demonstrating significant performance improvements over state-of-the-art methods, including improvements of 11% on Game of 24, 20% on Geometric Shapes, and 51% on Checkmate-in-One, while also reducing the computational cost compared to traditional multi-query prompting methods.

Strengths

1. **Improved Reasoning Accuracy:** BoT significantly enhances the reasoning accuracy of large language models by leveraging distilled high-level thoughts, allowing the models to approach problems with a pre-formed strategy that has been proven effective across tasks. 2. **Computational Efficiency:** The thought-augmented reasoning approach minimizes the need for complex and iterative query processes typical in multi-query prompting systems. By reusing structured thought templates, BoT reduces the computational overhead, leading to faster reasoning times. 3. **Robustness Across Tasks:** BoT exhibits a robust performance over a range of different and challenging tasks. This is attributed to the system's ability to adapt high-level reasoning thoughts to new problems, ensuring consistent performance without the need to tailor the system for specific tasks.

Weaknesses

1. **Dependence on Quality of Distilled Thoughts:** The effectiveness of BoT hinges significantly on the quality of the distilled thoughts stored in the meta-buffer. If the initial thoughts distilled from problem-solving processes are not sufficiently generalizable or are too simplistic, they may not provide the necessary depth for complex reasoning tasks. 2. **Scalability and Maintenance of the Meta-Buffer:** The paper introduces a dynamic system for updating the meta-buffer but does not deeply explore the long-term scalability and maintenance challenges associated with continuously growing and updating this repository. Managing an ever-expanding set of thought templates could lead to efficiency issues or dilution of useful thoughts. 3. **Risk of Overfitting to Distilled Thoughts:** There is a potential risk that the LLM might overfit to the specific styles or patterns of reasoning encapsulated in the thought templates, especially if these templates are derived from a limited set of problem-solving instances.

Questions

1. As the meta-buffer grows with more distilled thoughts, how does its size impact the retrieval time and overall performance of the model? 2. In cases where multiple applicable thoughts could be retrieved from the meta-buffer, how does BoT prioritize or choose among conflicting reasoning strategies? 3. Given the reliance on previously distilled thoughts, how does BoT handle situations where the foundational thoughts might be based on incorrect or outdated information?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

One of the primary limitations highlighted in the discussion of the Buffer of Thoughts (BoT) framework is its dependency on the initial quality of the meta-buffer. The performance of BoT is contingent upon the initialization of this buffer. If initialized with a weak model, the distilled thoughts stored in the meta-buffer may not be of sufficient quality or depth to facilitate effective reasoning across diverse tasks.

Official Comment by Reviewer hZUQ

Thank you for the comprehensive response and additional experiments provided. I find this to be a very interesting and excellent piece of work and look forward to meeting the authors at the conference. I have increased my score from 7 to 8, and I hope the authors will incorporate these suggestions into the final manuscript.

Response to rebuttal

I appreciate the authors' responses to my questions. I maintain my current score and evaluation.