Large Language Models Are Semi-Parametric Reinforcement Learning Agents

Summary

The paper proposes an LLM + RL architecture for text-based domains. The key idea is to run Q-learning on the side and augment LLM's prompt with Q values for available actions. The evaluation is done one WikiHow and WebShop. Acknowledging the rebuttal, I'm satisfied with authors responses and happy to increase the score.

Strengths

The paper's core idea---adding q-values to LLM prompts---is interesting.

Weaknesses

Clarity is the main issue with the paper. It is hard to read and hard to understand. Here are some of the issues that stand out: - In section 3.2 the Q-learning is introduced. It is not clear how exactly it is performed. Is function approximation used? If so then what is the network architecture. Is it tabular? If so then how come we are encountering exactly the same observations in the test set? - On line 163 the term "flattening" is used. This is not a common term in the literature, what is meant by it exactly? the eq.4 shgiws bootstrapping for Q-value estimation. Is that what is meant? - There is a substantial amount of heuristics that is used to make the system work, but they are not well explained and their influence on the results is not well evaluated. For example, line 227-232 describes categorisation of the web pages into categories. Is this used for q learning? What happens if this categorisation is not used? Same with 243-249 for wikihow. - It is not clear whether the difference to the baselines is significant. For example on WebShop ReAct has avg score of 66.6 (from original paper) and the proposed method has 68. Is that a significant difference? Success rate is 40 for ReAct in the original paper, but is reported 36 in this paper. Where does the difference come from? Since the proposed method achieves the success rate of 38 this warrants a clarification. - Positioning in the introduction is adding to the confusion. The conversation of about using the external memory is misleading, as what is later proposed is Q-learning augmentation.

Questions

Would authors please address the points outlined in the weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper does not have limitations section

Summary

The authors introduced Reinforcement Learning with Experience Memory (RLEM) to update the memory of the LLM agent, enabling it to evolve its capability without fine-tuning the parameters of the LLM. Extensive experiments were conducted on two RL task sets to evaluate the proposed framework. The experimental results demonstrate the superiority and robustness of Rememberer.

Strengths

1. The interesting perspective of exploration is a meaningful attempt to combine LLM with RL. 2. The paper is well-written, easy to understand, and the illustrations are exquisite. 3. The ample experiments demonstrate the effectiveness of the proposed method.

Weaknesses

The discussion and introduction about whether to use RL are not sufficient. It is unclear why RL must be used instead of other methods.

Questions

1. Why use RL instead of other methods? 2. If there are alternative methods to RL, should they be supplemented with comparisons in the experimental section?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Refer to weaknesses and questions

Summary

This paper proposes a framework to combine RL w/ LLM using an offline Q-learning setting. An experience memory component is proposed to store past experience for estimating Q values. Evaluation on WikiHow and WebShop demonstrate the effectiveness of the proposed method and framework.

Strengths

The paper proposes an interesting idea of combing LLM w/ RL. This is a relatively new field. The evaluation results on 2 tasks are strong.

Weaknesses

This work seems very empirical. One question I have is how to generalize this method to other tasks. It will be good to see some theoretical analysis.

Questions

1. It might be better to not use OpenAI logo to represent LLM in the figures. 2. In Equation (1), how is max calculated? 3. It is not clear how the reward is defined and can be obtained. 4. There are a lot of RL issues people have overcome over years, such as over estimation of Q values, sparse reward, using replay buffer to provide iid data. It seems the proposed method will have all these issues. How did this paper tackle them? How do we guarantee that this method will converge?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

It seems the proposed method of storing memory is not very scalable.

Summary

This paper introduces an interesting approach that harnesses the capabilities of large language models (LMs) to tackle reinforcement learning (RL) problems. The method involves estimating Q-functions using RL algorithms and providing advice to the LMs about actions with high and low Q-values. The expectation is that LMs will utilize their "remembering" abilities to effectively solve the problem. Empirical evaluation on two language-based reinforcement learning tasks demonstrate the enhanced performance achieved by the proposed method.

Strengths

1. The idea of combining a fixed large language model and a reinforcement learning component estimating action-value functions by presenting experience and advice in prompts is, as far as the reviewer is concerned, interesting. 2. The paper exhibits a well-structured organization, featuring clear and coherent writing that is easily comprehensible. Additionally, the experiments provide compelling evidence of the advantages offered by the proposed method.

Weaknesses

1. An essential assumption underlying the proposed method is that large language models (LMs) possess the capacity to "reason" by effectively recalling all instances of success and failure. Hence, it is crucial to engage in a discussion regarding the limitations of this assumption to establish a comprehensive understanding. Questions arise about the case where the observation space is extensive, and if LMs implicitly learn certain types of Q-functions. While the proposed method is simple yet seemingly effective, delving deeper into the understanding of simple methods often yields intriguing insights. 2. The ablation studies are not sufficient, which would provide valuable insights and address specific questions. 3. The applicability of the proposed method to tasks with observations in natural language is evident. However, it remains unclear how the LMs can be adapted to tackle problems involving image or vector-based observations. Further elucidation on this aspect is necessary to ensure a comprehensive understanding of the proposed method's adaptability and versatility across different observation types.

Questions

1. Please respond to the questions raised in the weakness section. 2. What is the architecture of the Q function and what is the reinforcement learning algorithm for learning the Q function? 3. An ablation study where the LLM does not have the access to the discouraged actions is expected. 4. Another critical issue, unrelated to the academic content of the paper, requires attention. It has been brought to attention that the usage of OpenAI's brand in figures, if without proper permission, is highly inappropriate and may constitute a violation.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors didn't discuss the limitations of the proposed method. The conclusion section summarizes the contribution of the paper.

Summary

The paper proposes REMEMBERER, a novel framework for Large Language Models (LLMs) that employs a persistent experience memory and a Reinforcement Learning with Experience Memory (RLEM) mechanism. This setup aims to enable LLMs to learn from previous interaction experiences in decision-making tasks, improving their policies. The experience memory acts as an external repository, storing experiences from past episodes, which the LLM can exploit without the need for fine-tuning its parameters. The framework, tested on two recent Reinforcement Learning (RL) task sets, demonstrated an improvement in performance over previous state-of-the-art (SOTA) models.

Strengths

The paper's primary strength lies in its innovative approach to RL using LLMs. The authors introduce an agent framework, REMEMBERER, that overcomes the typical limitations of existing approaches, such as the significant cost of fine-tuning LLMs. The proposal of RLEM, which updates the experience memory through RL training, enables the system to evolve its abilities in an efficient manner. Moreover, the empirical results validate the effectiveness of REMEMBERER, with it outperforming the SOTA models on two RL tasks. This paper is in line with a large number of recent papers that show that LLMs can be used to create agents.

Weaknesses

While the results are encouraging, the paper doesn't delve deep enough into the impact of different configurations of the REMEMBERER system. It would be beneficial to explore how varying the size and management of the experience memory as well as the similarity functions affects system performance. In addition, the paper focuses on improvement against SOTA models but doesn't provide a comparative analysis against other similar approaches that utilize memory or experience in RL, such as the very recent Voyager paper (which is too recent to compare against, but an example of these types of papers). A weakness of this approach is that it requires using a reward function, when reward functions might be unavailable or hard to create in difficult computer tasks. In comparison to other RL algorithms, the approach seems constrained to text domains.

Questions

What is x in the subscript O_x in the methods section?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

In the title, I think it should be “Large Language Models are Semi-Parametric Reinforcement Learning Agents” “Such an agent turns to be a semi-parametric system that can evolve through RL process” -> “Such an agent is a semi-parametric system that can evolve through an RL process” Overall, I think the paper can be cleaned up a lot with better English and notation.

Supplementary result about over-estimation

Hello, we implemented Double Q-Learning and would like to supplement several results here. **Estimation Error on WebShop** | Setting | Epoch | Estimation | Real Training Reward | Abs Error | Relative Error | |:-----------|:-----:|:----------:|:--------------------:|:---------:|:--------------:| | Full Model | 3 | 0.86 | 0.84 | 0.02 | 2.38 | | +DoubleQL | 3 | 0.71 | 0.75 | 0.04 | 5.33 | | +DoubleQL | 6 | 0.69 | 0.77 | 0.08 | 10.39 | **Estimation Error on WikiHow** | Setting | Epoch | Estimation | Real Training Reward | Abs Error | Relative Error | |:-----------|:-----:|:----------:|:--------------------:|:---------:|:--------------:| | Full Model | 3 | 2.48 | 2.60 | 0.12 | 4.62 | | +DoubleQL | 3 | 2.47 | 2.90 | 0.43 | 14.83 | | +DoubleQL | 6 | 2.70 | 2.90 | 0.20 | 6.90 | As shown by the results, iterative update of Double Q-Learning fails to ameliorate the estimation error in *few-step* training. Over-estimation is truly suppressed, however, serious under-estimation may be introduced. [1] * [1] Hado van Hasselt. Double Q-learning. NeurIPS 2010.

Reviewer Response

Thank you for your clarifications, revisions, and ablations. After reading your rebuttal and the other reviews I will keep my score as-is for now.

Keep Score

Thanks for your reply, but the current reply still doesn't address my concerns, so I'm currently choosing to keep the score.

Thanks for you response

The author's response has solved most of my concerns. Thanks a lot. It is interesting to see that the LLM does not learn a accurate Q function but a reasonable preference order. It makes the reviewer curious about what will happen in continuous action spaces as future work. The reviewer is still concerned that the proposed method can not be easily extended to problems with extensive observation spaces. Ratings have been increased accordingly. Remember to update your figures!

Representation of Atari observation in text format

Hello. It seems that we cannot produce new results on Atari in time and we will briefly describe our attempt to represent an Atari [1] observation in text here. Let's pick the environment "Breakout" as an example. States of three components are described in the observation: bricks, the ball, and the paddle. *E.g.*, ``` Bricks: (10, 92), (50, 92), (80, 92), (20, 86), (100, 86) Ball: Position: (70, 157) Velocity: 5 Direction: up +30 Paddle: 100 ``` As the bricks may be too many to present their position, we propose to include only a part of them through some heuristics (*e.g.*, the lowest ten bricks). Detailed information can be extracted directly from the RAM state to form such a representation if "RAM" observation mode is adopted. As for "RGB" mode, a simple detection model should help. This is a trivial plan of how to leverage a text LLM on video environments like Atari. To describe such visual-rich observations does require more efforts than text-rich observations sometimes, but it is still possible. * [1] https://www.gymlibrary.dev/environments/atari/breakout/

Looking forward to your reply

Hello. The author-reviewer discussion period is going to end. We wonder if our rebuttal solves your concerns. If there remains questions, we are willing to conduct further and deeper discussion with you.

Reviewers' responses

At the moment I still think that some RL (or other angle) comparison method is lacking, so I reserve my concerns