Automated Rewards via LLM-Generated Progress Functions

Summary

The paper presents an innovative framework that uses LLMs to generate task-specific progress functions for reinforcement learning. The proposed system aims to overcome the challenge of reward engineering in sparse-reward tasks by focusing on generating progress-based intrinsic rewards. The paper introduces a two-step approach that uses LLMs to generate progress functions and then applies count-based intrinsic rewards. The authors evaluate their approach, ProgressCounts, on the challenging Bi-DexHands benchmark, and show that it achieves comparable or better results than previous state-of-the-art methods while requiring significantly fewer samples.

Strengths

- The idea of simplifying reward generation by reducing it to the problem of estimating task progress is innovative. Leveraging LLMs to generate task-specific progress functions is a creative application that shows promise in reducing reliance on dense reward shaping. - The method shows significant sample efficiency gains compared to Eureka, requiring up to 20 times fewer reward function samples while achieving state-of-the-art performance on the Bi-DexHands benchmark. This demonstrates the potential of the approach in reducing the computational cost of training. - The experimental results on Bi-DexHands are impressive, as ProgressCounts outperforms both manually designed dense reward functions and existing automated reward generation methods in terms of task success rates.

Weaknesses

- The evaluation is conducted solely on synthetic benchmarks like Bi-DexHands and MiniGrid, which may not be fully representative of real-world environments. The generalizability of ProgressCounts to more diverse and real-world complex environments remains uncertain. - The method relies heavily on LLMs to generate progress functions, which introduces potential biases inherent to LLMs. These biases can affect the quality and reliability of progress estimates, particularly in environments where the LLM lacks specific domain knowledge or makes incorrect inferences. - The need for a pre-defined feature engineering library for each domain limits the method's automation potential. Although the feature library is relatively simple to create, this step still requires manual intervention and domain-specific expertise, which may limit scalability and practical deployment. - While the proposed count-based intrinsic reward framework is well-developed, the paper lacks a thorough exploration or comparison with other recent reward design techniques, such as curiosity-driven approaches or model-based reward mechanisms. This leaves open the question of whether ProgressCounts is the optimal method for all types of tasks. - The paper does not address the scalability of ProgressCounts when moving to tasks with highly complex environments or large state-action spaces. Specifically, the discretization strategy and count-based approach may face challenges when scaling to significantly more complex scenarios, potentially reducing sample efficiency.

Questions

How well does the proposed ProgressCounts framework generalize to more complex, real-world environments beyond synthetic benchmarks like Bi-DexHands and MiniGrid? Have any tests been conducted to validate its applicability to a broader set of domains? Given that the progress functions are generated using LLMs, how does the framework address potential biases that the LLM may introduce, particularly in environments where LLMs may not have specific domain knowledge? The paper relies on a discretization strategy and count-based intrinsic rewards. How scalable is this approach when applied to tasks with significantly larger state-action spaces or more complex environments? Would sample efficiency gains still hold under such conditions?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Summary

This submission concerns automatic reward shaping for RL tasks by utilizing LLMs for code synthesis, i.e., generating a shaping function. In particular, it improves on prior work (EUREKA) in constraining LLM output to "progress functions", which can serve as a basis for state binning, which in turn enables the use of count-based intrinsic rewards. Putting everything together, the authors demonstrate strong performance on a robot hand manipulation benchmark while significantly reducing the required samples of EUREKA (which generates variations of shaping functions by re-prompting the LLM with feedback obtained from running an RL training algorithm with the shaping function).

Strengths

For the most part, the paper is easy to read and follow; the motivation is clear and the method is described well. Generally, I find the link between automatically generating a symbolic notion of task progress by an LLM (task description to progress function) and utilizing it to derive count-based rewards (which can be very effective if designed well) original and interesting; the experimental results demonstrate the efficacy quite well. I also liked that the evaluation focused on required training runs and environment interactions, where the proposed method is vastly superior to EUREKA.

Weaknesses

First, I have doubts regarding the significance of this work. The main message -- you can get more out of an LLM's ability to translate language to RL specifications by placing additional constraints and supplying further human knowledge -- is timely and line with other recent works from the code generation community. On the other hand, the paper largely follows the problem setting of EUREKA, which, to my knowledge, has yet to demonstrate value beyond being an interesting application of LLMs. The original paper on defining rewards with LLMs (https://arxiv.org/abs/2306.08647) utilized model predictive control to directly translate the reward specifications to actions; count-based exploration does not allow for this, however, and relies on training a policy over millions of time-steps. Further points: - In Table 1, the same number of trials should be used for all methods. From 5.1. I understand that you select the best trial run, so a fair comparison should provide the same number of trials to all methods considered. Finally, the presentation could benefit from additional clarity, in particular the experimental section is at times hard to follow. Some concrete suggestions: - In the first sentence of the abstract, you might want to mention that you're ultimately concerned with RL. "Automated reward engineering" is not a term I could immediately place in a specific field. - Please provide the `Env` class. It's not clear to me whether the progress function has access to the same state as the policy or whether additional variables are provided - Supply hyper-parameters such as learning rate for PPO - The claim that the feature engineering library can be authored in minute (L164) is quite bold, in particular considering new domains and a lack of guidance for non-RL experts on what this library should contain. - In Figure 3 or in the corresponding text, be explicit on how many environment samples are used.

Questions

- Do your progress function break the MDP assumption? E.g., the BFS helper method for MiniGrid requires full access to the full environment state? - Did you try prompting the LLM to always supply progress variables where progress strictly means an increase in value? This way you would not need the additional variables. These questions should be answered in the text: - For the MiniGrid experiment, how many runs of ProgressCounts were done? - On which data do you base the range estimation of the progress variables? I understand you need several example rollouts (ideally successful) for this?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Summary

This work proposes an algorithm to generate progress functions with fewer samples. This is accomplished by mapping the original observation space to a lower-dimensional representation, discretizing this space, and calculating intrinsic rewards based on the inverse square root of visitation counts. Compared to previous baselines, this approach achieves state-of-the-art (SOTA) performance while utilizing fewer samples.

Strengths

- The use of domain-specific discretization within the projected subtask progression space is novel and yields promising results. - The experiments are thorough, with ablation studies providing valuable insights into the function of each design component. - The empirical results are robust, demonstrating improved performance alongside reduced sample complexity.

Weaknesses

In Figure 4, could you also provide results from Eureka? It would be interesting to see how the proposed method and Eureka performance improve as more samples are available.

Questions

How are the additional variables $[y_1, y_2, \ldots, y_k]$ used?

Rating

8

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Summary

The paper presents a novel framework leveraging Large Language Models (LLMs) to automate the generation of reward functions for reinforcement learning tasks. The authors claim significant improvements in sample efficiency and performance over prior state-of-the-art methods on the Bi-DexHands benchmark.

Strengths

1. The paper is well-organized and easy to follow, making complex concepts accessible. 2. The overall motivation behind using LLMs for automated reward engineering is compelling. 3. The experimental results are substantial, demonstrating clear performance gains over existing methods.

Weaknesses

1. The motivation for mapping progress to bins as a representation for intrinsic rewards is not clearly articulated. A common principle in intrinsic reward design is that the representation space should effectively capture the essential aspects of the original observation space. By incrementing counters in this space, the algorithm should be able to explore the full state space effectively. The authors need to provide a clearer rationale for why progress bins are chosen over other potential representations. 2. The method for selecting the best progress function among the generated options is not detailed (Line 287-289). The authors should provide a clear selection criterion to ensure reproducibility and allow for comparison of different progress functions. 3. The paper claims that progress-based bins are superior to simhash, yet it is not evident why this should be the case. Simhash is known to provide convergence guarantees by ensuring that increasing intrinsic rewards correspond to increased visits to the original full state space, though potentially requiring longer training times. 4. On Line 427, the authors suggest using the cumulative progress as a step reward for task completion, which is misguided. The correct approach, as per reward shaping principles, is to use the difference in progress.

Questions

see weaknesses above.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Response to reviewer feedback

## Summary of reviews We thank the reviewers for their insightful feedback and constructive suggestions. Our work presents an innovative framework for generating reward functions by reducing reward engineering to task progress estimation and leveraging LLMs to generate progress functions. This approach demonstrates substantial **sample efficiency gains**, requiring up to 20x fewer reward function samples than Eureka (ICLR 2024) while achieving state-of-the-art performance on Bi-DexHands (foaV, 7kFc). We provide **strong empirical results** on both Bi-DexHands and MiniGrid benchmarks, illustrating both superior performance and reduced computational costs (foaV, mw3c). Furthermore, we conducted **thorough ablation studies** to isolate and evaluate the contributions of individual design components, offering insights into their roles (foaV). By linking symbolic notions of task progress with count-based intrinsic rewards, our work contributes a novel method to the RL literature, which we validated with robust evaluations focusing on training efficiency and task performance (mw3c). ## Clarifications on some key points **Focus on comparison to Eureka:** Automated reward generation has the potential to vastly reduce the amount of human iteration required for training RL policies. **We present our work in comparison to Eureka (ICLR 2024) both because it was the prior SOTA work on Bi-DexHands and because it used LLM-based reward generation.** Bi-DexHands presents a substantial state-action space comparable to real-world robotics tasks. Both progress functions and traditional reward shaping rely on simplifying the complexity of state-action spaces, and our approach demonstrates better performance in these settings. **Creating a feature library:** We agree that creating a feature engineering library for new environments requires some human effort. However, this effort is modest compared to designing dense rewards or crafting viable observation spaces, which are typically required in RL. For practitioners already skilled in RL, we believe this tradeoff is beneficial–especially due to the SOTA performance and 20x efficiency gains. **We have revised the text to better align this claim with the expectations for RL practitioners.** **The choice of progress functions over SimHash:** We appreciate the request for greater justification of our choice of progress bins over alternatives like SimHash. Progress bins discretize the state space by leveraging domain knowledge, which empirically allows them to better capture task-relevant aspects of the environment. As seen in prior work [1], **human-informed hash functions using domain knowledge, while lacking better theoretical properties, empirically outperform generic representations like SimHash.** SimHash offers convergence guarantees, but progress bins provide practical benefits under finite training budgets, as evidenced by our empirical results in Table 1. **How we select the best progress function, and how we evaluate it:** Following Eureka’s experimental protocol, the progress function that achieves the highest task success during a single training run is selected **in an automated fashion (Lines 311-313, also in the original pdf)** and the selected function is evaluated over five additional runs with different seeds **(Lines 298-300, also in the original pdf). Please note the distinction between the function selection protocol (select best training run) and evaluation protocol (report the average performance of the selected function on 5 additional runs). This is the protocol from Eureka.** In the ablation presented in Table 1, the comparison is fair because the ablated methods simply are averaging across fewer evaluation trials. **Progress vs progress differences as reward:** We acknowledge that using cumulative progress as a reward may be less conventional than using progress differences. However, on Bi-DexHands we ran experiments using both approaches, and cumulative progress achieved an average success rate of 0.45, whereas using progress differences only achieved an average success rate of 0.32. Therefore, we included results directly using progress as reward to offer the best possible comparison to dense rewards. **We have presented the difference between these two dense reward approaches in our updated Appendix A.8.** [1] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.