Behavior Alignment via Reward Function Optimization

Designing reward functions for efficiently guiding reinforcement learning (RL) agents toward specific behaviors is a complex task. This is challenging since it requires the identification of reward structures that are not sparse and that avoid inadvertently inducing undesirable behaviors. Naively modifying the reward structure to offer denser and more frequent feedback can lead to unintended outcomes and promote behaviors that are not aligned with the designer's intended goal. Although potential-based reward shaping is often suggested as a remedy, we systematically investigate settings where deploying it often significantly impairs performance. To address these issues, we introduce a new framework that uses a bi-level objective to learn \emph{behavior alignment reward functions}. These functions integrate auxiliary rewards reflecting a designer's heuristics and domain knowledge with the environment's primary rewards. Our approach automatically determines the most effective way to blend these types of feedback, thereby enhancing robustness against heuristic reward misspecification. Remarkably, it can also adapt an agent's policy optimization process to mitigate suboptimalities resulting from limitations and biases inherent in the underlying RL algorithms. We evaluate our method's efficacy on a diverse set of tasks, from small-scale experiments to high-dimensional control challenges. We investigate heuristic auxiliary rewards of varying quality -- some of which are beneficial and others detrimental to the learning process. Our results show that our framework offers a robust and principled way to integrate designer-specified heuristics. It not only addresses key shortcomings of existing approaches but also consistently leads to high-performing solutions, even when given misaligned or poorly-specified auxiliary reward functions.

Paper

References (72)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer tPX36/10 · confidence 3/52023-06-22

Summary

This paper focuses on how to better combine primary rewards that are typically sparse with auxiliary rewards designed by human heuristics to maximize the primary rewards. The authors propose a bi-level optimization procedure, where the upper level optimizes a behavior alignment reward to guide agent to maximize the primary rewards and meanwhile minimizes the discount value to force the learned rewards to encode long-horizon information, and the lower level can be any policy searching algorithm. They provide both intuitive and theoretical analysis to show that the formulated procedure is not only robust to misspecified auxiliary rewards, but also overcomes the imperfection of policy optimization algorithms, such as biased policy gradients caused by omitted discounted term and no importance sampling in off-policy learning. Besides the theoretical analysis, the authors also provide two methods to empirically solve this untractable bi-level problem since the optimization of upper level problem requires characterizing the impact of learned rewards and discount value on the entire optimization process. One is a gradient based method that makes use of the implicit gradient denoted as Implicit Bi-level optimization and the other takes into account the complete path followed by the inner optimization loop denoted as Path-wise Bi-level optimization. The authors also conduct experiments on some case studies to show the robustness and extent to Mujoco tasks to show the scalability to high-dimensional continuous control.

Strengths

1. This paper presents a good motivation to motivate why this work is of great significance and why previous works like potential based methods fail (higher variance and only state-based potential function). 2. This paper dives deep to analyze the advantages of the proposed bi-level objective (Section 3), which is helpful for readers to catch the main idea of this paper. 3. The analysis in Section 3.1 seems novel to me. The proposed bi-level objective can not only address auxiliary reward imperfection but also the imperfection of policy optimization algorithms, which is novel and valuable.

Weaknesses

1. The resulting practical algorithms seem too complex to conveniently apply for higher-dimensional tasks like vision-based tasks. 2. The proposed method introduce a lot of additional hyperparameters as shown in Table 2.3.4.5, which may be hard to tune. For instance, in Table 3, $\alpha_\theta$ is tuned to 0.015625, which does not seem like a common value and may be carefully tuned. 3. The experimental results are not so convincing. In figure 3, Barfi only wins 1 out of 4 tasks. Barfi also cannot outperforms baselines in figure 2. 5. 4. The authors claim that Barfi requires a warm-up stage, after that, Barfi can fastly adapt to find good policies. However, the reasons why Barfi requires warm-up and how does Barfi performs without such warm-up remains unclear. 5. The empirical analysis part is hard to follow, it would be better if the authors could separate some large paragraphs and simplify the abundant detailed task descriptions but focuses on more in-depth analysis. There is also a recent work that focuses on correcting the imperfect rewards using a bi-level objective in the offline RL setting [1] [1] Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., Jia, Q. S., & Zhang, Y. Q. Mind the gap: Offline policy optimization for imperfect rewards. ICLR2023.

Questions

Please refer to weaknesses for details.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to weaknesses for details.

Reviewer tPX32023-08-13

Thanks for the detailed responses

Thank the authors for the detailed responses. I'm happy to keep my score unchanged (6 weak accept).

Reviewer GcHf6/10 · confidence 3/52023-07-02

Summary

This paper proposes a novel framework that employs a bi-level objective for learning a “behavior alignment reward” to find the exact reward specification that elicits the desired behavior in RL. Specifically, the reward function combines auxiliary rewards, defined by a designer’s heuristics, with primary rewards provided by the environment. The experiment shows the performance of the proposed method. However, I have some concerns about this paper. My detailed comments are as follows.

Strengths

1. In order to find the exact reward specification, the authors propose Barfi to dynamically adapts the agent’s policy optimization procedure and address other sub-optimalities introduced by algorithmic design choices. 2. The proposed method allows for the separation of the desired behavior specification from the optimization process with a learned reward function. 3. The authors show how the use of implicit gradients can enhance the efficiency of optimizing the reward function compared to traditional path-wise gradient-based meta optimization methods

Weaknesses

1. Figure 1 explains the effect brought by the proposed method, but does not reflect why the proposed method can change the direction of the gradient and dynamically correct the entire policy optimization process by changing the behavior alignment rewards. It is suggested to visualize the principle of the method in this figure. 2. As for the experiment results, I found that only a few results show that Barfi outperforms other methods, which makes me confused about the effectiveness of the method. Please specify the advantages of the proposed method. 3. The authors compared the proposed method with potential-based reward shaping. Is there any other kind of reward shaping method? It is recommended to compare the proposed method with more relevant work to demonstrate its effectiveness. 4. Some visualized results about the behavior of the agent with the proposed method are required to support the effectiveness. 5. In Figure 9, only two curves in each sub-figure. Please explain where are other curves. 6. This paper provides a detailed theoretical description of the proposed method. However, it is not easy to follow. It would be better to represent the method with a flowchart.

Questions

See the weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See the weaknesses.

Reviewer BidQ8/10 · confidence 4/52023-07-07

Summary

The authors propose a method (Barfi) that makes use of auxiliary rewards (provided by the designer), in addition to the rewards already defined for the environment. The method is able to use auxiliary rewards if they are helpful for the task, and avoid the auxiliary signal if they seem to be misspecified. The paper seems well written, for example, it motivates the need for such an algorithm, explains the objective function well, why it's needed, etc. and also seems to have decent experiments.

Questions

On Line 91 - The work [1] that shows that potential based reward shaping is not suited to Q learning is only for the tabular case, right? Would the results hold for gradient based deep RL algorithms? It would be interesting to see how Barfi works when the given auxiliary reward is sometimes helpful (for a part of the state-action space) and sometimes misspecified (for another part of the state-action space). Assuming a parameterization like on line 126, would it be possible to show what happens with a Gridworld environment when the auxiliary reward is designed in this way? Does this need a parameterization where $\phi_i$ also depend on $s,a$? Suggestions related to language and typos: 1. On Line 157 - "due its" -> "due to its" 2. On Line 127 - $\theta_3$ should be $\phi_3$? References: 1. Potential-based shaping and q-value initialization are equivalent, Wiewiora (2003) EDIT (16 Aug 2023): updated score from 7->8.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Reviewer 8Eto8/10 · confidence 4/52023-07-07

Summary

RL agents often have difficulty learning from sparse rewards, and thus, it is commonplace to introduce an additional auxiliary reward to guide the learning process. However, these heuristic auxiliary rewards can introduce new side effects and biases into learning. In the paper, the authors develop a novel framework for combining the auxiliary reward and true reward in a way that is aligned with the true reward (even given the *learning* process, in contrast to e.g. shaping theorem which says whether the new reward has the same opt policy but does not take into account the potential imperfections in the learning process). In particular, they formulate a bi-level optimization problem in which the outer problem is to learn a combination of the auxiliary and base reward that does well wrt to the true reward given a specific algorithm that solves the inner problem of computing a poilicy given the combined reward.

Strengths

This paper is original, a high-quality, principled solution to a problem with high practical significance. The paper is also very well-written and clear. Every choice is justified and the authors do a good job of contextualizing their work (e.g. in contrast with potential based reward-shaping). The experiments are meaningful, well-organized and conveyed. I appreciated that the experiments used a range of realistic auxiliary reward functions including heuristic-based reward functions, reward functions that encode a guess of an optimal policy, ones that change the optimal policy, ones that depend on only states, and ones that depend on both states and actions. The authors report results on robustness, scalability, to high dimensions, and even compute/memory comparisons. In addition, there are several ablation studies in the appendix.

Weaknesses

There is probably more related work to cite in terms of learning from auxiliary rewards, e.g. Inverse Reward Design https://arxiv.org/abs/1711.02827

Questions

n/a

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors address some limitations and extensions in their conclusion.

Area Chair tDnC2023-08-10

Author-Reviewer Discussion phase (Aug 10-16)

Today begins the Author-Reviewer Discussion phase, which lasts 1 week (**Aug 10-16**). I ask the reviewers to please **carefully read all other reviews and the author responses and (if appropriate) respond to author responses promptly.** If you've read the author response, please take the time to leave a comment, even if you have nothing to add. I also encourage both authors and reviewers to monitor OpenReview for further comments in order to enable as much back-and-forth as possible during this short period.

Reviewer GcHf2023-08-16

Thanks for the responses

Main of my concerns are addressed. I am happy to increase the rating to 6 (weak accept).

Reviewer BidQ2023-08-16

Rebuttal response

I am satisfied with the authors' comments and the experiments. I will increase my score to 8.

Program Chairsdecision2023-09-21

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC