Summary
This paper focuses on how to better combine primary rewards that are typically sparse with auxiliary rewards designed by human heuristics to maximize the primary rewards.
The authors propose a bi-level optimization procedure, where the upper level optimizes a behavior alignment reward to guide agent to maximize the primary rewards and meanwhile minimizes the discount value to force the learned rewards to encode long-horizon information, and the lower level can be any policy searching algorithm. They provide both intuitive and theoretical analysis to show that the formulated procedure is not only robust to misspecified auxiliary rewards, but also overcomes the imperfection of policy optimization algorithms, such as biased policy gradients caused by omitted discounted term and no importance sampling in off-policy learning.
Besides the theoretical analysis, the authors also provide two methods to empirically solve this untractable bi-level problem since the optimization of upper level problem requires characterizing the impact of learned rewards and discount value on the entire optimization process. One is a gradient based method that makes use of the implicit gradient denoted as Implicit Bi-level optimization and the other takes into account the complete path followed by the inner optimization loop denoted as Path-wise Bi-level optimization.
The authors also conduct experiments on some case studies to show the robustness and extent to Mujoco tasks to show the scalability to high-dimensional continuous control.
Strengths
1. This paper presents a good motivation to motivate why this work is of great significance and why previous works like potential based methods fail (higher variance and only state-based potential function).
2. This paper dives deep to analyze the advantages of the proposed bi-level objective (Section 3), which is helpful for readers to catch the main idea of this paper.
3. The analysis in Section 3.1 seems novel to me. The proposed bi-level objective can not only address auxiliary reward imperfection but also the imperfection of policy optimization algorithms, which is novel and valuable.
Weaknesses
1. The resulting practical algorithms seem too complex to conveniently apply for higher-dimensional tasks like vision-based tasks.
2. The proposed method introduce a lot of additional hyperparameters as shown in Table 2.3.4.5, which may be hard to tune. For instance, in Table 3, $\alpha_\theta$ is tuned to 0.015625, which does not seem like a common value and may be carefully tuned.
3. The experimental results are not so convincing. In figure 3, Barfi only wins 1 out of 4 tasks. Barfi also cannot outperforms baselines in figure 2. 5.
4. The authors claim that Barfi requires a warm-up stage, after that, Barfi can fastly adapt to find good policies. However, the reasons why Barfi requires warm-up and how does Barfi performs without such warm-up remains unclear.
5. The empirical analysis part is hard to follow, it would be better if the authors could separate some large paragraphs and simplify the abundant detailed task descriptions but focuses on more in-depth analysis.
There is also a recent work that focuses on correcting the imperfect rewards using a bi-level objective in the offline RL setting [1]
[1] Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., Jia, Q. S., & Zhang, Y. Q. Mind the gap: Offline policy optimization for imperfect rewards. ICLR2023.
Questions
Please refer to weaknesses for details.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Please refer to weaknesses for details.