Inverse Reinforcement Learning with the Average Reward Criterion

Summary

This paper addresses Inverse Reinforcement Learning (IRL) in the average-reward setting. The authors propose a Stochastic Policy Mirror Descent (SPMD) method to solve the Average Reward Markov Decision Process subproblem and use SPMD to propose the Inverse Policy Mirror Descent method to solve the IRL problem. The authors provide complexity results for these methods and experimentally validate them using the MuJoCo benchmark.

Strengths

- The authors develop a novel method and show its merits both theoretically and experimentally compared to existing methods. - The theoretical analysis is very thorough and the authors clearly outline the assumptions that are made. - The proposed IPMD method achieves good results on a variety of environments.

Weaknesses

- The paper is a bit light on experiment details (e.g. hyperparameters). More detail here would be appreciated (e.g. see Questions). - It’d be nice if the authors could provide an intuitive explanation for the result of Theorem 3.4. Equation 25 is a bit difficult to understand. - The authors make a fair number of assumptions in their theoretical analysis. It'd be nice if the authors added some discussion on which assumptions hold in practice and/or how they enforce those assumptions, as well as how restrictive these assumptions are.

Questions

- Does Assumption 3.2 hold in practice? Do the authors do anything to enforce these constraints? - What is $\omega$ in Equation (3) and Algorithm 1 chosen to be in practice? - Can the authors provide architecture details for the policy, Q network, and learned reward function in the appendix? - Equation 24 in Theorem 3.4 makes an assumption about step size. Is this assumption implemented in practice (i.e. in Algorithm 1 what is $\eta_k$ chosen to be? - It seems the authors assume access to a large number of expert trajectories (Appendix A.9 says expert demonstrations are collected for five million steps). I’m curious how IPMD would perform with a more limited number of expert trajectories (for instance, the f-IRL[1] paper has experiments where they just use 1 expert demonstration). Have the authors considered how number of expert trajectories affects performance of IPMD? [1] f-IRL: Inverse Reinforcement Learning via State Marginal Matching (Ni et al.)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

In this paper inverse reinforcement learning is studied when the teacher was using an average-reward criterion rather than discounted rewards with known discount factor. The paper proposes a stochastic first-order method starting from stochastic policy mirror descent for MDPs and continuing towards inverse policy mirror descent for solving the IRL problem. The paper contains also some numerical experiments based on MuJoCo benchmarks.

Strengths

The paper is very clear and elegant in its use of average reward method in RL and IRL. There are many papers on duality of this type although not yet specifically on the average reward criterion, although in NAC papers similar considerations were made. The supplementary material is strictly supplementary and very clear and comprehensive. It's great the hints for practical use are included.

Weaknesses

“To the best of our knowledge” is not a useful sentence in an abstract, in particular in an anonymous manuscript. It is more useful to discuss such opinions later in the paper in the context of related results (see e.g. previous NeurIPS conferences). Duality has been used in many similar contexts, but it could still be said more clearly why it is a good idea here. It is acceptable, if you want to avoid discussing natural AC, but then there is more to explain. Experiments are relatively few with just final results given, so no understanding why the proposed method is useful can be gained. The is no discussion of the experimental results, therefore, although one may guess, it is not clear why the proposed method struggles on Ant or whether there is any relation of the performances comparing tables 1 and 2. There is no attempt included to check whether the complexity results are tight or whether global optima are indeed found. Check: “Acrot” Assumptions 4.2 and 4.3 are formally stated in the main text, but do not seem to be used. In the appendix only 4.2 is mentioned. Ref. 28 is not mentioned in the text, perhaps a separate bibliography for the appendix would make sense. I find it a bit difficult to distinguish between all the Qs. The distinction between math-Q and cal-Q is probably necessary, For example, tilde-cal-Q (32) could as well be substituted by its definition. It occurs again only in (34) and (38), where at least a ref to the definition needs to be included. In “differential Q-function” (in italics in the main text vs. theorem environment) different fonts are used, which should be avoid this, ideally by using the math-Q font that seems to occur also in the theorem environment.

Questions

Can you explain more about the improvements (as in Humanoid RL) and deficiencies (as in Ant IRL or walker RL)? What feature of the algorithm contributes to the success or suboptimality of the performance in each problem? Are the assumption not strictly satisfied, or is the sampling the reason for any suboptimality? Which theorems have appeared in earlier work or have strong similarities to theorems in earlier work? I would assume that 1/(1-gamma) not much smaller than K, and the difference between them is related to the number of states and actions. In other words, is there a practical need for the O-symbols or is the complexity already implied by the known factor proportional to 1/(1-gamma)? Why does the performance show variablity even for simple problems (see Fig. 1)?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

The authors state in the supplementary material: "We anticipate no potential negative societal impacts concerning this research." which could seem questionable in the context of IRL, because IRL has the potential to uncover hidden motives in legal human behavior without the consent of a person. The research seems fine, but some discussion, beyond the mere statement would be useful.

Summary

This paper proposes an inverse reinforcement learning (IRL) algorithm for infinite horizon average reward Markov decision processes (AMDPs). At first, the authors show the stochastic policy mirror descent (SPMD) algorithm that achieves $\mathcal{O}(\varepsilon^{-1})$ rate of convergence. Then, the authors propose the inverse policy mirror descent (IPMD) algorithm that achieves $\mathcal{O}(\varepsilon^{-2})$ rate of convergence. The SPMD algorithm is compared with SAC on the MuJoCo benchmarks, and the SPMD achieves on-par performance with SAC. The IPMD algorithm is compared with f-IRL and IQL, and the experimental results show that IPMD is slightly better than f-IRL. Interestingly, the proposed algorithms perform much better than the baselines for the Humanoid environment that has many degrees of freedom.

Strengths

- Originality: RL and IRL algorithms for AMDP with strong theoretical basis are novel, although Dvijotham and Todorov proposed IRL under the linearly solvable Markov decision processes (LMDP). - Quality: The paper provides some strong theoretical analysis on AMDP. - Clarity: The authors show detailed steps to prove the theorems in Appendix. - Significance: The standard IRL algorithms deal with infinite horizon discounted reward problems under the assumption that the discount factor is known in advance. It is problematic because the expert's discount factor is usually unknown. The proposed method is promising because it can avoid tuning the discount factor.

Weaknesses

- Some recent studies regarding AMDPs are not mentioned in the manuscript.

Questions

Major comments - Is the SPDM algorithm for AMDPs (Algorithm 1) an on-policy method? The objective function for training $\bar{\mathcal{Q}}^{\pi, \zeta}$ (35) suggests an on-policy algorithm, but I did not find the explanation. In addition, (35) does not use a target network. Is it a benefit of the property of the on-policy algorithm? - I do not fully understand why $\nabla \omega (\pi)$ is usually unavailable, as described in Lines 143-144, because $\omega$ is selected by ourselves. Furthermore, it is unclear why its estimator is parameterized by $\phi$ because $\pi$ is parameterized by $\xi$ (33). - Please show $g(\theta; \zeta)$ in detail because it is used to update $\theta$. For example, is $g(\theta_k; \zeta^\pi_k)$ the estimator of $\mathbb{E}_{(s, a)~d^\pi} [\nabla_\theta c(s, a; \theta)]$? - The experimental results are promising, but one potential reason for successful results is that the reward function of the MuJoCo benchmark is well-shaped. For example, the reward of Walker2D is given by healthy_reward + forward_reward - control_cost: https://gymnasium.farama.org/environments/mujoco/walker2d/ . It may imply that the average reward formulation is more appropriate than the discount reward one in the MuJoCo benchmark. My interest is the performance of the proposed method when it is evaluated in sparse reward settings such as navigation tasks. Minor comments - The following paper proposes a framework based on stochastic mirror descent for AMDP. Please discuss the relationship between these papers and the proposed method. - Y. Jin and A. Sidford. (2020). Efficiently Solving MDPs with Stochastic Mirror Descent. In Proc. of ICML. G. - Neu et al. (2017). A unified view of entropy-regularized Markov decision processes. http://arxiv.org/abs/1705.07798 . - C.-Y. Wei, et al. (2020). Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes. In Proc. of ICML. - As mentioned in Strength, the following paper proposes a method to estimate the reward function under LMDP: K. Dvijotham and E. Torodov. (2010). Inverse Optimal Control with Linearly Solvable MDPs. In Proc. of ICML. It would be better to discuss their paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

4 excellent

Limitations

Minor comment: The authors briefly mention issues on potential societal impacts in A.10.

Summary

This paper aims to address the problem of inverse reinforcement learning (IRL) under the maximum entropy framework (MaxEnt-IRL) and an average reward criterion. The MaxEnt-IRL problem is formulated as a combination of an average reward Markov decision process (AMDP) and a dual IRL problem. The proposed algorithms, namely the SPMD algorithm and the IPMD algorithm, are utilized to solve the AMDP and dual IRL problem, respectively. The SPMD algorithm achieves a gradient computation step complexity of $\mathcal{O}(1/\epsilon)$ under general state and action spaces, while the IPMD algorithm has a complexity of $\mathcal{O}(1/\epsilon^2)$. The empirical studies conducted on the MuJoCo benchmark and various control tasks confirm the theoretical findings.

Strengths

1. The idea of formulating the IRL problem as a dual IRL problem, incorporating an AMDP as a subproblem, is interesting. 2. The paper extends a previous RL algorithm [1] to general state and action spaces with a general function approximation class for AMDPs. The paper is clear and easy to follow. 3. The proposed algorithms are well-founded and supported by theoretical analyses of average reward Markov decision processes. The convergence analysis of the algorithms is provided under certain assumptions on the approximation function classes. Overall, the paper is well-written and easy to follow. The proposed algorithms are well-motivated and supported by theoretical foundations. However, certain assumptions regarding function approximation classes and the critic step may be too restrictive, potentially resulting in the failure of convergence in scenarios involving deep neural networks. Despite these concerns, the theoretical contributions of the paper could be of interest to the community, and I recommend the acceptance of the paper. [1] Tianjiao Li, Feiyang Wu, and Guanghui Lan. Stochastic first-order methods for average-reward markov decision processes.arXiv preprint arXiv: 2205.05800, 2022.

Weaknesses

1/ The advantages of employing the average reward criterion over discounted IRL are not adequately explained in the paper. 2/ Assumption 3.2 assumes weak convexity and Lipschitz continuity for the approximated Q-functions. These assumptions may not hold in practice, particularly when neural networks are used for function approximation. As the global convergence result of SPMD relies on the convexity of the approximated Q-function and the regularizer h, it may not hold in practical scenarios. 3/ Assumption 3.6 imposes a restriction that the errors of the critic step, typically represented by neural networks, are bounded. This assumption may be overly restrictive, as estimations can be highly inaccurate in many cases. The convergence result for general function approximation classes may also fail to hold in practice. 4/ When the reward function could be parameterized by neural networks, which is normally the case in practice, Assumption 4.1 would be too strong. 5/ In the reward recovery experiment, it would be beneficial to compare the proposed algorithms with discounted IRL algorithms since the expert is trained in a discounted environment (the author trained the expert agent using SAC with a discount factor of 0.99). 6/ Including the running time of the proposed algorithms and the baselines would provide valuable insights.

Questions

Seen in the weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Seen in the weaknesses section.

Thank you for the clarification!

Thank you for the clarification!