e-COP : Episodic Constrained Optimization of Policies

Summary

In this paper, the authors propose a policy optimization algorithm for constrained Reinforcement Learning (RL) problem in a finite horizon setting. First, the authors develop a policy difference lemma for the finite horizon MDP problem. Following that, the authors combine a set of ideas to propose the e-COP algorithm. The authors show that the proposed algorithm is numerically much stable compared to the state-of-the-art solutions. Optimality of the proposed scheme is established under certain assumptions. Extensive simulations have demonstrated that the performance of the algorithm is better than existing methods.

Strengths

The paper is well-written and easy to follow. The results and claims presented in the paper appears to be correct.

Weaknesses

The idea of using a quadratic penalty term to improve the stability near the edge of the constraint set boundary is a purely heuristic based approach. The idea of approximating $\rho_{\pi}$ by the empirical distribution from the policy of the previous episode does not appear to be a good approximation. The idea of allowing the agent to act in a constrained manner even before the constraint is violated may lead to suppression in the original objective function.

Questions

1. The idea of using a quadratic penalty term to improve the stability near the edge of the constraint set boundary is a purely heuristic based approach. Is there any strong theoretical argument supporting such a choice? 2. In the related work section, the authors have discussed about drawbacks of using penalty terms in existing methods. Are these drawbacks applicable to the scheme proposed in the paper? 3. It is not clear how the time-dependent state occupation distribution is computed. The idea of approximating $\rho_{\pi}$ by the empirical distribution from the policy of the previous episode does not appear to be a good approximation. Although such approach may work well in infinite horizon problems where the optimality of stationary policy holds, it may not work well in the finite horizon setting. 4. The approach of using the second order penalty is not very novel and is widely used in practice for practical RL algorithms. What is the novelty of the proposed approach? 5. The idea of allowing the agent to act in a constrained manner even before the constraint is violated may lead to suppression in the original objective function. In other words, when the constraint is not too tight, the proposed scheme may unnecessarily impose additional level of constraint on the system. 6. How to ensure that we have sufficiently large $\beta$ and $\lambda_{t,i}$ which are well above the given lower bounds? The adaptive parameter selection process is just a heuristic. 7. The symbol in the last term of equation (7) is not clearly defined. 8. The algorithm proposed in the paper is an amalgamation of many ideas. Many of these ideas involve approximation and many are just driven from intuition without any formal guarantee of optimality. It is hard to see how this fusion work together in theory (e.g., satisfying the conditions given in Theorem 3.3).

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

The paper proposes a new algorithm for finite-horizon constrained RL problems. The algorithm is based on three ideas: PPO-like updates for the finite-horizon setting, P3O-like treatment of constraints with adaptive penalty coefficient, and quadratic damping penalty. The proposed algorithm outperformed previous algorithms in some benchmark tasks.

Strengths

A strong empirical performance of the proposed algorithm. The introduction of quadratic damping penalty to safe RL seems to be a new idea. The paper is easy to follow except some typos.

Weaknesses

Performance difference lemma for the finite-horizon setting is already known. See Kakade's thesis. Since PPO-like update is based on the performance difference lemma and PPO's idea, I do not see it is novel. P3O-like treatment of constraints with adaptive penalty coefficient is also not novel. There are some parts I do not really understand. In particular I wonder if the proposed algorithm really does what Algorithm 2 states. Concretely Algorithm 2 suggests to save policy parameters for each time step, which sounds to be unusual. (Also I guess Line 6 of the algorithm has a type: $\pi_{k, t} \gets \pi_{k, t+1}$ should be $\pi_{k, t} \gets \pi_{k-1, t}$.) If the algorithm is implemented as above, it is unfair to compare to previous algorithms, because previous algorithms use a much fewer number of parameters. If the algorithm is not implemented as above, there is a serious issue in presentation. Eq 8 seems to contain many typos. For example $\min_{\pi_{k, t}} L(\theta, \lambda, \beta)$ does not make sense since the loss do not have $\pi_{k, t}$. (Sorry for the sloppy notation, but I believe the authors can understand what I mean.) Also $\pi_{k, t}^\star = \min_{\pi_{k, t}} L(\theta, \lambda, \beta)$ does not make sense since $\min_{\pi_{k, t}} L(\theta, \lambda, \beta)$ is a scalar.

Questions

- Does the proposed algorithm what Algorithm 2 states? Algorithm 2 suggests to save policy parameters for each time step, which sounds to be unusual. - I looked into the code, but it seems it will not run since `_episodic_reward_loss` and `_get_surr_cadv` methods are undefined. I checked if the Github repo of omnisafe, but I cannot find those functions. Is the code intended only for reference? (Indeed the code is provided as a text file, not python file.) - I currently think the main contribution of the paper is the introduction of quadratic penalty dumping to safe RL. Therefore I would like to see results of ablation studies to prove the importance of quadratic penalty dumping. Is there any result? (Or am I missing?)

Rating

4

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

If the proposed algorithm really does what Algorithm 2 states, there is an issue of huge memory requirement as the horizon gets larger. I think it should be discussed. Otherwise I do not see any potential negative societal impact.

Summary

The authors introduce a policy optimization algorithm for episodic constrained RL problems, e-COP. In general, Lagrangian formulation is used for constrained optimization problems, however, the constraints are not always satisfied in real world applications. The solution approach avoids Hessian matrix inversion. The algorithm approach uses deep learning-based function approximation, a KL divergence-based proximal trust region and gradient clipping popular in improving the generalization and avoiding vanishing gradients in proximal policy optimization algorithms. The authors use SafeGym to benchmark their algorithm against the state-of-the-art algorithms such as PPO, FOCOPS, CPO, PCPO, and P3O.

Strengths

The e-COP algorithm in most cases outperforms all other baseline algorithms but fails on Humanoid, AndReach, and Grid where it provides the second best optimal results. The authors used an extensive set of baselines -- few using Lagrangian approximations. The idea is original. In general, the paper is well-written and clear. There are few grammatical errors and typos.

Weaknesses

The authors do not talk about the limitations of the algorithms. There is a mention on the complexity of the Grid environment resulting the e-COP not to perform as best. It will be good for the completion of the paper to have a short section on limitations.

Questions

I would like to know whether any hyper-parameter tuning has been done for the baseline algorithms? It is possible that the Lagrangian formulation may perform better after tuning.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do not adequately address limitations.