State-wise Constrained Policy Optimization

Summary

This paper discusses an important topic about safe reinforcement learning, which explores the state-wise issue. It is a significant problem because, in the real world, state-wise constraints are one of the most common and challenging constraints in safety-critical applications. Most safe RL methods focus on cumulative safety, which may need to be improved to ensure safety during deploying RL in real-robot applications.

Strengths

1. The writing quality is good. 2. Theoretical results sound good to me. 3. The code is provided in this study.

Weaknesses

1. The experimental results are very weird, especially for the reward performance. For example, in Figure 1, all the baselines’ reward values are almost the same; please check the environments' reward settings. This is also shown in Figure 4. 2. The method cannot ensure safety while deploying the method in real-world applications; demos that the authors provided also confirm this point, that is, sometimes the agent will violate safety constraints and crash into obstacles.

Questions

1. How does the study consider high-dimensional optimization? Generally, RL algorithms can handle 8- or 10-dimensional tasks on MuJoCo tasks, e.g., CPO, PPO Lagrangian, and CRPO. 2. How does the study define the reward and cost settings? 3. Why does this study categorize CSC and safety Layer methods as hierarchical policy? Does this study mean these methods are HRL?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

1. It is hard for me to find some new insights from this study; the problem and the solution are not new. As for the problem, it is already defined in many papers, e.g., CPO, PCPO, and CRPO; as for the solution, it just revises the cost setting from a trajectory, and does not ensure safety at each step. 2. As for the theoretical analysis, most of the findings could be known from CPO by revising the cost optimization settings.

Summary

This work tackles the problem of state-wise safety in the reinforcement learning problem. To this end it introduces the framework of Maximum Markov Decision Process and an algorithm State-wise Constrained Policy Optimization (SCPO) to solve the problem. Numerical results illustrate the performance of the authors algorithm.

Strengths

The paper is well written and supported by extensive numerical results.

Weaknesses

Novelty is slightly limited. Similar state augmentation mechanisms have been studied before. The algorithm developed here arises from a somewhat straightforward application of trust region methods to such state augmentation mechanisms.

Questions

Additional details on the motivation behind the Maximum Markov Decision Process formulation would be beneficial. At first read it is slightly confusing as to why state-wise constraints are introduced under this formulation. The framework of Maximum Markov Decision Process introduces a state augmentation. These mechanisms have been used in the past [Sootla, Calvo-Fullana]. Specially as the augmentation introduced by the authors is similar to augmenting the state space with the dual variables. While the algorithm introduced by the authors here is clearly different, a comparison with these augmentation methods should be done. A. Sootla, A. I. Cowen-Rivers, J. Wang, and H. B. Ammar. Enhancing safe exploration using safety state augmentation. arXiv preprint arXiv:2206.02675, 2022 M. Calvo-Fullana, S. Paternain, L. F. O. Chamon, and A. Ribeiro. State augmented constrained reinforcement learning: Overcoming the limitations of learning with rewards. arXiv preprint arXiv:2102.11941, 2021.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Addressed.

Summary

This paper introduces a novel approach to solve safe RL tasks. This approach is based on a new method SCPO and corresponding MMDP framework. Authors show the efficacy of this method by providing mathematical guarantees. Additionally, authors give useful practical implementation tips to improve reproducibility of their work.

Strengths

originality - This paper extends CMDP framework to account for state-wise safety guarantees. It's a fairly natural extension, and authors did a great job of explaining the shortcomings of existing methods and how theirs addresses the gaps. quality - The paper is well-written and includes thorough mathematical analysis and practical tips for implementation. - Experimental section includes fair comparison with existing SOTA methods. clarity - The paper uses consistent notation with previous papers (i.e. TRPO). - Including pseudocode (in appendix) is useful. significance. - The paper has a good significant impact on safe RL. The presented method is novel and beats existing methods on the selected benchmarks. Authors also provided concrete future direction for this work.

Weaknesses

- Not necessarily a weakness, but including intuitive explanation of proposition 1 and 2 would be helpful to increase overall readability of the paper. - Paper heavily relied on concepts TRPO, but didn't give explanations about them in the background section.

Questions

N/A

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Yes

Summary

This paper introduces a novel framework called Maximum MDP to address the problem of state-wise constrained policy optimization, namely the authors considers limiting the expected maximum state-wise cost rather than the cost for each state. Similar to the TRPO/CPO framework, the authors derived a worse-case constraint violation guarantee and practical algorithm which were shown in be effective in a set of robotic locomotive tasks.

Strengths

- I really like the simplicity of the idea proposed by the authors, state-wise safe RL is a much more common requirement for real-world safety critical problems but most literature on safe RL focus on constraining the cumulative cost. Reformulating the problem as constraining expected maximum state-wise cost eliminates the scalability issues associated with state-wise safe RL and effectively transforms the problem into a cumulative cost problem. - Furthermore, this formulation allows the authors to easily adapt existing algorithms for cumulative constraints (CPO) with little modifications. - Experiment section is well-structured and answers key practical questions associated with the proposed algorithms.

Weaknesses

- Theorem 1 doesn't really depend on the specific formulation of MMDP and is really a finite-horizon variant of the policy improvement theorem from TRPO/CPO (which I think is a nice contribution in itself), I think the authors should try to convey this - There is a major error in the proof in the appendix, note that equation 20 is not correct since $I - P$ is not invertible

Questions

- One related reference which seems to be missing is [1] which similarly uses a trust region based method but for cumulative costs, one question I have here is can SCPO be applied to other trust-region based algorithms such as PCPO or FOCOPS? - Overall, I do like the main idea proposed by the authors and I think the general approach is sound and the empirical improvements are valid. However the issue in the proof essentially invalidates Theorem 1. While I am fairly sure a finite horizon variant of the policy improvement theorem from CPO/TRPO does exist, the authors need to make sure the results are correct. At present, I cannot recommend this paper for acceptance at this venue, however I am willing to revise my score if the authors do fix the mathematical issues during the rebuttal period. - [Minor] Achiam et al.'s Constrained Policy Optimization seem to appear twice in your list of references [1] Zhang et al. (2020) First Order Constrained Optimization in Policy Space

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Response to author rebuttal

I want to thank the authors for you detailed rebuttal. I have looked through the authors rebuttal as well as reviews from other reviewers. I believe the proof for Theorem 1 is now correct, though just one quick comment is that intuitively I think a simpler proof may exist by simply taking advantage of the equivalence between infinite horizon discounted and finite horizon undiscounted MDPs (see Chapter 5 of Puterman), but there could be complications I have not considered. Since my main concern with the original paper was the mathematical proof, I am satisfied with the authors' response. I agree with the authors that state-wise constrained RL problems are a class that deserves separate treatment and the authors' simple approach to tacking this challenging problems brings value to the community. Therefore I am revising my score to recommend for acceptance.