Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms

Summary

This paper studies the safe RL problem with a generalized stepwise safety chance (probability) constraint, essential for many safety-critical systems with RL. The authors propose a meta-algorithm to solve this problem (MASE), by combining unconstrained RL with an uncertainty quantifier to guarantee safety with probability. They study two variants for MASE, one for the linear model and the other for GP. Experiments on grid-world and safety gym show that MASE with GSE formulation achieves SOTA compared to the baselines.

Strengths

The paper is well-written and easy to follow. The proposed GSE problem is more general than the CMDP formulation with an additive expectation constraint. Especially, the GSE problem is an important problem to study for many real-world safety-critical systems, such as autonomous cars and cars. The proposed method looks sound and correct to me. The MASE algorithm essentially builds on Assumption 3.4 (uncertainty quantifier) which could also be a potential bottleneck or limitation, but the author did a good job by proposing a general linear model and GP for it. The experimental results show better results in terms of safety violations, compared to other CMDP-based approaches.

Weaknesses

1. The author may want to at least discuss this ICML paper in their related work. The hard safety chance constraint is highly relevant to the GSE problem in this paper, although it is indeed using different approaches to solve the problem. Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., ... & Zhu, Q. (2022). Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments. arXiv preprint arXiv:2209.15090. 2. The authors should discuss the limitations of this work 3. In Method, MASE needs to compute a safe action set, how complex this computation is? In order to compute it, what kind of assumptions of the underlying environments do you acquire?

Questions

1. Page 3 Lines 96 to 98, although I can understand what the authors mean, we should always keep in mind that s_h (a_h) are random variables, you cannot let a random variable be less than a value. It has to be in Pr() format, even if the probability is 1.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I would like to see the authors' opinions on the limitations in their responses.

Summary

The authors present a novel safe RL algorithm for safety constraints with probability one. First, the authors present a problem formulation that can be used to derive different common safe RL formulation (state constraints, accumulated constraints etc). Then the authors leverage a sophisticated technique to reshape the reward accounting for safety information. In particular, they predict if the following actions will be safe by learning the action safety state similar to safety shield techniques. Their shields are learning using Gaussian processes and offer safety predictions with uncertainty estimates.`

Strengths

1. The algorithm is technically sound, novel and quite interesting as an idea. 1. The authors present an extensive analysis for the case of generalized linear CMPDs. The theoretical results seem to be correct, but I have a few minor concerns regarding the statements 2. The numerical results are impressive but shown only for one environment.

Weaknesses

1. The statement of Theorem 3.1 reads there are instances of the GSE problem that are not equivalent and cannot be transformed into Problem 1, 2 or 3. This is not shown. 1. The fact that Problem 2 can be transformed into the GSE problem does not mean the GSE problem is more general than Problem 2. If the GSE problem can be transformed to Problem 2 as well, then they are equivalent. Furthermore, if the accumulated cost is used as a state as in Lemma A.1, then the problems cannot truly be equivalent. I think the authors should rephrase these results and make more accurate claims. 1. The method does remind me of safety layer techniques [Dalal 18] and [28] with a more sophisticated reward-shaping approach. Can the authors provide a short discussion on the relation to these papers? 1. The numerical results can be improved * I recommend adding boxplots to the simulation results to see the distributions of the traces. For instance the boxplots of the trajectories for the final epoch as in [28] * I recommend providing more epochs in the experiments. I suspect that the algorithms generally achieve similar performance, but the algorithms with probability one constraint simply converge slower. * [Dalal 18] Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., & Tassa, Y. (2018). Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757. * [Yu 22] Yu, Haonan, Wei Xu, and Haichao Zhang. "Towards safe reinforcement learning with a safety editor policy." Advances in Neural Information Processing Systems 35 (2022): 2608-2621.

Questions

1. Please provide details on the safety gym environment. This does not seem like a standard one. 1. How the algorithm scales with a number of epochs/states? My concern is that the GP is not the most scalable model for RL

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

There are two limitations of the approach: 1) the scalability of the algorithm and 2) the restrictiveness of the problem definition. While the authors discuss the latter limitation and show that this problem formulation is important, I didn’t find a discussion on scalability.

Summary

Paper considers the generalized safe exploration problem (problem 4) and compares it to the other safe exploration problems in the literature, leading to Thm 3.1 that concludes that it is more general than the others. The authors then introduce MASE, a meta-algorithm for safe exploration, that attempts to solve the GSE problem with the ability to execute an emergency stop "beforehand" as opposed to others that have done it afterwards. Section 6 then presents a more practical algorithm using GP models, which is then used in section 7 to compare the new approach with several baselines.

Strengths

The paper is well presented, with several technical results about the safety analysis, culminating in Thms 5.6 and 5.7 which show that GLM-MASE guarantees safety with high probability for every time step. Several numerical experiments are presented to compare with unconstrained and SOA constrained RL algorithms.

Weaknesses

The development in the paper assumes access to an emergency stop action that enables the agent to avoid violating a safety constraint. This assumptions seems difficult to achieve in practice as for many agents of interest, stopping is not a safe state. Further, the action would typically be state dependent, requiring at a minimum a emergency action policy (rather than action). Finding either action or policy seems non trivial and could possibly be overly simplifying most problems of interest. Given the comment about [33] at the bottom of page 4, I would have expected to see a comparison of these approaches to imposing an action to avoid an unsafe state before/after a training epoch in the numerical results. If that is included in section 7, suggest highlight that discussion point more. The footnote on page 3 makes reference to this being a conservative approximation of that in safe RL problems with chance constraints, and consigns the discussion Appendix B. It would seem like more clarification than that is needed here. Also, the more recent work by Pavone in this area will be of interest: * Lucas Janson, Edward Schmerling, and Marco Pavone, “Monte Carlo motion planning for robot trajectory optimization under uncertainty.” In Robotics Research, pages 343–361. Springer, 2018. * Anirudha Majumdar and Marco Pavone “How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics,” https://doi.org/10.48550/arXiv.1710.11040 Fig 2 shows that MASE satisfies the constraints, but if I understand the plot correctly, this is achieved with very conservative margins (constraint at 20, values are typical ≤ 5). This is similar to Saute TRPO, but perhaps suggest why the performance (episode return) is so weak compared to the unconstrained solutions (that don’t violate the constraints by much). Is there a way to trade off this conservatism to achieve better performance? Very hard to see the frequency with which TRPO Lagrangian violates the constraints given the lines/colors on fig 2.

Questions

Does the comment about safe RL problems with chance constraints suggest that there is a better answer available already? The text says that problem 5 in App B is hard to solve, but presumably the authors of [20] and [22] did so? Can their results be compared to the ones here?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See little discussion of this point in the paper.

Summary

This paper addresses the problem of safe reinforcement learning, in particular safe exploration. In a nutshell, the authors provide an algorithm that is supposed to go beyond standard 'safety' measures in safe RL, such as the constrained MDP setting. There, an agent is bound to satisfy an additional (expected) cost constraint. Here, the authors postulate that an agent must satisfy a constraint almost surely or with high probability. As a key feature of their approach, the authors assume that a so-called 'emergency stop action' is available that allows the agent to always have a fallback. The authors provide a general framework in the form of an algorithm, prove its theoretical guarantees, and evaluate the method on a set of standard benchmarks.

Strengths

The authors tackle a very important problem, safe exploration in RL. Moreover, they identify correctly that the standard constrained RL (or constrained MDP) setting is generally insufficient to ensure safety during exploration. In principle, the constrained RL setting provides just an incentive to act safely during training, and even after training, safety depends on an expectation to satisfy a safety constraint. Generally, the paper is well-written and easy to follow.

Weaknesses

I like the paper in general, but in its current state cannot be accepted, in my opinion. The reason is a severe lack of related work. Most importantly, the authors seem unaware of a flavor of safe RL that is often referred to as 'shielding.' In these settings, a so-called shield 'blocks' unsafe actions according to some pre-defined safety measure. [1] was the first paper to introduce this aspect, [2] introduced shields for almost-sure properties in partially observable environments, [3] provides shields that satisfy a property with a certain probability, [4] provides a shielding mechanism for multi-agent settings. There are many more relevant works. The general shielding framework depends on varying assumptions, but the general procedure looks like the MASE algorithm. Note that safety during exploration/training is the most important motivation for shielding. I encourage the authors to thoroughly compare these (and more) works to explain the contribution and novelty better. [1] Alshiekh et al.: Safe Reinforcement Learning via Shielding. AAAI 2018 [2] Carr et al.: Safe Reinforcement Learning via Shielding for POMDPs. AAAI 2023 [3] Jansen et al.: Safe Reinforcement Learning Using Probabilistic Shields. CONCUR 2020 [4] Melcer et al.: Shield Decentralization for Safe Multi-Agent Reinforcement Learning. NeurIPS 2022 Moreover, I am not convinced by the experimental evaluation. The emergency action seems central to the approach, but I fail to see how it has been integrated into the standard benchmarks. Then, how often is it called during training by the agent? How does it impede an agent's exploration rate? minor comments l92: In the definition of a policy, it seems to be non-stochastic. In such a multi-objective setting, it might be beneficial to use stochastic policies. Have you considered this? l 125/126: in a probabilistic setting, there might be states with a high probability of violating a safety constraint but do not already violate it. It could be interesting to consider such information in the value function.

Questions

1. Compare your approach to the shielding framework. 2. How is the emergency action affecting the agent, please discuss in line with the experiments. 3. How is the 'blocking' of actions realized for an RL agent on a technical level?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

1 poor

Limitations

The limitations are not properly addressed, see my earlier comments on the evaluation.

Thanks

I thank the authors for their careful reply. With the new experimental results and the proper placement in the literature, I will increase my score.

Final remarks

I thank the authors for the responses and the new experiments. I have two remarks: **Terminology issue in Theorem 3.1 [Weakness 1 and Weakness 2].** I agree with the formulation that Problems 1,2, and 3 can be transformed to GSE for Theorem 3.1. I also suggest clarifying that GSE can be used to solve other problems, but there is no (to our best knowledge) direct proof that solving other problems would not solve GSE. Unless there is proof and that could be another interesting addition. **Comparison with [Ref-3], [Ref-4], and [28] [Weakness 3]** Just a minor remark that I think was alluded to in other responses. I believe your algorithm for probability one constraint can be seen as a nice generalization of [28] with (again) a nice form of shielding. This is naturally a matter of opinion, but I found this could be an interesting connection. If some connections can be made perhaps the theory of [28] can be extended to MASE. I also suggest adding boxplots to the final version of the paper not to the appendix. I feel this would be a fair comparison to [28] showing the superior performance of MASE.

Responses to further comments.

We appreciate your valuable suggestions and thank you for increasing the score! We will ensure that all reviewers' valuable feedback is reflected in the camera-ready paper.