Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning

Summary

Naive model predictive shielding may overly restrict exploration thereby preventing an RL agent from learning a policy with good performance. In order to prevent this, the authors propose a method to optimise a backup policy that is provably safe using an online planner. An approximate model such as double integrator or differential drive is used for planning. Improvements are demonstrated on five benchmarks that involve static or dynamic obstacle avoidance as compared to provably safe and approximately safe RL methods. A provable guarantee is provided for recovery regret.

Strengths

Presentation is clear with backing proofs and demonstrable results Problem that is being solved is clearly delineated and addressed using sound techniques Experimental comparisons are performed rigorously with attention to detail

Weaknesses

Literature review and comparisons are partial to the RL literature. There is a long-standing literature in control [A, B, C] to use an approximate model to plan using predictive control. A whole host of methods to learn a safety-filter/shielding on the fly has been explored with robust optimization-based offline and online control techniques. Most of these methods would implicitly solve the problem this paper is trying to address. However, it is interesting that the paper uses the Q function in the online optimization. This aspect is novel and unique to this paper. It is unclear how much computation and time it takes to run MCTS online at each time in order to do dynamic shielding at runtime. Dynamics model such as double integrator and differential drive are too simple. It would be interesting to see how well these would work with more complicated and/or higher-dimensional dynamics. [A] Breeden, Joseph, and Dimitra Panagou. "Predictive control barrier functions for online safety critical control." 2022 IEEE 61st Conference on Decision and Control (CDC). IEEE, 2022. [B] Wabersich, Kim P., and Melanie N. Zeilinger. "Predictive control barrier functions: Enhanced safety mechanisms for learning-based control." IEEE Transactions on Automatic Control 68.5 (2022): 2638-2651. [C] Wabersich, Kim P., et al. "Data-driven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems." IEEE Control Systems Magazine 43.5 (2023): 137-177.

Questions

Have the authors explored the space of RL training algorithms and methods to test this approach? Are there any advantages of using DMPS if the performance policy is not using RL and uses imitation learning or no learning at all? Exploration is important only for RL.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have discussed limitations and there is potential for the approach to scale even though the experiments in the paper are on simple examples.

Summary

This paper proposes a new method for safety shielding. More precisely, the authors extend Model Predictive Shielding (MPS), where an agent reverts to a safe backup policy if, for the next predicted state, this policy would not be able to guarantee safety anymore. MPS is often overly conservative, particularly in cases where the backup policy is very different from the final policy (for example, it may only consider breaking, while the final policy may be able to steer around an object). To improve this, whenever an action is potentially unsafe, the agent first uses a short-horizon planner to see if there exists some safe action that may be better than the one of the backup policy (i.e., one for which the backup policy could still recover in the future, but for which our learned agent predicts a higher reward). The authors formalize this framework and show recovery regret for this framework diminishes exponentially with the horizon. Next, they show that an implementation of this framework outperforms prior methods, both in terms of performance and the number of required shield invocations.

Strengths

The topic of the paper, safety shielding, is relevant and significant. Safe RL (and particularly, safety shielding) is a promising line of research but is often overly conservative in practice: the methods proposed in this paper take a step toward reducing this problem while still giving formal guarantees about safety. The topic is relevant for the NeurIPS community (particularly those interested in RL), both as a method that could immediately be used or to extend the method to more complex settings (i.e., with a stochastic/unknown model). The paper is well-written and easy to read: the intuition behind the method is clear, and the analysis of the results is easy to follow. The framework is well formalized (using visualizations where helpful), and the given pseudo-code helps with reproducibility. The experiments are extensive and convincingly show the advantages of the proposed method.

Weaknesses

Apart from some minor remarks that I add below, this paper has one main weakness: it does not clearly indicate the computational complexity of its method nor the scalability. The results do not show computation times, and (as far as I could tell) no mention is made of either the average planning time or some time limit for this planning phase. From some ball-parking, the additional time required for this method may be significant (solving up to millions of short-horizon planning problems), so a quantification of this computational cost should be provided. Some more minor remarks: * The paper only mentions how the framework is implemented (i.e., what RL & planning method it uses) in the appendix: it would be nice to (briefly) mention this in the results section as well; * In Table 2, the results of CPO and TD3 are not bold, even though some are equal to those of the best frameworks: this should be fixed; * One limitation of the proposed framework is that it assumes the environment is deterministic: it would be nice to mention this in the limitations section.

Questions

As mentioned in the 'weaknesses' section, I have one main question: how does the computational complexity of your method compare to those of the benchmarks, particularly MPS? I will change my rating if this question is not adequately answered.

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations are adequately addressed.

Summary

The authors introduce Dynamic Model Predictive Shielding (DMPS) an extension of Model Predictive Sheilding (MPS) that adress some of its key limitations, such as overconservatism when deploying the backup policy which consequently hinders exploration of the neural 'task' agent and slows down conergence. The key innovation of DMPS is that it incoropoates a local planner for dynamic recovery that leverages both a pre-computed backup policy and the neural 'task' policies Q values for planning for short and long horizon returns while maintaining safety of the system. DMPS is a provably safe reinforcement learning (PSRL) method, meaning that it guarantees safety of the system (regardless of the underlying neural 'task' policy) by only exploring within the set of safe and recoverable states defined by the backup policy. This realised by planning for a fixed n step trajectory and checking whether the system can be recovered by the backup policy given the action proposed by the agent. The authors demonstrate that DMPS outperforms MPS and other baselines in terms of performance and safety in various benchmarks. It also emphasizes the importance of aligning reinforcement learning agents with real-world safety requirements, while discussing some of the limitations of their approach.

Strengths

The paper has several strengths: I find that the paper is very well written and easy to follow, with sufficient details in necessary places and abstractions in other places where the details may not immediately matter, as such, it is a very nice read. The theoretical analysis of the recovery regret is convincing and interesting. Furthermore, the overall framework is attractive from the point of view that it is provably safe, something I personally find is crucial for deploying RL in the real world, rather than safe at convergence or in expectation like a lot safe RL methods. I find that the dynamic planning module is an innovative solution to the intuitive issue faced by most shielding methods (Figure 2) and I feel that this work constitutes a step in the right direction for improving shielding methods and making them more practical. The experimental evaluation I feel is strong and thorough as in most cases DMPS clearly outperforms MPS and REVEL, although I think it is missing something (see weaknesses).

Weaknesses

The key weakness of the PSRL framework is the reliance on a perfect (or sufficiently accurate) dynamics model of the environment, the safety performance of the backup policy and the computation of the safe invariant set. In contrast to the first shielding approaches for RL [1], which operate primarly on discrete state and actions spaces, DMPS does not need to compute the shield before learning can begining which significantly reduces the engineering overhead before training. This of course comes at a cost, in practice the shields in [1] are substatially more lightweight during "inference", (although in theory there could be exponential blow up) in part due to only operating on discrete or discretized state/action spaces but also because a lot of the necessary computation is done before hand. This is a key limitation of DMPS as it relys on planning at each timestep which might be costly and infeasible for on-board computation or edge devices. Fruthermore, it seems that there is still a significant amount of prior knowledge required for DMPS to work effectively, first we have to have a "perfect" dynamics model (for provable guarantees) secondly I presume we need to handcraft a backup policy and then compute its invariant safe set so as to plan for recoverability. The first limitation is mentioned in the paper but not really discussed in much detail, the second limitation is find is crucial and I don't think is really mentioned in the paper. In particular it is a non-trivial challenge to come up with a backup policy that has a maximal safe invariant set, perhaps for the environments the authors consider it is easy (just decelerate) but for more dynamics environments and in general this is not the case and I feel like more discussion about both these limitations (i.e. the limitations of the PSRL setting) is needed. While I find the experimental evaluation compelling I feel it is slightly misleading and it is missing something. In Table 2 CPO and TD3 score the same or higher in a few of the static benchmarks but there scores are not in bold, is there a reason for this that I am missing? I also feel like a comparison to PPO-Lag or DDPG-Lag would really help make the results that bit more convincing. All that being said, in principle I advocate for acceptance of this paper. [1] Alshiekh, Mohammed, et al. "Safe reinforcement learning via shielding." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

Questions

Most of my questions are technical: For each of the enviroments you consider how are the backup policies constructed and how are the invariat safe sets determined? For each of the environments what are the maxmimum number of steps needed to deccelerate the vehicle to zero or avoid obstacles and is your choice of n=5 sufficient? What would be suitable ways of modelling the environment from experience to obtain uncertainty estimates, for example would Gaussian Process modelling suffice? Do you assume any observation noise or just perfect access to the current state, if not how would you incorporate this into your framework?

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

See weaknesses.

Summary

The approach called dynamic model-predictive shielding for safe reinforcement learning is proposed as an improvement over its static counterpart. The main idea is to optimize for expected return on action with respect to the reinforcement-learning task when choosing a shielding backup action, and to incorporate planning horizon prediction into learning for the policy to learn to avoid unsafe actions. This dynamic version is evaluated on several static and dynamic obstacle-avoidance benchmarks and compared to static model-predictive shielding and three more planning-based approaches.

Strengths

The core idea of the approach is interesting and potentially valuable: to achieve synergy between safety and optimal performance in model-predictive shielding via incorporating planning into policy learning and taking expected performance into account during backup planning. Similar attempts have been done previously. In comparison, this work proposes a novel notion of "recovery regret" as a heuristic to guide mutual integration of planning and reinforcement learning. The strength of the paper is in extensive evaluation and comparison to multiple approaches. The notion of recovery regret can also be of independent interest for model-predictive shielding research. Dynamic shielding outperforms other approaches in the evaluation in terms of the number of shielding invocations, which indicates synergy between planning and learning over time.

Weaknesses

Potential weaknesses of the approach are in scalability of the planner and tightness of the probabilistic bounds on safety. Minor: - "more optimal recovery approach" --> an optimal/a better

Questions

Questions 1. In Figure 1, what are green and red lines, and a blue blob? 2. How does the local planner scale with respect to the look-ahead? 3. Does the local planner have to recompute the look-ahead prediction every time it is invoked or does it reuse previous results if the agent continues along the same trajectory? 4. What is the overhead of the planner's computations? 5. How does the planning limit affect safety and optimality guarantees? 6. MCTS typically struggles to plan for overly constrained problems and complex planning tasks. How does the approach scale with respect to the planning task complexity?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors explicitly discuss limitations which are fairly common for problem domain.

Summary

The paper seeks to address provably safe RL problems where safety must be ensured even during training. It proposed DMPS, which enhances prior Model Predictive Shielding approach, to dynamically select safe actions when danger is imminent. DMPS employs local planner to plan for recovery actions and the planner objective consists of both short-term and long-term rewards. Feedback from the planner can then be used to incrementally train the neural policy to guide it towards safe policy set.

Strengths

1. Quality * Overall, the approach described in the paper is sound and it combines many established components (e.g. backup policy, local planner, estimate future reward using model unrolling and Q-estimate) to facilitate safe RL. * The paper provides theoretical bound on the recovery regret as the sampling limit in the local planner approaches inifinity. 2. Clarity * The paper is written in a clear and lucid manner. The figures, algorithm and equations are structured in a way that is easily understandable to the readers.

Weaknesses

1. Originality * The main difference between DMPS and MPS is the use of local planner when backup policy is triggered. The technical approach used in DMPS is not particularly new as there are already some similar approaches of estimating a safety Q-value and perform planning based on the Q-value [1, 2]. 2. Significance * The only difference between DMPS and the prior MPS seems to be the local planner and (as discussed in point 1) this local planner is not particularly novel. Having said that, I do agree that the proposed DMPS does show improvement over MPS in some experiment scenarios. * While the paper mentions a small planning horizon is sufficient for the local planner to plan safe yet rewarding actions, I feel that this may not be true in most cases. To steer the agent back to safety (and yet rewarding), a long sequence of actions may be required. If the planning horizon is set too small, then DMPS falls back to backup policy and the performance would be the same as MPS. In this case, I guess the only solution is to increase in planning horizon and in turn increase the computational overhead of DMPS exponentially. * The local planner requires perfect information of the transition and the transition must be deterministic. This may restrict its applicability, especially given that there're prior work on model-based RL where transition can be stochastic and learned instead. References [1] Clavera, I., Fu, Y. and Abbeel, P., Model-Augmented Actor-Critic: Backpropagating through Paths. In International Conference on Learning Representations. [2] Thomas, G., Luo, Y. and Ma, T., 2021. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34, pp.13859-13869. [3] Nagabandi, A., Kahn, G., Fearing, R.S. and Levine, S., 2018, May. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA) (pp. 7559-7566). IEEE. [4] Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems. 2018. [5] Janner, M., Fu, J., Zhang, M. and Levine, S., 2019. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32.

Questions

1. In the first sentence of Section 6.2, you mentioned that the total return is averaged over the last 10 training episodes. Are you evaluating them using the same (single) random seed and only use the last 10 training episodes for evaluation? 2. Given that both the states and actions are continuous, how do you apply MCTS to the local planner? 3. TD3 only maximizes a single reward objective. In your experiments, I guess you performed some sort of reward shaping for it to balance between safety and reward. Can you elaborate how do you incorporate safety into its objective and is there any weighting used? 4. Similarly for CPO, how do you incorporate safety into it? Do you specify a safety violation constraint? 5. (Related to Qn 3 & 4) I am surprised that TD3 and CPO rapidly overfits to conservative policy in dynamic environment. What do you think is the reason and is the weighting between safety and reward dynamically tuned?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper discussed the known environment model as its limitation. I agree that this is a limitation which warrants further investigation. As studied in [5], it is very challenging for a model to accurately predict future trajectories with long horizon. Since DMPS relies on having an accurate environment model for safety adherence, this may limit its applicability to practical scenarios where environment model is not given and needs to be learned. Another related point is that it is unclear what value to set for the local planner horizon. Different tasks may require corrective action sequence of different lengths. Setting the horizon too short may revert DMPS performance back to MPS and setting the horizon too long may increase the computational overhead beyond acceptable threshold.