Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model

Response to Reviewer axZi

We thank the reviewer for the constructive comments and positive feedback on our paper. Regarding the concerns of the reviewer axZi, we provide the following responses: > **W1 & Q1 & W2.1 Some of safe online RL methods can be easily extended to offline RL settings, such algorithms should be included as a baseline.** We thank the reviewer for these works and we are happy to compare against them in our revised paper! We have added comparison with Sauté [1] and RCRL [2], since Sauté [1] is relatively versatile and RCRL [2] also adopts HJ Reachability to enforce hard constraints. Results can be found in Table 6, Appendix D.7 in our revised paper. - Note that applying safe online RL methods in offline settings faces distributional shifts. To address this, we add a BC constraint in Sauté [1] and RCRL [2] similar to an offline RL method TD3+BC [3] to combat distributional shifts. Also, we swept the conservatism strength to find a good balance between safety constraint and behavior constraint to obtain the best possible results. - However, Table 6 shows that we cannot obtain good results with direct adaptations of safe online RL methods into offline settings, even with carefully tuned hyper-parameters, due to the intricate coupled training processes. - FISOR, however, enjoys good results thanks to its decoupled learning objectives. Besides the experiments, we briefly discussed these safe online RL methods here: - For Sauté[1], similar to CPQ[4], safety constraints are handled by integrating them into the state space and modifying the objective accordingly. They struggle to handle zero cost limit, thus we still consider it to be a type of soft constraint method. - For [5], the intertwined learning process (barrier function, model, policy) is not suitable for offline scenarios. - For [6], its online safe exploration with environments is not compatible for offline settings. > **W2.2. Discuss the experimental results on the reward performance.** - The comparison criteria strictly follows the DSRL benchmark [7] (Q1 in general response), where safety is prioritized over rewards. We believe this makes sense as discussing the rewards is only meaningful when safety constraints are met in safety-critical tasks. - Previous methods may prioritize reward maximization while overlooking safety constraints. Thus, some baselines can obtain high rewards. Figure 3 shows that some baselines even encounter more safety violations than the ones trained without any safety constraint. This shows that existing safe offline RL methods are hard to strike the right balance between reward maximization and safety satisfaction. Thus, we didn't evaluate solely based on rewards, but filtered out the safe agents, and then compared the rewards, strictly following the DSRL [7]. We believe this evaluation protocol is more reasonable. > **W2.3. Discuss more on the Main Results** - Why does CPQ satisfy the safety constraints? - CPQ is an over-conservative method that prioritizes safety rather than reward maximization. Specifically, CPQ harshly penalizes unsafe areas by pulling up the cost values on those regions using value penalty [8]. However, value penalty is proved over-conservative [9], making CPQ primarily focus on safety constraints. For example, CPQ obtains good safety results on MetaDrive tasks but performs poorly in terms of rewards. - In addition, the offline value regularization is hard to tune [10], making CPQ perform not very well on other tasks. - Why FISOR does not perform well in Safety Gym tasks? - No. Table 1 shows that FISOR meets safety requirements (normalized cost<1 according to the DSRL evaluation protocol) and meanwhile achieves competitive rewards on all SafetyGym tasks as compared to baselines. Moreover, FISOR has the lowest average cost scores as compared to all other baselines. It would be appreciated if the reviewer could tell us why the reviewer thinks FISOR does not perform well in Safety Gym tasks. > **Q2. Could you explain the (empirical) relationship of the performance if FISOR and the data coverage? Is FISOR more efficient in terms of the data coverage compared to other baselines?** - We have added experiments on small data settings in Figure 10, Appendix D.8 in our revised paper. FISOR shows great stability across different dataset sizes, while other methods fail miserably under small data settings. We think the robustness in the low-data regime of FISOR is attributed to its stable decoupled learning procedure and policy extraction using weighted behavior cloning. > **Q3. Why only CPQ satisfies the safety constraints within the baseline methods?** Please see the responses for W2.3 for details. Note that although being very conservative, CPQ still fails to satisfy safety constraints in many tasks as the value regularization of CPQ is hard to tune. We use bold black and blue to represent safe agents in Table 1. For specific evaluation protocol, please see the general response in Q1 and Q2.

Thank you!!

I respectfully appreciate the authors for their clarifications and for adding new experiments. I've read the authors' feedback, other reviews, and revised manuscript. **W1 & Q1 & W2.1: Some safe online RL methods can be easily extended to offline RL settings, such algorithms should be included as a baseline.** - Thank you for adding informative experiments. I do appreciate the authors' effort in a short period. I am satisfied and convinced about this point. **W2.2, W2.3: Discuss the experimental results on the reward performance.** - Though I more or less agree with the authors, there is room to consider whether the authors' proposed method (FISOR) successfully solves the Safety-Gym environment given the low reward performance. I do agree that FISOR performs **better** than the baselines in most of the benchmark tasks, but there are several tasks that I personally think all algorithms could not solve. To put it in an extreme way, a very conservative policy (e.g., continuing to stay at the initial safe position) may be evaluated as a better policy if an agent does not have to get much reward. **Q3: Data coverage in offline RL settings.** - Thank you!! I think it is a nice addition for future readers. **W2.2 & Q3: Regarding CPQ** - Thank you for your explanation. I understand the reason why CPQ satisfies safety constraints. The authors have addressed most of my concerns at the time of the initial review, I increased my score from 6 to 8. I think this paper has been substantially improved. Thank you!!

Response to Reviewer 3zVo

We thank the reviewer for the constructive comments and positive feedback on our paper. Regarding the concerns of the reviewer 3zVo, we provide the following responses. > **W1. The authors may need to add more references to cover "safe RL with hard constraints" to better motivate this work and claim the differences.** We really appreciate the reviewer for providing these relevant works and we are more than happy to include them in our revised paper! Here, we briefly summarize the differences and relations with these related works: - For [1], the authors proposed a method for learning safe policy with joint soft barrier function learning, generative modeling, and policy optimization. Compared to HJ reachability, barrier functions may be overly conservative [2]. Also, the intertwined learning process is not suitable for offline RL scenarios. - For [3], although it considers state-wise constraints, this type of problem still allows for a certain degree of constraint violation. Therefore, we classify it as a soft constraint. Additionally, we have included more discussions of related works in Appendix A. Here, we provide a brief summarization. - For [4], although this method can achieve zero constraint violations, it requires domain knowledge of the task to design specific safety constraints, and thus is not suitable for general and more complex tasks. - For [5], the authors primarily focus on the expansion of safe regions in online exploration. However, interaction with the environment is not permitted in an offline setting. In contrast, we learn the feasible region directly from the dataset. > **W2. It seems that the baselines in the experiments all consider soft safety constraints.** To the best of our knowledge, **FISOR is the first that considers hard constraints in the offline setting**. Please see Table 4 for details. This is why we only compared soft constraints in the safe offline benchmark DSRL [6]. - The soft constraint baselines reported in our paper can be transformed into hard constraints by setting the cost limit to $0$. However, as shown in our paper (Figure 3), they suffer from extremely inaccurate estimations of the cost value and leads to increased safety violations (Table 2). - Some safe online RL methods that consider hard constraints can be extended to the offline RL settings by introducing some regularization to combat distributional shift. We combined RCRL [7], a safe online RL method that also uses HJ reachability, with TD3+BC [8] to adapt RCRL into the offline setting. The results can be found in Table 6, Appendix D.7 in our revised paper. Table 6 shows that RCRL+TD3BC fails miserably as it is hard to achieve a good balance between reward maximization, constraint satisfaction and distributional shift avoidance. FISOR, on the other hand, obtains great performance in the offline setting due to its decoupled learning procedure. > **Q1. How "accurate" the learned regions are? How does the learning region error affect the optimization problem and safety violations?** Please see Figure 1 for illustration. The learned feasible regions in FISOR are far more accurate than the ones learned by cost value functions. - In addition, we can select a small $0<\tau<1$ in $L_{rev}^{\tau}(u)$ of Eq.(8) to avoid infeasible states being mistakenly learned as feasible states. We have added demonstrative results in Figure 6, Appendix D.1 of the revised manuscript. Results show that a small $\tau$ will induce a smaller-sized learned feasible region, which is acceptable in safety-critical scenarios as it will induce safer policies. Figure 4(a) also supports this argument. - Moreover, estimation errors may increase as the amount of offline data decreases. We have added experiments on small dataset settings in Figure 10, Appendix D.8. Results show that FISOR is robust to small dataset settings as compared to baselines, thanks to its great training stability. [1] Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. ICML 2023. [2] Lyapunov density models: Constraining distribution shift in learning-based control. ICML 2022. [3] State-wise safe reinforcement learning: A survey. 2023 [4] Model-free safe control for zero-violation reinforcement learning. CoRL 2021. [5] Safe reinforcement learning in constrained Markov decision processes. ICML 2020. [6] Datasets and Benchmarks for Offline Safe Reinforcement Learning. 2023 [7] Reachability constrained reinforcement learning. ICML 2022. [8] A minimalist approach to offline reinforcement learning. NeurIPS 2021.

Response to Reviewer FU8i

We thank the reviewer for the constructive comments. Regarding the concerns of the reviewer FU8i, we provide the following responses. > **W1. There seems to be an inconsistency between the goal and the actual approach. The paper aims for 100% guarantee safety. However, the approach still needs to estimate the feasible areas and then yield the value function threshold. There could be estimation error and this estimation requires very thorough dataset provided, which may not be practical in reality.** We agree with the reviewer that we use deep neural networks to estimate the feasible regions and approximation errors could exist. However, we must clarify the following major misunderstandings. - First, note that the feasible areas assessed using HJ reachability are **far more accurate and robust** as compared to those measured using cost value functions, which are widely used in previous works, as demonstrated in Figure 1. Therefore, introducing HJ reachability into safe offline RL contributes a major step towards ensuring safety in safe offline RL settings. - Moreover, It can be quite challenging to accurately specify feasible regions in high-dimensional continuous control tasks without functional approximators such as deep neural networks. For example, MetaDrive tasks have extremely complex state spaces with over 250 dimensions. In this sense, it is **impractical** to enforce hard constraints without functional approximators. Specifically, previous safe online RL methods, that aim for 100% safety guarantee, also require to estimate the feasible regions, and approximation errors also exist [1][2][3]. None of these methods can achieve the ideal 100% safety guarantee unless the full state-action space is fully explored. To further address the reviewer's concern, we evaluated FISOR in small dataset settings. Please see Figure 10, Appendix D.8 for details. - Briefly, Figure 10 shows that a small offline dataset doesn't greatly impact FISOR's safety or performance. **This shows that FISOR is robust for small-sample scenarios**. - In addition, FISOR is the first offline RL method that considers hard constraints and proposes a stable decoupled training procedure. As shown in Table 1, FISOR is the only method that can ensure safety in all 26 benchmark tasks, while maintaining the highest reward in most of the tasks as compared to other safe baseline agents. This is a remarkable result since the safe offline RL problem with hard constraint satisfaction itself is an extremely challenging task, as it requires seeking the tricky balance among reward maximization, safety constraints satisfaction and distributional shift avoidance. > **W2. The learned infeasible states may be false negative. How do you assure that the infeasible states are really unsolvable?** Controlling $\tau$ in Eq. (8) adjusts the feasible region's conservatism. A higher $\tau$ near 1 means learning near-maximum feasible regions, while lowering $\tau$ leads to a smaller feasible region. - We added the visualization of learned feasible regions using different $\tau$ values in Figure 6, Appendix D.1 in our revised paper. Figure 4(a) also supports this argument. - In practice, we select $\tau=0.9<1$, from which a smaller feasible region will be learned and enforces harder safety constraints, where small errors will not likely lead to false negative but false positive. However, a false positive is acceptable since it will lead to a more conservative and safe policy. Although this may lead to some reward loss, it is acceptable in safety-critical scenarios. - If false negative leads to unsatisfactory unsafety by chance, we can increase the algorithm's conservatism by reducing the value of $\tau$. > **Q1. How do you guarantee the gap between the ground-truth infeasible regions and the learnt infeasible regions?** Please see the responses to W2 for details. > **Q2.1. How do you measure the normalized cost in the evaluation? Why is it not zero for cases that are 100% safe?** For the calculation method of normalized cost, please see the general response Q1. We respond to the latter question in W1. [1] Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions. RSS 2020. [2] Model-free safe reinforcement learning through neural barrier certificate. RAL 2023 [3] Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. ICML 2023.

Response to Reviewer 5fk7

We thank the reviewer for the constructive comments and positive feedback on our paper. Regarding the concerns from the reviewer 5fk7, we provide the detailed responses separately as follows: > **W1. & Q1. What is the criterion for an agent to be safe? Is it formulated as a threshold on the cost or do you count actual safety violations and only indicate those?** - Please refer to Q1 and Q2 in general response for details about our evaluation protocol. > **W2.1. How does limited data affect the method ?** - Limited data may have negative impacts on the performances, which is a common issue with existing offline algorithms, as high-performance offline policy learning usually requires a reasonable amount of offline training data as well as good state-action space coverage. - We have added new experiments on small data settings in Figure 10, Appendix D.8 in our revised paper. Figure 10 shows that FISOR can still maintain satisfactory safety levels with limited data, with a lower level of safety performance degeneration as compared to other methods. This could be attributed to the decoupled training approach in FISOR, which enhances policy learning stability under a low-data regime. > **W2.2. & Q2. What are other limitations?** Besides the limitation mentioned in the conclusion, please see Appendix F for more discussions on the potential limitations of FISOR. For other potential limitations: - FISOR contains four hyper-parameters, such as the forward/reverse expectile and policy extraction temperatures. However, we find that FISOR is not very sensitive to hyper-parameter changes and is robust to different tasks with the same hyper-parameters. We used **only one set** of hyper-parameters ($\tau=0.9$ for both forward/reverse expectile value), $\alpha_1=3, \alpha_2=5$) and achieved consistently good safety performances and reward returns among 26 tasks, as shown in Table 1. > **W2.3. & Q3. Are there safety constraints that cannot be handled or that are substantially more difficult?** We appreciate the reviewer for the constructive comment. We have added the following limitation in Appendix F. - In our setting, FISOR considers a very general form of constraint, which sets the safety violation signal to $0$ if safe and $1$ if unsafe. This allows flexible extension to a wide range of hard constraint satisfaction problems. However, this setup may not be applicable to constraints involving environmental disturbances [1] or probabilistic constraints [2]. These safety constraints might impact cost value estimations, thereby affecting the performance of the algorithm. > **W3. The figures are a bit blurry.** We thank the reviewer for this kind comment. We have increased the resolution of the images. We would really appreciate it if the reviewer could provide more specific suggestions so that we can make further improvements. [1] Safe reinforcement learning using robust control barrier functions. RAL 2022. [2] Separated proportional-integral lagrangian for chance constrained reinforcement learning. IEEE Intelligent Vehicles Symposium 2021.