Library

Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation

Safe reinforcement learning (RL) is crucial for deploying RL agents in real-world applications, as it aims to maximize long-term rewards while satisfying safety constraints. However, safe RL often suffers from sample inefficiency, requiring extensive interactions with the environment to learn a safe policy. We propose Efficient Safe Policy Optimization (ESPO), a novel approach that enhances the efficiency of safe RL through sample manipulation. ESPO employs an optimization framework with three modes: maximizing rewards, minimizing costs, and balancing the trade-off between the two. By dynamically adjusting the sampling process based on the observed conflict between reward and safety gradients, ESPO theoretically guarantees convergence, optimization stability, and improved sample complexity bounds. Experiments on the Safety-MuJoCo and Omnisafe benchmarks demonstrate that ESPO significantly outperforms existing primal-based and primal-dual-based baselines in terms of reward maximization and constraint satisfaction. Moreover, ESPO achieves substantial gains in sample efficiency, requiring 25--29% fewer samples than baselines, and reduces training time by 21--38%.

cs.LGReinforcement Learning in RoboticsAnomaly Detection Techniques and ApplicationsComputer Science
View graphSource

Paper

Similar papers

Safe Reinforcement Learning via Episodic ControlZhuo Li, Derui Zhu, Jens Großklags2025 · IEEE Access · 2 citations In Library
CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement LearningA. Belouadah, Sylvain Kubler, Y. L. Traon2026 · 0 citations In Library
Safe Policy Optimization with Local Generalized Linear Function ApproximationsAkifumi Wachi, Yunyue Wei, Yanan Sui2021 · arXiv (Cornell University) · 12 citations In Library
Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient ManipulationShangding Gu, Bilgehan Sel, Yuhao Ding et al.2024 · AAAI Conference on Artificial Intelligence · 30 citations In Library
Safety Guided Policy OptimizationDohyeong Kim, Yunho Kim, Kyungjae Lee et al.2022 · 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · 3 citations In Library
Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated RewardsHanping Zhang, Yuhong Guo2025 · arXiv (Cornell University) · 0 citations In Library
Iterative Reachability Estimation for Safe Reinforcement LearningMilan Ganai, Zheng Gong, Chenning Yu et al.2023 · Neural Information Processing Systems · 47 citations In Library
Sampling-Based Safe Reinforcement LearningLuca Vignola, Bruce D. Lee, Manish Prajapat et al.2026 · arXiv.org · 0 citations In Library
Data Efficient Safe Reinforcement LearningSindhu Padakandla, K. J. Prabuchandran, S. Ganguly et al.2022 · 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · 4 citations In Library
Efficient Exploration Using Extra Safety Budget in Constrained Policy OptimizationHaotian Xu, Shengjie Wang, Zhaolei Wang et al.2023 In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer qrLu5/10 · confidence 3/52024-06-19

Summary

This paper proposes a safe RL algorithm that adjusts the sample number during one on-policy update based on the conflict between the reward gradient and the cost gradient. When the policy's cost value is near the constraint threshold and there is a gradient conflict, a larger sample number is used; otherwise, a smaller sample number is applied. The paper also provides theoretical results about convergence rates, reducing oscillation, and sample efficiency. Finally, the paper evaluates the algorithm on two safe RL benchmarks.

Strengths

- It is an interesting and innovative attempt to improve sample efficiency for safe RL by dynamically adjusting the sample number of on-policy updates based on gradient conflict. - Some efforts on theoretical analysis are provided to justify the proposed approach. - Empirical results demonstrate the capability of this work to improve performance and sample efficiency.

Weaknesses

- The sample manipulation is an interesting mechanism, and the gradient conflict, as a manipulation signal, occurs widely in many safe RL algorithms, including both primal and primal-dual methods. However, it is solely applied to PCRPO, which may weaken its generalization and persuasiveness and makes it seem like a minor improvement tailored to PCRPO. - The description of the main backbone PCRPO is insufficient, making it difficult to understand the whole algorithm pipeline without prior knowledge of PCRPO. See question 1, 2. - Some points about the theoretical results remain to be clarified. See questions 3 and 4. - Some experimental settings are debatable. See question 5.

Questions

1. Eq. (3)(4) of PCRPO, serving as an essential component of your algorithmic pipeline, is kind of indigestible. How to set $x^r_t$, $x^c_t$? And how is Eq. (3) functioning as 'projecting reward and cost gradients onto their normal planes'? It would be better to add more details or illustrations about PCRPO (at least in the appendix). 2. I see the $h^+,h^-$ dynamically change in rows 4, 7 in Algorithm 1 ESPO. Is it an existing component of PCRPO or a new trick in ESPO? 3. From the description of Theorem 4.1 and Proposition 4.2, I cannot find the association between sample manipulation and these two theoretical results. Can I view them as conclusions about PCRPO and not tailored to ESPO? 4. It seems that Assumption A.7, which bridges sample size and performance, serves as a very important theoretical base to verify the sample size manipulation. There should be more intuitive explanations and discussions to justify this assumption. 5. I guess you use a fixed number of training epochs for all algorithms. But training epochs are not a good metric for comparison, especially when each epoch corresponds to different sample steps, so maybe the x-axis of Figures 2 and 3 should be set to sample steps rather than training epochs to fairly demonstrate the sample efficiency of all algorithms. Besides, Table 1 should report the sample steps used to reach the same performance rather than the sample steps in a fixed number of epochs.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The limitations are discussed in the paper.

Reviewer qrLu2024-08-08

Thanks for the authors' response. The supplementary experimental results are comprehensive, and the additional clarifications address most of my confusions. Thus, I decide to raise the score to 4. However, I still hold a slightly negative opinion due to two main concerns: (1) the limited applicability of sample manipulation to broader algorithms (the paper structure seems overly tailored to PCRPO, even though some additional experimental results about TRPO-Lag and CRPO are provided), and (2) the weak connection between the theoretical analysis and the main contribution (sample manipulation). Just a reminder, it seems there is a typo in your response A2 where $g_r-\frac{g_r\cdot g_c}{||g_r||^2}g_c$ should be corrected to $g_r-\frac{g_r\cdot g_c}{||g_c||^2}g_c$ according to the paper.

Authorsrebuttal2024-08-08

Dear Reviewer, Thank you for engaging in discussion with us and for appreciating our new experimental results! We will discuss on your further insightful comments. We have corrected the typo to $g_r - \frac{g_r - g_c}{\|g_c\|^2}g_c$. Regarding your other comments: - **Adapting the presentation to highlight the generalization of sample manipulation to extensive safe RL algorithms and problems.** The reviewer is correct that we did a step-by-step presentation to introduce the proposed algorithm ESPO, it is indeed built on PCRPO which serves as an example in our algoirthm framework. As the reviewer suggested, we have decided to adapt the presentation for further clarification (in section 4 and appendix) by following the structure as below: - **Showing the key module sample manipulation** --- depending on the conflict signal of reward and cost gradients. - **How to integrate it into diverse safe RL algorithms**: introducing PCRPO and then the corresponding ESPO (PCRPO + sample manipulation) as a detailed example; introduce other examples regarding the primal method CRPO and primal-dual method TRPO-Lagrangian. (The experimental results are presented in Section 5 and **Q1** in the general response.) - **How to integrate it into more complex safe RL problems** --- multi-objective safe RL problems. (The experimental results are presented in **Q1** in the general response.) We appreciate the reviewer for enabling us to verify the generalization power of our sample manipulations and helping us improve the current paper organization. - **Main contributions on the theoretical side:** Thanks for raising this question, which is indeed an essential contribution in our theoretical part. A brief answer would be: our theoretical analysis is a more general framework that is not only for sample manipulation module, but extensive primal-based safe RL algorithms. The separation of the theoretical framework from sample manipulation is not a flaw, but an intentional advantage for generalization. Specifically, recall that we provide three provable advantages for ESPO --- 1) convergence in terms of both the optimal reward and the constraint requirement (Theorem 4.1); 2) efficient optimization with reduced oscillation (Proposiiton 4.2); sample efficiency with sample size manipulation (Propositon 4.3). - **The sample efficiency guarantee directly results from sample manipulation.** The reviewer is correct in noting that the provable advantages of ESPO also depend on other modules since all modules influence the optimization process. But sample manipulation plays a key role in supporting the sample efficiency guarantees for ESPO. - **The separation ability of the theoretical framework from sample manipulation is not a flaw, but an intentional advantage for generalization.** We would like to highlight that our theoretical guarantees (also the technical tools) for optimization perspective --- convergence (Theorem 4.1) and the advantages of reduced optimization oscillation (Proposition 4.2) --- can potentially work for a lot more primal-based safe RL algorithms even without sample manipulation (such as PCRPO). This is attributed to the fact that the theoretical analysis can be (mostly) decomposed into an optimization part and a statistical part, where sample manipulation primarily has an impact. Notably, PCRPO is currently one of the state-of-the-art safe RL algorithms, while still needing theoretical guarantees for convergence. Our theoretical results for ESPO also hold for the prior work PCRPO, which are direct implications of our results. This work closes this gap for PCRPO and can be useful to provide convergence guarantees for extensive primal-based safe RL algorithms.

Authorsrebuttal2024-08-12

Dear Reviewer, Thank you for your valuable comments. We have responded to your comments regarding the differences compared to PCRPO and our theoretical contributions. If you have any further questions or comments, please don't hesitate to let us know. As the rebuttal deadline is approaching, we hope our response can address your concerns. We appreciate your time and expertise in reviewing our work. With gratitude Authors

Reviewer by8k5/10 · confidence 4/52024-07-11

Summary

The paper introduces an approach, Efficient Safe Policy Optimization (ESPO), aimed to improve the efficiency of safe reinforcement learning (RL). ESPO tries to enhance sample efficiency through sample manipulation, addressing the challenges of sample inefficiency in safe RL, which often requires extensive interactions with the environment to learn a safe policy. The proposed method dynamically adjusts the sampling process based on the observed conflict between reward and safety gradients.

Strengths

1. The paper is well motivated, articulates the challenges in existing safe RL methods and justifies the need for dynamic sample manipulation. 2. ESPO's dynamic sample manipulation based on gradient conflicts seems relevant to the field of safe RL, potentially reducing computational costs and improving learning efficiency. 3. The paper provides both theoretical analysis and empirical validation.

Weaknesses

1. While the paper evaluates ESPO on two benchmarks, additional evaluations on more diverse and complex environments would strengthen the generalizability claims. 2. More in-depth comparisons with a broader range of SOTA methods could provide a clearer picture of ESPO's relative performance.

Questions

1. How does ESPO perform in environments with high-dimensional state and action spaces compared to low-dimensional ones? 2. What is the impact of noisy gradient estimates on the performance and stability of ESPO? 3. Can ESPO be effectively integrated with off-policy or model-based RL methods to further enhance sample efficiency?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

1. ESPO's reliance on gradient conflict signals for sample manipulation might limit its applicability in environments where gradient estimation is noisy or unreliable. 2. The method's scalability to very large or real-time environments is not thoroughly explored, raising questions about its practical deployment in such settings. 3. The sensitivity of ESPO to various hyperparameters, such as learning rates and sample size thresholds, needs further investigation.

Reviewer oXwv6/10 · confidence 3/52024-07-12

Summary

The paper introduces Efficient Safe Policy Optimization (ESPO), an approach that enhances safe reinforcement learning by dynamically adjusting sample sizes based on gradient conflicts. ESPO optimizes reward and safety, improves convergence stability, and reduces sample complexity. The experiments shows the proposed approach outperforms existing methods for both higher reward and improved safety with fewer samples and less training time.

Strengths

1. The paper introduced Efficient Safe Policy Optimization (ESPO), which dynamically adjusts sample sizes based on observed conflicts between reward and safety gradients. The approach is novel 2. The paper provides a comprehensive theoretical analysis of ESPO, including convergence rates and optimization stability. 3. The paper conducted experiments on the Safety-MuJoCo and Omnisafe benchmarks, ESPO demonstrates significant improvements over existing primal-based and primal-dual-based methods.

Weaknesses

The paper experimented on SafetyReacher-v4, SafetyWalker2d-v4, and SafetyHopper/AntVelocity. Those safety tasks does not test generalization, such as those with safety gym.

Questions

None

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discussed limitations.

Reviewer oXwv2024-08-07

Apologies for the confusion. Notice that the experiments conducted in the paper mainly use velocity as the safety constraint and is a fixed threshold. In the original safety gym environment (openai/safety-gym), the safety constraints are typically obstacles and hazards (e.g., a car reaching the goal without hitting obstacles/overlapping with hazards). The location of the obstacles/hazards varies in each run, thus testing generalization in some sense. I felt the safety-velocity benchmarks are easier than the openai safety-gym original settings (commonly used in SafeRL community). However, I acknowledge this is a weakness but not a solid rejection reason.

Authorsrebuttal2024-08-08

Thank you for engaging in the discussion and providing further clarification of the question. We really appreciate the reviewer for proposing this constructive suggestion to improve the quality of our work, making the generalization ability of ESPO more convincing and clearer. We provide **new experiments** as well as more dicussions. The results show the generalization ability of the proposed method ESPO to not only varying unsafe factors, but also diverse types of safety constraints. - **New experiments on SafetyCarGoal1-v0 and SafetyPointGoal1-v0: ESPO has generalization ability to handle varying unsafe factors**. As the reviewer suggested, we conduct experiments on two new benchmarks SafetyCarGoal1-v0 and SafetyPointGoal1-v0: the car or the robot ball needs to navigate to the Goal’s location while circumventing Hazards that could vary in each running time. Hazards bring risks that could result in costs when an agent enters unsafe regions [5]. We follow the same experimental settings in this paper, The results show the required number of training steps when they reach the desired performance (reward) and satisfy the safety constraints, which show the superior sample efficiency of the proposed method ESPO. - **The proposed algorithm ESPO also has the generalization ability to handle diverse types of safety constraints and a combination of them.** Notably, we would also like to highlight that we also conducted experiments on safe RL with constraints not only on velocities (SafetyHopperVelocity-v1 and SafetyAntVelocity-v1), but also others such as robot’s control force energy (SafetyReacher-v4, SafetyWalker-v4, SafetyHumanoidStandup-v4, see Section 5.1). ESPO showed superior performance on not only sample efficiency, but also reward perforamnce and safety satisfication. **Table 1: Comparisons of the required sample steps for achieving the same desired reward while ensuring safety (cost limit: 15) on SafetyCarGoal1-v0 and SafetyPointGoal1-v0.** | Task \ Algorithm | ESPO (Ours) | CUP [1] | PPOLag [2,3] | PCPO [4]| |-----------------------------------|-------------|------|-------|---| | SafetyCarGoal1-v0 (Reward:6.6) | 1.9 M | 4+ M | 2.4 M | 2.3 M| | SafetyPointGoal1-v0 (Reward:3.7) | 0.7 M | 1.1 M | 4+ M | 1.7 M| > [1] Yang, L., Ji, J., Dai, J., Zhang, L., Zhou, B., Li, P., ... & Pan, G. (2022). Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35, 9111-9124. [2] Ji, J., Zhou, J., Zhang, B., Dai, J., Pan, X., Sun, R., ... & Yang, Y. (2023). Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304. [3] Ray, A., Achiam, J., & Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1), 2. [4] Yang, T. Y., Rosca, J., Narasimhan, K., & Ramadge, P. J. Projection-Based Constrained Policy Optimization. In International Conference on Learning Representations, 2020. [5] Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., ... & Yang, Y. (2023). Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36.

Authorsrebuttal2024-08-12

Dear Reviewer, Thank you for your constructive comments. In response, we have deployed our method to Safety-Gym tasks to demonstrate its generalizability, including experiments on safe robot and car navigation in obstacle/hazard environments (SafetyCarGoal1-v0 and SafetyPointGoal1-v0 tasks). If you have any further questions or comments, please don't hesitate to let us know. As the rebuttal deadline approaches, we hope our response adequately addresses your concerns. We sincerely appreciate your time and expertise in reviewing our work. With gratitude, The Authors

Reviewer qn1v6/10 · confidence 4/52024-07-12

Summary

This paper presents a novel algorithm for safe reinforcement learning, ESPO, which independently collects gradient information for reward optimization and constraint satisfaction. It then makes a dynamic choice about how to combine these two gradients in order to find an optimal safe policy. In addition, the proposed algorithm dynamically adjusts the number of samples used for each gradient computation resulting in a more sample-efficient learning process than existing techniques. Theoretical results show that ESPO achieves near-optimal reward and constraint satisfaction with high probability. Additional results show that ESPO spends more time in safe regions during training than prior work. In experiments, ESPO is able to achieve comparable or better reward and safety behavior to prior approaches while requiring less training time.

Strengths

- Safe RL is quite an important area, and advancements in efficient safe RL make it more applicable to real-world scenarios. - The proposed algorithm is quite intuitive and handles a tricky issue (oscillation) appearing in other safe RL algorithms - The algorithm is well-grounded in theory with Theorem 4.1. I also appreciate 4.2, showing the improved constraint satisfaction at training time compared to existing work. - The experimental results are promising. ESPO achieves comparable or better results in terms of both reward performance and constraint satisfaction to existing work, but uses fewer samples.

Weaknesses

- There is minimal discussion of the coefficients $x_t^r$ and $x_t^c$ even though these seem like they should be critical to the algorithm's performance. - It seems that the hyperparameters of ESPO require careful tuning which likely inhibits the deployment of this algorithm in practice. (I'm looking at Tables 5 and 6 for this claim which show that the hyperparameters are quite different for different benchmarks.) - In some cases the either $h^-$ or $h^+$ is infinite, meaning the algorithm never performs a pure constraint satisfaction step or a pure reward optimization step.

Questions

- How are the coefficients $x_t^r$ and $x_t^c$ computed? What impact do they have on performance? - How are the other hyperparameters ($\chi^+$, $\chi^-$, $h^+$, $h^-$) chosen? Does it require careful tuning for each environment?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately discuss limitations.

Reviewer qn1v2024-08-12

Thank you for your response and I apologize for the error in my question. I did indeed mean to ask about $\zeta^+$ and $\zeta^-$ rather than $\chi^+$ and $\chi^-$. I will keep my score in favor of accepting the paper.

Authorsrebuttal2024-08-12

Dear Reviewer, Thank you for your follow-up message and for clarifying the focus of your question. We are grateful for your favorable scoring towards our paper. Your insights are valuable to our work. Thank you once again for your thorough review and consideration. Best regards, The Authors

Authorsrebuttal2024-08-12

Dear Reviewer, Thank you for raising your score. Your constructive comments are valuable to our work. We sincerely appreciate your thoughtful review and the effort you put into reviewing our work. Best regards, The Authors

Authorsrebuttal2024-08-12

Dear Reviewer, We sincerely appreciate your valuable comments on our paper. In response, we have deployed our method to more challenging tasks and other algorithms, including: (1) Primal-based methods (e.g., CRPO [1]); (2) Primal-dual based methods (e.g., TRPOLag [2,3]); (3) Safe multi-objective reinforcement learning (e.g., CRMOPO [4]). The results of these experiments demonstrate that our method exhibits superior performance compared to state-of-the-art baselines across diverse tasks and algorithms in terms of safety, reward, and sample efficiency. These findings suggest that our sample manipulation approach can serve as a general method for safe RL and potentially extend to multi-objective learning scenarios. If you have any further questions or require additional clarification, please don't hesitate to let us know. As the rebuttal deadline approaches, we hope our response can address your concerns, and we are grateful for your time and expertise in reviewing our work. With gratitude, The Authors > [1] Xu, T., Liang, Y., & Lan, G. (2021, July). Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning (pp. 11480-11491). PMLR. [2] Ray, A., Achiam, J., & Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1), 2. [3] Ji, J., Zhou, J., Zhang, B., Dai, J., Pan, X., Sun, R., ... & Yang, Y. (2023). Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304. [4] Gu, S., Sel, B., Ding, Y., Wang, L., Lin, Q., Knoll, A., & Jin, M. (2024). Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning. arXiv preprint arXiv:2405.16390.

Authorsrebuttal2024-08-13

Dear Reviewer, Thank you for raising your score. We sincerely appreciate your constructive feedback, which is valuable to our work. We are grateful for your time and effort in reviewing our work. Best regards, The Authors

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC