Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning

Summary

1. This paper introduces the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Policy Inference (CPI) for encoding arbitrary constraint thresholds during policy optimization. 2. The proposed method can enable learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining.

Strengths

1. training efficiency and zero-shot adaptation capability: the trained policy can adapt to varying safety constraint requirements during deployment without retraining. 2. The theoretical analysis for the q-function estimation and constraint violations. 3. The extensive experiments are implemented to demonstrate the effectiveness of the method.

Weaknesses

See questions.

Questions

1. How reasonable/strong is assumption 2? Especially if latent vectors are highly dimensional. In experiments, what model do you use to estimate the latent vector z(\epsilon)? 2. What is the main technical challenge in extending CVPO [30] to the constraint-conditional scenarios? 3. How does this method scale when there are multiple constraints? 4. How does this work compare with the following missing literature with similar topics: Meta CMDP: Khattar, V., Ding, Y., Sel, B., Lavaei, J., & Jin, M. (2022, September). A CMDP-within-online framework for Meta-Safe Reinforcement Learning. In The Eleventh International Conference on Learning Representations. Non-stationary CMDP: Ding, Y., & Lavaei, J. (2023, June). Provably efficient primal-dual reinforcement learning for CMDPs with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence. Wei, H., Ghosh, A., Shroff, N., Ying, L., & Zhou, X. (2023, April). Provably Efficient Model-Free Algorithms for Non-stationary CMDPs. In International Conference on Artificial Intelligence and Statistics (pp. 6527-6570). PMLR.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Summary

The paper addresses Safe Reinforcement Learning with varying safety constraint requirements (Versatile Safe Reinforcement Learning). The paper proposes to learn a versatile Q function, which is decomposed as the product of feature functions of state and action, and the latent features of the constraint thresholds. The Q function will be used in a variational EM algorithm to find the constraint-satisfying policy that maximizes the return. The proposed method shows good safety and decent returns in experimental robotic tasks such as Run and Circle, outperforming simple constraint-conditioned and linear combination baselines.

Strengths

- The target problem is interesting and seems novel in the setting of Safe RL - The paper has relevant theoretical analysis to back up the method

Weaknesses

- The writing is unclear, especially in the pretraining phase. It is better to have an algorithm in the main text to summarize the process and connect the components in the method. It is hard to connect Fig. 1 to the method section. For example, where are the two bottom boxes in the pretraining phase described in the method section? - The experiment result overall is not really good. Although the safety criteria show good performance, the reward counterpart is not. In many tasks, the proposed method underperforms the baselines regarding the rewards. How to ensure that the trade-off is acceptable and the behaviour is desirable in Safe RL setting? Overall, more tasks should be tested to confirm if the lower performance is a consistent issue in the approach. Furthermore, there are claims on sample efficiency, yet no report on it can be found in the main text. It should be noted that running time should be considered as well because as admitted in the discussion, the method can be slow due to complicated optimization steps.

Questions

See Weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The method is complicated with different stages of optimization. Can it work in higher dimensional domains such as visual state space?

Summary

Safe reinforcement learning approaches have so far focused on the case where cost thresholds are static and known a priori. In contrast, this work addresses RL environments where cost constraints may change over time. To do this, the proposed algorithm learns a policy which is conditioned on the given cost threshold, allowing the policy to adapt to new cost thresholds at test time. The policy is derived from a pair of learned Q functions, one for rewards and one for costs. The Q functions are decomposed into a feature for state-action pairs and a feature for the safety threshold. The safety threshold feature uses a relatively low-dimensional model in order to provide better generalization without needing to see a large set of thresholds at training time, making training more efficient. In experiments, the proposed algorithm outperforms Lagrangian approaches to safety in a suite of relatively high-dimensional benchmarks.

Strengths

The problem of versatlie safe RL is interesting, potentially important, and (to my knowledge) novel. The threshold in CMDP-based safe RL work is often arbitrary, so it seems useful to be able to modify that threshold without retraining the system. Moreover, there may be some environments where the cost threshold changes over time. The proposed high-level approach is a very elegant solution to this problem. I particularly like the idea of separating out the effect of $\epsilon$ into a lower-parameter model which should generalize better than a full neural model. The way this allows for efficient training without sacrificing generalizability to new safety thresholds is quite impressive. Similarly, separating out Q-value learning from policy construction is an interesting way to generalize while maintaining the power of neural learning. The experiments are fairly extensive and show impressive results. In particular, CPPO demonstrates much better safety than other approaches on most benchmarks, while maintaining comparable performance. I think Figure 2 is a good illustration of the advantages of CPPO, showing how the number of observed cost closely tracks the allowed cost. This suggests that CPPO is using all of its available safety budget to achieve high rewards.

Weaknesses

I think the problem statement could do with a bit more motivation. While I do believe the versatile safe RL problem is interesting, it's not clear to me how often these kinds of problems actually arise. The given example of a car operating in highway vs. urban environments provides some intuition, but I'm not clear on what the cost signal is that would be changing in that case. An extra example or more explanation might help clarify this. I found the development in Section 4 to be difficult to follow. I'm not very familiar with the work on RL as inference, so maybe this explanation is sufficient for more expert readers. For less expert readers, I think it would help to add a bit more intuition or maybe some examples if feasible. For the experiments, I think it would help to include some baselines from the safe RL literature beyond basic Lagrangian approaches. For example, you might compare with CPO or Conservative Safety Critics (among others). These approaches are generally tailored to a single threshold, but a comparison could be made by training them with a strict threshold then using a more relaxed threshold at test time. In principle, CPPO should achieve a higher reward since it is able to take advantage of the increased safety budget.

Questions

Do you have any intiution about why TRPO-Lag achieves better safety on Ant-Run than CCPO? Is there some feature of the benchmark or the algorithm which makes this a particularly challening case for CCPO?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have adequately addressed some limitations and potential societal impacts in the paper.

Summary

The authors introduce the versatile safe RL problem, in which we want to maximize the expected reward, subject to an expected constraint, but for a range of constraint thresholds $\epsilon\in \mathcal E$ (compared to the traditional safe RL problem, which only aims to find a solution for one $\epsilon$). This is an interesting problem setting, with good motivation and potential real world application in learning a policy that works under many circumstances. The authors also provide an algorithm CVPO, that nicely combines the control as inference framework with the EM framework to improve the likelihood of optimality while trying to stay within a feasible set.

Questions

On Page 4, Line 134 - what is $K_f$? is it a constant for $f \in \{r, c\}$? Similarly, on line 146, I understand that $Poly(\epsilon, p)$ is a $p$-degree polynomial of $\epsilon$, but shouldn't it have a vector output since $z_f^*$ is a vector? Also, on line 158, why is it okay to normalize epsilon to [0, 1] since realistically constraint thresholds can be arbitrary? On Page 5, Line 197, the authors say - "The key strength of using the variational inference framework lies in its ability to encode arbitrary threshold conditions during policy learning, as shown in (7), a feat that is challenging for other methods, such as those based on primal-dual algorithms." Could you elaborate why this is challenging for other methods? For testing generalization to unseen thresholds, the authors train on data from $\epsilon$=20,40,60 and test with $\epsilon$=10,20,30,40,50,60,70. This is relatively close to the thresholds in the training data. It would be nice to see this algorithm trained with thresholds in [20, 30, 40] and tested with thresholds in [50, 60, 70]. Ideally, the algorithm may be able to provide feasible policies, but it is also important to test if the reward achieved is highest possible. If it is able to handle thresholds well outside the training set, that would be a good thing and it will show that the algorithm is quite versatile. If it is unable to do so, then, it should be discussed a bit further i.e. what unseen thresholds can we realistically expect good performance on? Suggestions related to language and typos: 1. On Page 5 - "We utilize the safe RL as inference framework ..." - since this is proposed by this work, it would be better to say "we introduce the ...". EDIT (16 Aug 2023): Updated score from 6->7.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

The study investigated a versatile, safe reinforcement learning problem and proposed the Conditioned Constrained Policy Optimization (CCPO) algorithm. CCPO may be helpful in ensuring safety under the settings of various constraint limits. Moreover, the experiment results demonstrate that CCPO works well.

Strengths

1. The problem is interesting. 2. The method may be useful for safe RL research.

Weaknesses

1. The code the study provided is not convenient to run; we have to download the code file by file. Could you provide the whole code? Such that we can examine the algorithm better. 2. Please check the whole paper, and improve the writing quality, e.g., on page 3, line 101, please check the sentence, and revise “z(1)” as “(1)”.

Questions

1. How can we handle non-convex optimization while ensuring safety? 2. If the cost limits are linear, what will happen? 3. Could you compare it with CPO-based methods?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

1. The balance between reward and cost is not handled well. 2. Although the Lagrangian-based methods can not perform well on various cost-limit settings, their reward performance is better than CCPO.

upgrade score

Thank you for your feedback. I am pleased with the response and intend to revise the score in favor of acceptance.

Rebuttal response

I am satisfied with the authors' response. I will increase my score to 7.

Rebuttal response

Thank you for answering my questions. The rebuttal mostly cleared my concerns. Hence, I will raise my score to 7. Regarding the point "we can conclude that CCPO outperforms baselines in most experiments", I think the authors need to report the results by following the "evaluation criteria in safe RL". For example, in the result table, methods that satisfy constraints should be grouped and compared in terms of rewards. It is best if more tasks are considered to ensure that the trade-off is acceptable in various settings.