Off-Policy Primal-Dual Safe Reinforcement Learning

We thank the reviewer for all the valuable comments. Please refer to the point-to-point responses below. ## W: Clarification about the proof that ALM does not alter the position of local optima. Thank you very much for pointing out this issue, and we clarify that this actually results from a typing error. Specifically, in our original submission, we mistyped the actual objective \begin{equation} \min\_{\lambda\geq 0}\max\_{\pi}\mathbb{E}\_{s\sim \rho\_\pi,a\sim\pi(\cdot|s)}\left[\widehat{Q}\_r^\pi(s,a)\right] - \frac{1}{2c}\left[\left( \max \{0, \lambda-c(d-\mathbb{E}\_{s\sim \rho\_\pi,a\sim\pi(\cdot|s)}\left[\widehat{Q}\_c^{\text{UCB}}(s,a)\right]) \} \right)^2-\lambda^2\right] \end{equation} as \\begin{equation} \\min\_{\lambda\geq 0}\\max\_{\\pi}\\mathbb{E}\_{s\\sim \\rho\_\pi,a\\sim\\pi(\\cdot|s)}\\left\\{\\widehat{Q}\_r^\\pi(s,a) - \\frac{1}{2c}\\left[\\left( \\max \\{0, \\lambda-c(d-\\widehat{Q}\_c^{\\text{UCB}}(s,a) \\} \\right)^2-\\lambda^2\\right] \\right\\} \\end{equation} by missing an inner expectation. Apparently, the actual objective does not require an interchange between the expectation and the max or square operator, and thus does not cause the trouble mentioned by the reviewer in the proof that ALM does not alter the position of local optima. To see why we actually used the objective with an inner expectation, we can first transform this objective into an equivalent form (i.e., Objective (4a)(4b) in our revision, page 4): \\begin{cases} \\min\\limits\_{\\lambda\\geq 0}\\max\\limits_{\\pi}\\mathbb{E}\\left[\\widehat{Q}\_r^\\pi\\right] - \\lambda\\left(\\mathbb{E}\\left[\\widehat{Q}\_c^{\\text{UCB}}\\right] - d\\right)-\\frac{c}{2}\\left(\\mathbb{E}\\left[\\widehat{Q}\_c^{\\text{UCB}}\\right] - d\\right)^2,&\\text{if}\ \\frac{\\lambda}{c}>d-\\mathbb{E}[\\widehat{Q}\_c^{\\text{UCB}}] \\\\ \\min\\limits\_{\\lambda\\geq 0}\\max\\limits\_{\\pi}\\mathbb{E}\\left[\\widehat{Q}\_r^\\pi\\right] + \\frac{\\lambda^2}{2c} ,&\\text{otherwise}. \\end{cases} When $\frac{\lambda}{c}\leq d-\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$, the objective does not involve $\widehat{Q}\_c^{\text{UCB}}$, which is reasonable since we have $d\geq\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$. When $\frac{\lambda}{c}>d-\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$, differentiating the objective w.r.t. the policy yields \\begin{equation} \\mathbb{E}\\left[\\nabla\_\\pi\\widehat{Q}_r^\\pi\\right]-(\\lambda-c(d-\\mathbb{E}\\left[\\widehat{Q}\_c^{\\text{UCB}}\\right])\\cdot\\mathbb{E}\\left[\\nabla\_\\pi\\widehat{Q}\_c^{\\text{UCB}} \\right], \\end{equation} which is equivalent to \begin{equation} \mathbb{E}\left[\nabla\_\pi\widehat{Q}\_r^\pi-(\lambda-c(d-\mathbb{E}\left[\widehat{Q}\_c^{\text{UCB}}\right])\cdot\nabla\_\pi\widehat{Q}\_c^{\text{UCB}} \right]. \end{equation} By replacing $\nabla\_\pi\widehat{Q}\_r^\pi$ with the SAC objective, this gradient corresponds exactly to the policy update in our pseudo code (Line 11 of Algorithm 1 in the **original version** of our paper) and also to the actual implementation (Lines 163-208 in `./CALsubmission/sac/cal.py` in `CAL_code.zip` which can be found in our supplementary materials). As for the dual update for $\lambda$, according to this objective, we should perform $\\lambda \\leftarrow \\max\\left\\{0, \\lambda - \\alpha\_\\lambda \\mathbb{E}\_{s,a\\sim\\mathcal{B}} \\left[d-\\widehat{Q}\_c^{\\text{UCB}}(s,a) \\right]\\right\\} $ when $\frac{\lambda}{c}> d-\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$, and perform $\lambda\leftarrow \max\\{0,\lambda-\frac{\lambda}{c}\\}$ otherwise. However, empirical results show that exclusively performing the former update rule is sufficient to obtain a satisfactory performance, so we adopt this form in our pseudo code (Line 14 of Algorithm 1 in the original version of our paper) and also our implementation (Lines 221-229 in `./CALsubmission/sac/cal.py`). We have modified corresponding parts in our paper (Section 3.2 and Appendix A). Please refer to our revision and let us know if there is still any confusion.

## Q1: Why do we have to reformulate the augmented Lagrangian? According to our above clarification about the proof that ALM does not alter the position of local optima, the two formulations are equivalent after we add the missing inner expectation in the reformulated objective. To make the objective more intuitive and easier to understand, in the revised version, we have replaced the objective in Section 3.2 with: \\begin{equation} \\begin{cases} \\min\\limits\_{\\lambda\\geq 0}\\max\\limits\_{\\pi}\\mathbb{E}\\left[\\widehat{Q}\_r^\\pi\\right] - \\lambda\left(\\mathbb{E}\\left[\\widehat{Q}\_c^{\\text{UCB}}\\right] - d\\right)-\\frac{c}{2}\\left(\\mathbb{E}\\left[\\widehat{Q}\_c^{\\text{UCB}}\\right] - d\\right)^2,&\\text{if}\\ \\frac{\lambda}{c}>d-\\mathbb{E}[\\widehat{Q}\_c^{\\text{UCB}}] \\\\ \\min\\limits\_{\\lambda\\geq 0}\\max\\limits\_{\\pi}\\mathbb{E}\\left[\\widehat{Q}\_r^\\pi\\right] + \\frac{\\lambda^2}{2c} ,&\\text{otherwise}. \\end{cases} \\end{equation} The reason why we did not use this form in our original manuscript is that we wanted to simplify the objective into one single line. However, as the reviewer suggests, this might cause confusion, so we have modified our manuscript to mitigate this issue. ## Q2: About the constraint threshold in Hopper. In theory, according to our description of how ALM helps addressing the instability and over-conservativeness issue, $c=0$ (without ALM) means that the performance can become instable and over-conservative. Therefore, the fact that **"when $c=0$, the test-cost seems to be not that bad in the sense that average is below the threshold"** in Hopper does not contradict the above statement since the reward in this case oscillates violently and is overly conservative. ## Q3: About the page number for [Luenberger et al., 1984]. Thanks for the suggestion. The part relevant to the derivation of augmented Lagrangian in [Luenberger et al., 1984] is Pages 445-453. We have added the page number to the place where the ALM objective first appears (Section 3.2) in our revision. ## Q4: Why does taking minimum over $\\pi$ implies underestimation bias? We misplaced the min operator in the inequality in Section 3.1, and the true inequality we used is actually$ \\mathbb{E}\_\\epsilon [{\\min_\pi}({Q}\_c(s,\\pi(s)) +\\epsilon)]\\leq\\min\_\\pi {Q}\_c(s,\\pi(s)) $ for any given $s$. We have corrected this in the revision. In terms of why this inequality is generally true, there is an intuitive experiment in [Thrun & Schwartz, 1993] (Section 2) that helps illustrate this phenomenon, which is also used to serve as the motivation of addressing the overestimation bias in $Q$ function approximation in the TD3 paper (Section 4 in [Fujimoto et al., 2018]). The key point here is that due to the function approximation noise, some of the Q-values might be too small, while others might be too large. The min operator, however, always picks the smallest value, making it particularly sensitive to underestimations. ## Q5: Introduce and rephrase the theorem in A.M. Duguid. Thanks for the advice. We have introduced the corresponding theorem as a lemma in Appendix B, where we also show how this lemma leads to the fact that there exists a $c^*>0$ such that if $c>c^*$, the damping matrix $A$ is positive definite at the constrained minima.

We thank the reviewer for the valuable comments. Please refer to the point-to-point responses below. ## W: About the performance with multiple competing constraints. We agree that this is an interesting research problem. Theoretically, there is no essential difference when switching to the multi-constraint setting since we can simply replace the scalar $Q\_c$ functions and the dual variable with a vector. However, in practice, depending on how these constraints relate to each other, the difficulty in balancing different constraints in policy optimization may vary. One practically meaningful constraint we can come up with in MuJoCo tasks is the amount of torque of the agent's joints (a large torque may shorten the lifespan of the motor). Yet, we are not sure if this torque constraint would compete with the speed constraint used in our paper, and we are currently investigating this. To be more specific, we are conducting experiments to determine an appropriate value for this constraint in different tasks and for different joints in a single robot. Also, we are trying to incorporate this constraint with the speed constraint to see how the methods perform in such a setting. We will append these results in our future revision as soon as we have finished this experiment. ## Q1: About the cost value ensemble. > How the cost value ensemble in (3) is created? Specifically, we maintain $E$ bootstrapped $Q\_c$ networks, i.e., $\\{\\widehat{Q}^i\_c \\}\_{i=1}^E$, which share the same architecture and only differ by the initial weights. These weights are initialized by sampling from a normal distribution (The corresponding part in our code: `torch.nn.init.xavier_uniform_(m.weight, gain=1)`). All the networks are trained on the same dataset. > Do you have multiple (i.e., $E$) parallel parameterized models that each estimate the cost in parallel? Yes. During cost estimation, each $\\widehat{Q}^i\_c$ estimates the cumulative cost in parallel and independently. > Should the expectation in (3) be replaced by an empirical mean (i.e., $\\frac{1}{E}\\sum\_{i=1}^{E}$)? Yes, the reviewer is right. It would be more accurate to use an empirical mean, and we have modified this definition in our revision. ## Q2: About the statement that UCB "encourages the policy to choose actions with low uncertainty in cost estimation." Yes, the reviewer is right. Actions favored by the conservative policy optimization are those not only yield low uncertainty in cost estimation but also result in high reward and constraint satisfaction. We have modified the corresponding expression in our revision (Section 3.1). ## Q3: About the notation for the hyperparameter $c$ in ALM. Thanks for the suggestion. We will change the notation to avoid confusion in our future revision, but we have to temporarily maintain this notation in the current revision (available in this webpage) since some questions from other reviewers are related to this notation.

We thank the reviewer for all the valuable comments. Please refer to the point-to-point responses below. ## W1: About the form of the upper confidence bound. Thank you very much for pointing out this inaccuracy. It is true that in linear MDPs where the transition kernel and reward function are assumed to be linear w.r.t. the state-action representation, the upper confidence bound has a specific form that does not correspond to Eq. (3) in our paper. We have corrected this part in our revised version (Section 3.1). ## W2: About how the bootstrapped value ensemble is constructed and why we chose this form of uncertainty estimation. Specifically, we maintain $E$ bootstrapped $Q\_c$ networks, i.e., $\\{\widehat{Q}^i\_c \\}\_{i=1}^E$, which share the same architecture and only differ by the initial weights. From the Bayesian perspective, this ensemble forms an estimation of the posterior distribution of $Q\_c$, with the standard deviation quantifying the epistemic uncertainty induced by a lack of sufficient data. Despite its simplicity, this technique is widely adopted and is proved to be extremely effective in many previous works. We have added the above description into our revised version (Section 3.1). ## W3: More space in the main paper for the theory behind ALM. Thanks for the suggestion. We have re-stated the monotonic energy decrease as a proposition and added a proof sketch of it in the main paper to provide an overall insight of the theory (Section 3.2). ## Q: "How many Q networks did you use in the ensemble and how did you choose this hyperparameter?" As described in Appendix D.3, we set the ensemble size to 4 for MuJoCo tasks and 6 for Safety-Gym tasks. In theory, increasing the number of $Q\_c$ networks will improve the reliability of uncertainty quantification, but at the cost of increasing the computational burden. We choose the two hyperparameters because they strike a good balance between computational complexity and performance (we are running an additional ablation study for the ensemble size, and we will update our revision once we have completed experiments). However, note that we did not fine tune this hyperparameter for each task, so it is likely that a better choice of the ensemble size might exist for many tasks.