Many real-world domains require safe decision making in uncertain environments. In this work, we introduce a deep reinforcement learning framework for approaching this important problem. We consider a distribution over transition models, and apply a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures. We provide robustness guarantees for this framework by showing it is equivalent to a specific class of distributionally robust safe reinforcement learning problems. Unlike existing approaches to robustness in deep reinforcement learning, however, our formulation does not involve minimax optimization. This leads to an efficient, model-free implementation of our approach that only requires standard data collection from a single training environment. In experiments on continuous control tasks with safety constraints, we demonstrate that our framework produces robust performance and safety at deployment time across a range of perturbed test environments.
Paper
Similar papers
Peer review
Summary
The authors consider a distribution over transition models and tackle the safe RL problem by applying a risk-averse perspective towards model uncertainty through coherent distortion risk measures. The proposed formulation can ease the burden of solving a min-max problem, which is often encountered in many worst-case safe RL algorithms. The authors also theoretically show that their formulation is equivalent to a specific class of distributionally robust safe RL problems.
Strengths
- This work proposes a new formulation of safe RL, by considering a distribution of transition models and applying the distortion risk measure toward the model uncertainty, which circumvents the burden of solving min-max problems. - This work theoretically proposes that the reformulated problem is equivalent to a specific class of distributionally robust safe RL problems.
Weaknesses
- It is better to include the proof of Lemma 2 in Appendix B.2 for self-contained. Also, explaining the results of Lines 225-226 is better. - It seems that the performance of the proposed method can not surpass existing methods, especially adversarial RL. Also, the experiments presented are not sufficient. - Although applying the distortion risk measure $\rho$ seems to be promising, I'm wondering about the necessity of doing so since it introduces more complex computation processes and seems to contribute little to the experiment results.
Questions
- What's the choice of the function $g$ (line 224-225) in your experiments? - In lines 229-230, is there only one Q function for calculating all sampled transitions? - In Table 1, what’s the meaning of the bold numbers? I think you should highlight the best results instead of yours. - Also, please explain the necessity and merits of adopting the formulation proposed in this paper.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
NA
Are you satisfied by the answers?
Dear reviewer, Would you please indicate whether the authors' response is satisfactory for you? If not, please engage with the authors, so we can get a better assessment of this work. Thank you, Area Chair
I am following up on this, especially given that your review is currently the most critical one. Do you find the authors response convincing or do you have a serious issue against this paper? Thank you, Area chair
I would like to thank the authors for their time to provide detailed responses to my concerns, and thus I'll increase my score to 5. The main advantage of adopting the proposed RAMU framework as claimed by the authors is its robustness guarantees and avoiding solving the min-max problem. However, I'm concerned by its actual contribution to solving practical tasks, since the empirical improvement is not significant to me as shown in Table 1. Also, I think you should bold the best method instead of your method in the experiment section.
Thank you for your response! In addition to the advantages that you have mentioned (main benefits #1 and #2 in our rebuttal), we also want to emphasize that our framework can be applied across a broad range of practical tasks because it requires minimal assumptions on the training process (see main benefit #3 in our rebuttal). Our approach only requires standard data collection from a single training environment, so it can even be applied in settings that require real-world data collection for training. This is not true of the popular baselines that we compare against in our experiments. Domain randomization requires detailed simulator access during training, and adversarial RL requires potentially dangerous adversarial interventions throughout training. These assumptions can be undesirable or impractical in many real-world settings. Our RAMU framework achieves similar or improved performance compared to these popular baselines, without requiring these additional assumptions on the training process (see item #3 under the key takeaways bullet in our rebuttal).
Summary
This paper proposes a methodology for distributionally robust RL via the use of risk measures and leveraging risk (Fenchel) duality, dealing with what they call model uncertainty. The paper introduces the RAMU Q function and Bellman operators respectively, which are based on modifying the standard risk-neutral operators by inducing risk-awareness over the choice of the transition model. The RAMU Q function and respective Bellman operators exhibit nice properties, due to the class of risk measures considered (distortion risk measures). Then, the paper describes a model-free implementation of the proposed approach with a single training environment, leveraging interesting results for statistical estimation of risk functionals (specifically distortion risk measures). Lastly, the proposed approach is verified on a number of numerical benchmarks.
Strengths
Overall, I think this is a paper with potential, and it was a pleasure reading it. The RAMU framework is indeed interesting and seems to be effective, although I have some concerns (see below). The estimation of the Bellman operators in Section 6 within the RAMU framework can indeed be computed efficiently, as the authors point out, which is an advantage of the approach. The application of the results of Jones and Zitikis is quite interesting. The experimental section is quite detailed and the comparisons with existing methods is well-thought, especially with a large number of similar alternative approaches. Also the experiments bring out the effect and efficacy of risk-averse RL well.
Weaknesses
Risk duality and equivalence to minimax (distributionally robust) optimization (under certain assumptions) is very well known and established (in essence it is just Fenchel duality). Therefore, I do not think it can be claimed as novelty or contribution of this paper by itself (e.g., in lines 5 to 7 in the abstract). I would suggest that the authors rephrase to highlight more specifically their contributions. In the discussion about risk measures, I do not see any explanation on *why* one should choose to work with distortion risk measures or coherent risk measures, and why the axioms proposed by Majumdar and Pavone are appropriate. Axioms are subjective, so in principle there is no universal reason to use one set of axioms over another set of axioms. Of course distortion risk measures exhibit amazing analytical properties and probably this is among the main reasons justifying the authors chosing this class, but the discussion and point needs to be made. The RAMU cost function as defined in line 156 is introduced somewhat arbitrarily, in the sense that it is unclear if this is somewhat related to a base problem such as (2). While I understand the motivation and indeed it *seems* to make sense, usually one would start from such a base problem, possibly risk-averse in a certain way that makes intuitive and operational sense, and then build their way towards Q factors and Bellman optimality conditions. Theorem 1 follows directly from the application of risk duality. I am not sure if this result "deserves" to be a theorem. I would think that a stating as a proposition would be more suiting. As a general comment, the theory in this paper is somewhat limited, however I have enjoyed reading the paper still. The discussion under "Generative distribution of transition models", line 236 onwards is quite convoluted and involved without particular reason in my opinion. While I finally understood what happens (I think!), I believe that the authors should make an effort to write this part in a much clearer manner.
Questions
What is a "distribution over transition models" in lines 41-42? This is very ambiguous at this point. Also, related in Lines 72-74: Defining this product is somewhat obscure. For instance, does this make sense when S and A are uncountable sets (or you implicitly assume finite states and actions throughout)?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Yes
Are you satisfied by the answers?
Dear reviewer, Would you please indicate whether the authors' response is satisfactory for you? If not, please engage with the authors, so we can get a better assessment of this work. Thank you, Area Chair
I would like to thank the authors for taking the the time and providing detailed responses to my comments. I am mostly satisfied with the answers provided, except possibly with the point regarding using coherent or risk measures in [W2]. While the first bullet is solid, since such classes of risk measures have exceptional properties, the second bullet is highly subjective (for instance, there is a huge debate in the finance literature about the axioms comprising the class of coherent risk measures, and there is absolutely no consensus. It happens that such nice classes of risk measures are used in academic research, because of analytical tractability, which again is a very solid reason why to use them). For now, I will keep my score unchanged, but I am leaning favorably towards this paper.
Summary
This paper presents a Temporal Difference (TD) learning method for addressing the ``Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning'' problem. Specifically, the authors consider a Constrained Markov Decision Process (CMDP) combined with Bayesian uncertainty sets. They are trying to attain a policy which is both robust and safe; Meaning that it optimizes a nested risk measure of discounted return, while satisfying certain guarantees over discounted cost. The proposed method is implemented and evaluated by comparing its performance against adversarial RL, Safe RL, and domain randomization approaches across multiple environments.
Strengths
- This paper addresses a noteworthy problem. As a general matter, I believe that risk-aware policy selection methods for Bayesian MDP, as also studied in [1, 2, 3] is an interesting and open area of research. - Even though the problem is notation-heavy, the paper is well presented and maintains a high level of readability. - The authors have proven their implementation's effectiveness across five different tasks. Furthermore, the authors' utilization of a large number of training samples adds credibility to the algorithm's performance evaluation. [1] Giorgio Angelotti, Nicolas Drougard, & Caroline Ponzoni Carvalho Chanel. (2023). An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes. arXiv preprint arXiv:2105.13431. [2] Lobo, E. A., Ghavamzadeh, M., & Petrik, M. (2020). Soft-robust algorithms for batch reinforcement learning. arXiv preprint arXiv:2011.14495. [3] Petrik, M., & Russel, R. H. (2019). Beyond confidence regions: Tight Bayesian ambiguity sets for robust MDPs. Advances in neural information processing systems, 32.
Weaknesses
- **Theoretical results**: The primary focus of this paper is the implementation and experimental evaluation; there are limited theoretical results in this paper. - **Comparisons**: In the ``strengths'' section of my review, I mentioned three risk-aware policy selection methods for Bayesian MDP methods that I am aware of [1, 2, 3]. These methods have similar objectives, albeit having two key differences (i) Instead of CMDPs the objective is for normal MDP and does not require safety guarantees (ii) the risk measure is (might be) applied over entire stochasticy $\mu$ rather than recursively over each $\mu_{s, a}$. As both algorithms use samples of models to have a Monto Carlo estimation of the risk measure, the $\theta$ update in Algorithm 1 is particularly similar to the $\operatorname{RiskEvaluation}$ operation introduced by [1]. - Firstly, I would suggest that the authors include these papers in the ``uncertainty in reinforcement learning'' part of the related works. - Secondly, it would have been more informative if the paper compared its proposed method to one of these existing methods, instead of focusing on comparisons with adversarial RL and domain randomization. - **Presentation**: Theorem 1 requires a more concrete and explicit formulation. $\zeta_{a, s}$ is introduced as ``depending on $\rho^+$ and $\rho$''; without properly defining the dual presentation of coherent risk measures beforehand. This lack of clarity makes it very challenging for non-expert readers to understand the paper. In conclusion, while the paper proposes an interesting problem and provides valuable experimental results, there is a lot of room for improvement. Thus, I rate this paper as a borderline accept.
Questions
- **The Objective**: It would be interesting to observe the performance of the proposed algorithm by varying the learning rates for $\theta_r$ and $\theta_c$ and exploring different risk measures for $\rho^+$ and $\rho$. For instance imagine we set $\rho^+$ as $CVaR_{\alpha_r}^+$, and $\rho$ as $CVaR_{\alpha_c}$. As $\alpha_c$ decreases, the algorithm becomes more strict on violating safety conditions, and as $\alpha_r$ decreases, the algorithm becomes more robust. I am curious to see the differences in experimental outcomes under such settings. - **Evaluation**: In the experiments, different algorithms are being compared based on the safety percentage, and total rewards. I would like to see their comparison based on (an estimated version of) the Lagrangian form of the actual objective (4) (while fixing the set of hyperparameters for all methods). Since the proposed algorithm directly aims to optimize this objective, it is reasonable to expect it to outperform other methods in terms of this specific evaluation metric. The conduct of such an evaluation can serve as a sanity check.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
no limitations
Are you satisfied by the answers?
Dear reviewer, Would you please indicate whether the authors' response is satisfactory for you? If not, please engage with the authors, so we can get a better assessment of this work. Thank you, Area Chair
Summary
The paper introduces a deep reinforcement learning framework for safe decision-making in uncertain environments. The authors propose a risk-averse approach towards model uncertainty using coherent distortion risk measures. They provide robustness guarantees for the framework by showing its equivalence to a distributionally robust safe reinforcement learning problem. The framework is efficient and model-free, utilizing standard data collection from a single training environment. Experiments on continuous control tasks with safety constraints demonstrate the framework's robust and safe performance across a range of perturbed test environments.
Strengths
[+] The paper is logically clear and well-written, making it easy to follow. [+] The paper starts from theory, first defining a new Q-function and a new Bellman operator, then proving the equivalence between the new Bellman operator and distributionally robust Bellman operators with respective ambiguity sets, thereby verifying the rationality of the definition and the contraction property of the new Bellman operator. This provides a theoretical basis for the algorithm design. [?] What is the relationship between safety and robustness? The paper theoretically proves that the newly proposed RAMU Bellman operator has robustness guarantees. However, it seems not to mention much about its relationship with Constrained Markov Decision Process (CMDP) and how to ensure that the point converged to by the RAMU Bellman operator is safe. [?] Related to the previous question, why consider both safety and robustness here? It seems that considering both safety and robustness is crucial for the Theorem 1. Is it necessary to clarify this point more explicitly in the abstract? For example, the current abstract's first sentence emphasizes the importance of safety, the line 5 separately emphasizes robustness guarantees, and the line 7 re-emphasizes robustness, which can be somewhat confusing.
Weaknesses
see above
Questions
see above
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
see above
Are you satisfied by the answers?
Dear reviewer, Would you please indicate whether the authors' response is satisfactory for you? If not, please engage with the authors, so we can get a better assessment of this work. Thank you, Area Chair
Summary
The paper introduces the Risk-Averse Model Uncertainty (RAMU) framework for safe reinforcement learning in uncertain environments. RAMU incorporates a distribution of transition models and applies a risk-averse perspective using coherent distortion risk measures. The framework offers an efficient, model-free implementation through one single training environment. Experimental results demonstrate the framework's ability to produce robust, safe performance in perturbed test environments. Unlike existing distributional robust (DR) approaches, RAMU eliminates the need for minimax optimization.
Strengths
- The proposed perturbation function models the distribution of environment transition models, providing a foundation for practical implementation of distributionally robust algorithms. - Compared to existing methods, the RAMU framework is implemented efficiently. It avoids complex minimax optimization, which is the major obstacle to applying Distributional Robustness methods to DRL.
Weaknesses
- The problem addressed in this paper is the handling of model uncertainty in safe reinforcement learning (RL) scenarios. However, it appears that the issue of model uncertainty in safe RL is similar to the problem in standard RL. Since the proposed RAMU method is not specifically designed for safe RL, it would be more convincing if it were compared to existing distributionally robust methods such as those discussed in [1] and [2]. - The formulation of the distributionally robust safety problem in the article lacks clarity. The problem definition starts with directly modifying the Q function to the DR Q function in Eq4, which is a shortcut approach. It would have been more appropriate to first define the problem and then derive the suitable form of the Q function. Although solving the Eq4 definition is straightforward by plugging in the safety RL method with DR RL, it is important to note that the worst-case transitions associated with the reward and cost (the $\beta$ in Eq5 and Eq6) are not the same. However, in reality, there is only one transition, which makes the proposed method more conservative in practice. - The transition function f(s, s') mentioned in line 257 may only be effective for tasks involving robot control, where dynamics follow linear patterns and such perturbations are effective. For tasks involving image inputs, it remains unclear what kind of function f should be used.
Questions
- I understand that the contribution of the paper lies in proposing a distributionally robust (DR) safe reinforcement learning method that does not require a minmax operation. However, given that there is no direct connection between DR algorithm design and safety, it is essential to compare it with existing DR methods such as [1] and [2]. - The proposed algorithms does have similarities to policy smoothing algorithms, as mentioned in [3] and [4]. It is plausible to consider that combining policy smoothing with adversarial RL could potentially yield better results compared to the RAMU framework. In other words, exploring the combination of observational robustness and action robustness may provide improvements over predefined model robustness? Ref. [1] Robust Reinforcement Learning using Offline Data [2] Distributionally Robust Q-Learning [3] Deep Reinforcement Learning with Robust and Smooth Policy [4] Policy Smoothing for Provably Robust reinforcement learning.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
As mentioned in the Conclusion section of the paper, the choice of the model distribution µ and risk measure ρ in the RAMU framework is user-defined, and RAMU framework only addresses robustness with respect to model uncertainty and safety defined by expected total cost constraints. Crruently, this paper only consider the dynamics of model transition follow linear patterns, which may not to be suitable for tasks that involve image inputs.
I believe the core contribution of this article lies in providing a simple and effective robust method to address model uncertainty in MDPs. However, the method itself does not directly address the safety aspect or the CMDP problem. The article needs to clarify the connection between safety and robustness. While the article only addresses the robustness issue, the overall background of the article is strongly tied to the concept of safety, which seems strange to me. Additionally, there is significant room for improvement in the writing of this article. However, I would like to increase the score to 5 fot the good idea of distribution over transition models.
Thank you for your response! We agree that the core contribution of our paper is a simple and effective method for incorporating robustness to model uncertainty in a deep RL setting. We consider safe RL modeled by a Constrained MDP (CMDP) as our starting point, as safety is often a prerequisite for real-world decision making applications. We incorporate robustness to model uncertainty in the safe RL setting by applying our RAMU framework to both the objective (rewards) and safety constraint (costs) of a CMDP. As demonstrated in our experimental results, this leads to policies that achieve robust performance *and* robust safety across test environments (i.e., robustness in both components of a CMDP). If safety is not relevant in an application, our framework could also be applied to provide robustness in a standard MDP, which would result in a special case of the update in (4) without the safety constraint. In the updated version of our paper, we will clarify how our problem formulation leads to robust performance *and* robust safety constraint satisfaction in a CMDP. Thank you for helping us to improve our paper.
Response
The authors have responded to most of my empirical concerns. Therefore, I will increase my rating from 5 to 6. Regarding Q1: First it was only a suggestion, but comparison between RAMU (Wang 0.75) and RAMU (Expectation) does not cover what I mentioned there. It is only for 2 risk measures, and also does not investigate different $\alpha_c$ and $\alpha_r$.
Decision
Accept (poster)