One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

Summary

This paper proposes a new model-based offline RL algorithm that learns a policy which is risk-averse (wrt aleatoric uncertainty measured by some dynamic risk measure) and pessimistic (wrt epistemic uncertainty from modeling). First, the algorithm learns a posterior distribution over MDP transitions given the dataset, using an ensemble of neural nets. Then, the algorithm samples successor states from the worst-case perturbation of the learned model, effectively modeling the Bellman equation for dynamic risk.

Strengths

The method is clean and the high-level ideas are clearly explained. Extensive experiments are also encouraging and paper is pretty well written.

Weaknesses

1. My understanding is that this method is targeting the dynamic risk of the coherent risk measure with envelope B_p. However, all of the evaluations, as well as most of the paper, seem to suggest that 1R2R is a good algorithm for risk-neutral and static risk objectives (which the authors indeed show in experiments). It's not clear to me why optimizing for the dynamic risk should result in good performance for risk-neutral or static risk objectives, and so the method seems more hand-wavy. For example, if we assume that model learning succeeds, can we prove any PAC bounds wrt risk-neutral or static risk objectives? 2. It would be interesting to have some ablations on the success of 1R2R: are the improvements in performance mostly due to pessimism wrt aleatoric uncertainty, or epistemic uncertainty, or simply model-based offline RL? Also, please see Questions section.

Questions

1. Why is Line 184 labeled "Problem 1"? I don't see a Problem 2. 2. Why do you take on the Bayesian perspective for learning \bar T? Another way of learning the model is the MLE, ie train a single neural net to maximize log likelihood of successor state, so I'm wondering why you choose to learn P(T|D) and then derive \bar T from that? 3. How is the adversarial perturbation actually computed in practice (Line 10)? Does you need to perform a two-stage optimization procedure? 4. How are the hyperparameters of the method selected, and how many online samples were used for hyperparameter tuning? (In theory, papers about offline RL should only be using offline samples for algorithm design, but in practice, this is almost always violated. So, it would be nice to report how many online samples were used, especially given that this paper introduces many more hyperparameters (ie Table 4)). 5. Have you compared with ATAC (Adversarially Trained Actor Critic for Offline Reinforcement Learning), which is one of the SOTA offline RL methods?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Please see weaknesses/questions.

Summary

> Our concerns are addressed by the authors. Thanks for your effort. We will upgrade the rating to weak accept. This paper introduces a model-based risk-averse algorithm that utilizes risk aversion as a mechanism to jointly address the distributional shift problem and risk-related decision-making problems in risk-sensitive offline RL. The authors employ a risk-averse risk measure to simultaneously perturb the belief distribution and transition functions in an adversarial manner, enabling risk aversion towards both epistemic uncertainty and aleatoric uncertainty. Risk aversion to epistemic uncertainty reduces transition probabilities to state-action pairs that are out-of-distribution, and risk aversion to aleatoric uncertainty helps avoid actions that are inherently risky. The authors conduct experiments in both deterministic and stochastic environments, confirming the superior performance of the 1R2R algorithm.

Strengths

1. The paper is well written. The original contributions are highlighted clearly. 2. The paper provides a concise and understandable introduction to the background and related work. It is highly reader-friendly, ensuring ease of comprehension for the intended readers. 3. The paper demonstrates a well-organized structure, and the approach of utilizing risk aversion to address the distributional shift problem is innovative and holds theoretical viability.

Weaknesses

1. While adopting a joint risk-aversion mechanism for both epistemic uncertainty and aleatoric uncertainty is simple and efficient, I think this approach is less flexible compared to previous methods which can address distributional shift and risk-sensitivity separately. 2. The design of ablation experiment merely removes the risk-sensitivity, which I think to be overly simplistic. Ablation Experiments part requires appropriate expansion, such as comparing the computational costs with the approaches using distributional value functions, and comparing the performance between using dynamic risk measures and using static risk measures. 3. I think 1R2R does not have a significant advantage over other baselines (i.e., RAMBO). Can the authors discuss more in-depth regarding this issue?

Questions

1. How does the argmin is performed in Line 10 of Algorithm 1? 2. In my opinion, instead of characterizing this approach as risk-aversion methods, I think it resembles a conservative value update method. By introducing adversarial perturbations to the transition function, it increases the likelihood of transitioning to low-value successor states. Can the author clarify this? 3. For the experimental results of D4RL MuJoCo, how can you validate that the performance improvement of the 1R2R algorithm is indeed due to addressing the 'distributional shift' problem? 4. In Line 139, the equation is written as ($Z_{\mathrm{MDP}}=\sum_{t=0}^{\infty} \gamma R\left(s_t, a_t\right)$). Shouldn’t this be ($Z_{\mathrm{MDP}}=\sum_{t=0}^{\infty} \gamma^{t} R\left(s_t, a_t\right)$)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

This approach is less flexible compared to the previous methods which can address distributional shift and risk-sensitivity separately.

Summary

This paper considers the problem of offline reinforcement learning for risk-averse decision-making with the distributional shift. The core insight of this paper is to incorporate epistemic uncertainty (from the distributional shift) and aleatoric uncertainty (from usual statistical errors) together and develop a way to simply penalize high uncertainty, which can handle both problems simultaneously.

Strengths

I think the biggest strength of this paper is the novel idea of combining both epistemic uncertainties from the distributional shift and aleatoric uncertainty together and using a simple risk-aversion algorithm to handle both problems. I think this key idea can be applied not only to offline reinforcement learning but also to other areas where the distributional shifts are key, e.g., predictions and causal inference problems.

Weaknesses

I do not think there is any obvious weakness in this paper. I only have some suggestions below.

Questions

**How to choose the distribution over MDPs to capture the epistemic uncertainty?** One of the key ideas in this paper is to use the distribution over MDPs to represent the epistemic uncertainty over the real environment. As far as I understand, on page 7 (Implementation Details), the authors suggest using an ensemble of neural networks to estimate different distributions over the MDPs. It seems to me that this choice of model classes is actually the fundamentally important part of capturing epistemic uncertainty. For example, if we include a large number of different neural networks (e.g., more than 100), the epistemic uncertainty is guaranteed to be larger than cases when we include a smaller number of neural networks (e.g., 5). **But, without an explicit model or assumption about the distributional shifts, how can researchers choose what models to estimate MDPs for approximating the epistemic uncertainty?** **In Section 5 (Experiments), how did the authors choose the model class for capturing the epistemic uncertainty?** I think this is an important question as there is an inherent tradeoff --- if we include more models, we can be more robust to a wide range of potential distributional shifts, but if we include too many models, we might be too conservative. **Is there any theoretical guidance about how to approximate the epistemic uncertainty?**

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper clarifies its limitation clearly.

Summary

One Risk to Rule Them All (1R2R) is an offline-RL method that seeks to reduce aleatoric and epistemic uncertainty. Their method is simple; the authors introduce the notion of risk in the bellman update by adding learned adversarial perturbation to the learned transition dynamics models in the model-based setting. This is different from prior risk-based methods which adversarially perturbate the value function as opposed to the TD function.

Strengths

- The narrative/flow is clear; this work focuses on minimizing both aleatoric and epistemic uncertainty, both of which hamper offline RL agent performance. - Proposed methodology is simple, just add a learned adversarial perturbation to TD model. - Figure 1 is an excellent figure that illustrates that the learned risk-aware value function penalizes both uncertainties. Yet, it is difficult to absorb/fully understand, and it might be better to split it into two subfigures. For example, highlight regions with different colors outside of dataset and regions where TD uncertainty is high. (It is also nice to have to compare the value function to CQL's too, to show that CQL might be aleatoric unaware) - Evaluation is thorough and evaluated over difficult environments with large continuous action spaces.

Weaknesses

- The intro was not clear. At the end, it says that epistemic uncertainty is avoided through model-ensemble variance. However, the analogy to aleatoric uncertainty was not made clear. It might be better to straight-up state that risk-averse RL applied in the offline setting reduces both uncertainties for reasons X and Y. - Authors claims method is simpler than prior work due to only considering risk; however, the risk incorporates model-based approaches, which is much more complicated and much harder to work in practice. - Why does risk-averse RL reduce aleatoric uncertainty? (good to show math in appendix) It seems to be a miraculous cure to a problem. - Methodologically, this paper is not novel; it seems more like A + B, where A = model-based ensemble in MOPO and B = risk-averse RL objective in CvAR, applied in the offline RL setting. The real novelty seems to stem from identifying applicability of risk in offline RL. - The Risk background is difficult to understand; suggest removing it and immediately diving into the Bellman risk-sensitive representation. - It'd be nice to have a figure illustrating the final algorithm architecture and well as typical reward x training iteration graphs in the evaluation. - Where is the evaluation against MOPO/COMBO, another model-based offline RL method? It's be worthwhile to show that this work is the SOTA model-based offline RL approach. - Suggesting removing one of the evaluation environments and focusing more on ablation and analysis. - Why does 1R2R flounder on expert dataset (medium expert)? - Another way to avoid such uncertainty is the field of safe RL, which seeks to minimize constraint violations while maximizing environmental reward (Recovery RL). There is a lot of prior work in this field and is somehow obfuscated in this paper. -Nit: There are several typos, including in Algorithm 1 line 9, and Section 3 Static and Dynamic Risk MDP formulation.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Unrelated, it would be nice to have a model-free version, as I'd generally avoid model-based approaches (as does industry). For example, if you add the adversarial perturbation to the distributional value function (instead of the learned TD function), despite prior work. - I'd upgrade this paper to strong accept (as possibly higher) if it explains WHY adding risk-aversion reduces epistemic and aleotoric uncertainty, as this is a very beautiful insight. Bonus if there is math.