Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning
In domains such as finance, healthcare, and robotics, managing worst-case scenarios is critical, as failure to do so can lead to catastrophic outcomes. Distributional Reinforcement Learning (DRL) provides a natural framework to incorporate risk sensitivity into decision-making processes. However, existing approaches face two key limitations: (1) the use of fixed risk measures at each decision step often results in overly conservative policies, and (2) the interpretation and theoretical properties of the learned policies remain unclear. While optimizing a static risk measure addresses these issues, its use in the DRL framework has been limited to the simple static CVaR risk measure. In this paper, we present a novel DRL algorithm with convergence guarantees that optimizes for a broader class of static Spectral Risk Measures (SRM). Additionally, we provide a clear interpretation of the learned policy by leveraging the distribution of returns in DRL and the decomposition of static coherent risk measures. Extensive experiments demonstrate that our model learns policies aligned with the SRM objective, and outperforms existing risk-neutral and risk-sensitive DRL models in various settings.
Paper
References (46)
Scroll for more · 34 remaining
Similar papers
Peer review
Summary
This paper aims to extend the work of Bauerle and Glauner [1] on static spectral risk measures (convex combinations of CVaR) in Markov Decision Processes (MDPs) to the context of distributional reinforcement learning (RL). Sections 4 and Appendices A and B reformulate the approach from [1] using distributional value functions. Theorem 1 provides a bound on the performance of the policy derived from greedy action selection over an augmented state space (x, s, c). Algorithm 2 proposes the TD error computation for distributional value function, contrasting with methods like QR-DQN and IQN, by directly addressing for static spectral risk measures. Theorem 2 extends the decomposition from coherent risk measures [2] to a broader class of spectral risks, increasing the generalizability of the approach to a wider array of risk-sensitive applications. Finally, the experiments validate the proposed algorithm, offering evidence of its efficacy and robustness within this distributional risk-sensitive framework. References: [1] Nicole Bauerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathematical Methods of Operations Research, 94(1):35–69, 2021. [2] Georg Ch. Pflug and Alois Pichler. Time-Consistent Decisions and Temporal Decomposition of Coherent Risk Functional. Mathematics of Operations Research, 41(2):682–699, 2016.
Strengths
The strengths of this paper lie in its deep understanding of the current state of research in risk-averse reinforcement learning (RL) and the limitations of recent work in risk-averse distributional RL (DRL). Specifically, the paper identifies key challenges: (1) dynamic and fixed risk DRL approaches often lack interpretability, and (2) the dual representation of coherent risk measures encounters issues during policy optimization. The authors’ solid grasp of the mathematical foundations behind risk measures enables them to combine and present a concise and theoretically sound introduction.
Weaknesses
Despite the authors’ strong grasp of the limitations in risk-averse distributional RL research, this paper has several notable weaknesses: (I) The primary weakness lies in the limited originality of the contributions. Much of the content in Theorems and Lemmas in Appendix A, B, and Section 4 is directly adapted from [1], with only minor modifications to distribution value function representation. This reliance raises questions about the novelty and depth of the contributions, compared to [1]. (II) While the Introduction and Preliminaries are well-articulated, Sections 4 and 5 suffer from clarity issues. Section 5 appears disconnected from previous sections. Spectral risk measures can be represented as convex combinations of CVaR, Theorem 2 in Section 5 leverages this property to extend the dual decomposition from coherent risk to general spectral risk measures. However this dual decomposition is unrelated to algorithm 1 and 2 in the earlier sections. (III) Did not discuss the main limitation of extending static spectral risk MDP to model-free/distributional RL. Specifically: - (a) There is no analysis of the convergence properties of the algorithm 2 TD loss update (a static risk variant similar to [2]). - (b) Missing analysis for approximation errors and guarantees arising from quantile discretization $\tau_i$. - (c) Contraction analysis is missing, considering that the minimizer of the Huber loss may not be unique (see [3]). References: [1] Nicole Bauerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathematical Methods of Operations Research, 94(1):35–69, 2021. [2] Shen, Yun, et al. "Risk-sensitive reinforcement learning." Neural computation 26.7 (2014): 1298-1328. [3] Rowland, Mark, et al. "An analysis of quantile temporal-difference learning." (2023).
Questions
(I) What are the contribution of section 4 compared to [1]? Please clearly indicate which methods are re-written from [1] and what is new in this paper, in the main body and also appendix A and B. (II) Please address the limitation pointed out in Weaknesses section (III):Did not discuss the main limitation of extending static spectral risk MDP to model-free/distributional RL. Specifically: - (a) There is no analysis of the convergence properties of the algorithm 2 TD loss update (a static risk variant similar to [2]). - (b) Missing analysis for approximation errors and guarantees arising from quantile discretization $\tau_i$. - (c) Contraction analysis is missing, considering that the minimizer of the Huber loss may not be unique (see [3]). References: [1] Nicole Bauerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathematical Methods of Operations Research, 94(1):35–69, 2021.
Rating
3
Confidence
4
Soundness
3
Presentation
2
Contribution
2
Thank you for your valuable feedback. We appreciate your recognition of our work's strength. We address the highlighted weaknesses and questions below. > (I) What are the contributions of section 4 compared to [1]? Please clearly indicate which methods are re-written from [1] and what is new in this paper, in the main body and also appendix A and B. Our work builds on [1] in several important ways, and while we share some common ideas, none of our analysis is a direct rewrite of [1]. For example, [1] does not include the convergence analysis that we present in Theorem 1 for the inner optimization step. It is also important to highlight that our approach extends beyond simply modifying the value function in [1] to a distributional form. In our method, the distributional value function learns the return distribution for each state-action pair, from which the Q-value is derived using the function $h$. In contrast, the Q-values in [1] directly capture the risk, and the Q-value function there satisfies the Bellman equation. The convergence analysis for this approach requires a different set of mathematical tools compared to our work. The separation of the return distribution function and the Q-value function is central to the analysis presented in Section 5 and Theorem 2 of our work. Finally, for the outer optimization, we use a closed-form solution, which contrasts with the global optimization approach presented in Sections 4.3 and 5.1 of [1]. > (a) There is no analysis of the convergence properties of the algorithm 2 TD loss update (a static risk variant similar to [2]). Our convergence analysis in Theorem 1 serves a similar purpose to Theorem 3.2 in [2], namely to show that the value function resulting from the policy iteration process converges to the value function of the optimal policy. However, due to differences in our problem setup, we adopt a distinct approach. For instance, our distributional value function provides the return distribution for each state-action pair, from which Q-values are derived using the function $h$. Additionally, in contrast to our work, [2] does not use a static risk measure. The step-wise risk measure they employ does not require an extended state or bilevel optimization to find the optimal policy, which leads to differences in the analysis compared to our approach. > (b) Missing analysis for approximation errors and guarantees arising from quantile discretization τi. Thank you for your insightful suggestion. We agree that the theoretical analysis of approximation errors arising from quantile discretization in the risk-sensitive setting is valuable and could provide additional depth to our approach. However, given the scope of our current work, we have focused on developing an algorithm for finding the SRM-optimized policy and exploring the interpretability aspects of the optimal policy, as outlined in the paper. That said, we certainly recognize the merit of investigating approximation errors arising from quantile discretization in the risk-sensitive setting, and we see this as an interesting direction for future research. We look forward to exploring this further in subsequent work. > (c) Contraction analysis is missing, considering that the minimizer of the Huber loss may not be unique (see [3]). The analysis of the Quantile Temporal-Difference (QTD) method presented in [3] is also relevant to our work, as they assume a fixed policy throughout their paper and discuss the convergence of the QTD method to a set of fixed points in the context of policy evaluation. **References:** [1] Nicole Bauerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathematical Methods of Operations Research, 94(1):35–69, 2021. [2] Shen, Yun, et al. "Risk-sensitive reinforcement learning." Neural computation 26.7 (2014): 1298-1328. [3] Rowland, Mark, et al. "An analysis of quantile temporal-difference learning." (2023).
We would like to provide additional comments on the strengths and weaknesses you have highlighted. > Strengths: The strengths of this paper lie in its deep understanding of the current state of research in risk-averse reinforcement learning (RL) and the limitations of recent work in risk-averse distributional RL (DRL). Specifically, the paper identifies key challenges: (1) dynamic and fixed risk DRL approaches often lack interpretability, and (2) the dual representation of coherent risk measures encounters issues during policy optimization. The authors’ solid grasp of the mathematical foundations behind risk measures enables them to combine and present a concise and theoretically sound introduction. Thank you for highlighting the strengths of our paper. We would like to clarify that our work does not utilize the dual representation of coherent risk measures. Additionally, the challenges associated with the dual representation of coherent risk measures during policy optimization are not relevant to our study. We hope this addresses any potential misunderstanding, and we are happy to provide further clarification if needed. > Weaknesses: Section 5 appears disconnected from previous sections. Spectral risk measures can be represented as convex combinations of CVaR, Theorem 2 in Section 5 leverages this property to extend the dual decomposition from coherent risk to general spectral risk measures. However this dual decomposition is unrelated to algorithm 1 and 2 in the earlier sections. Thank you for your comment. We believe there may be a slight misunderstanding in the terminology used. The term "dual decomposition" does not accurately describe the property leveraged in Theorem 2. Additionally, spectral risk measures are indeed a subclass of coherent risk measures, so referring to "extending the dual decomposition from coherent risk to general spectral risk measures" may not be appropriate in this context. We would like to clarify that Theorem 2 is not an extension of the decomposition theorem in [1]. Instead, Theorem 2 in our work is the first to demonstrate the application of the decomposition theorem to a risk measure broader than CVaR, specifically SRM in our case. This is achieved through the use of the distributional value function. Section 4 of our work focuses on finding the optimal policy with an SRM objective, while Section 5 explores how the optimal policy can be interpreted at different time steps using Theorem 2. **References:** [1] Georg Ch. Pflug and Alois Pichler. Time-Consistent Decisions and Temporal Decomposition of Coherent Risk Functional. Mathematics of Operations Research, 41(2):682–699, 2016.
Comments to Authors' rebuttal
I have revised my rating based on the authors' response. (I) The novelty of this paper is incremental (extending the value function in [1] to a distributional form) but noteworthy, including a closed-form solution for the outer optimization and their SRM decomposition theorem for policy execution. However, concerns about the paper's completeness persist due to the absence of essential analyses. (a) The author response "Our convergence analysis in Theorem 1 serves a similar purpose to Theorem 3.2 in [2], namely to show that the value function resulting from the policy iteration process converges to the value function of the optimal policy." However, this does not address the convergence of the stochastic approximation (gradient descent) updates in Algorithm 2. There is no formal analysis either in the main paper or the appendix proving that the TD loss updates used in Algorithm 2 converges to their desirable objective. This leaves a key gap in the theoretical justification of the method. (c) The author response "The analysis of the Quantile Temporal-Difference (QTD) method presented in [3] is also relevant to our work, as they assume a fixed policy throughout their paper and discuss the convergence of the QTD method to a set of fixed points in the context of policy evaluation." However, the lack of contraction analysis in their specific SRM setting is concerning, especially since the minimizer of the Huber loss may not be unique (as highlighted in [3]). This raises questions about the reliability of convergence for the proposed method. Without a formal contraction analysis, it is unclear if the proposed loss function contracts sufficiently to guarantee convergence to an optimal solution. Furthermore, the paper would be significantly strengthened if the authors could provide a theoretical analysis that ensures convergence to the optimal policy under specific conditions or assumptions. As highlighted by Reviewer RikA, the numerical experiment performance is not convincing in demonstrating that the proposed algorithm converges to the optimal solution. The authors themselves note that their algorithm is "more prone to converging to suboptimal policies" in their response. This limitation is evident in Table 1, where the CVaR performance for $\alpha \in$ {$ 0.1,0.3,0.5,0.7$} is suboptimal for the QR-SRM model that optimize for these cases. A similar trend is visible in Tables 2 and 3, further underscoring the need for contraction analysis. Without a formal contraction analysis, it is unclear whether the proposed method can reliably converge to the appropriate risk averse optimal solution. [1] Nicole Bauerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathematical Methods of Operations Research, 94(1):35–69, 2021. [2] Shen, Yun, et al. "Risk-sensitive reinforcement learning." Neural computation 26.7 (2014): 1298-1328. [3] Rowland, Mark, et al. "An analysis of quantile temporal-difference learning." (2023).
Thank you for raising your score. We sincerely appreciate the time and effort you have taken to provide your valuable feedback. We address your concerns below. > (a) There is no formal analysis either in the main paper or the appendix proving that the TD loss updates used in Algorithm 2 converges to their desirable objective. > (c) Without a formal contraction analysis, it is unclear if the proposed loss function contracts sufficiently to guarantee convergence to an optimal solution. Furthermore, the paper would be significantly strengthened if the authors could provide a theoretical analysis that ensures convergence to the optimal policy under specific conditions or assumptions. Theorem 1 demonstrates that with sufficiently many iterations, the policies derived from the sequence of return distributions resulting from the distributional optimality operator $\mathcal{T}^{\mathcal{G}\_{l}}$ is close to the optimal policy. The distributional value iteration algorithm in Algorithm 2 combines this operator with a quantile projection operator, which projects the return distribution $\eta_{k,l}$ onto the $N$-quantile representation. Under mild assumptions, such as well-behaved reward distributions, the sequence of return distributions resulting from this combined operator converges to its fixed point. In the next comment, we present the proof, which has also been added to our manuscript in Appendix K. Thank you for bringing this to our attention. > As highlighted by Reviewer RikA, the numerical experiment performance is not convincing in demonstrating that the proposed algorithm converges to the optimal solution. The authors themselves note that their algorithm is "more prone to converging to suboptimal policies" in their response. This limitation is evident in Table 1, where the CVaR performance for alpha 0.1-0.7 is suboptimal for the QR-SRM model that optimizes for these cases. A similar trend is visible in Tables 2 and 3, further underscoring the need for contraction analysis. Without a formal contraction analysis, it is unclear whether the proposed method can reliably converge to the appropriate risk averse optimal solution. As noted in our response to Reviewer RikA, our statement about algorithms being “more prone to converging to suboptimal policies” refers specifically to those optimizing CVaR. This is because CVaR objectives focus solely on the left tail of the return distribution. While CVaR, as a subclass of spectral risk measures, is widely studied due to its simplicity, it has limitations in terms of flexibility and performance. To illustrate this point, the last row of Table 1 shows results using a spectral risk measure that is uniquely enabled by our algorithm. For this comparison, we specifically used our own algorithm (QR-SRM) with the CVaR objective instead of QR-CVaR from [1], ensuring that all of the components of the algorithm except the risk measure remain the same. This highlights that varying the alpha in the CVaR objective has a relatively small impact on the resulting policy. In contrast, switching to a spectral risk measure that assigns a weight to the expected return can yield significantly better results, as seen by comparing the second and last rows of Table 1. This result underscores both the flexibility of our algorithm in selecting objectives and the substantial advantage of spectral risk measures over CVaR. We recognize that this example has caused some confusion among reviewers, and we will update it in the manuscript for more clarity. **References:** [1] Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. The MIT Press, 2023. ISBN 978-0-262-37402-6.
As described in Section 3.3 of the manuscript, we parameterize the return distribution using a quantile representation. Specifically, we employ a quantile projection operator, $\Pi\_Q$, to map any return distribution $\eta$ onto its quantile representation with respect to the 1-Wasserstein distance ($\mathrm{w}\_1$). Therefore, $\Pi\_Q \eta = \hat{\eta} = \frac{1}{N}\sum\_{i=1}^{N} \delta\_{\theta\_i}$ with $\theta\_i=F\_{\eta}^{-1}\left(\hat{\tau}\_i\right), \hat{\tau}\_i=(\tau\_{i-1}+\tau\_i)/2, 1 \leq i \leq N$ corresponds to the solution of the following minimization problem: $ \text{minimize } \mathrm{w}\_1(\eta, \eta^\prime) \text{ subject to } \eta^\prime \in \mathscr{F}\_{Q,N} $ where $\mathscr{F}\_{Q,N}$ is the space of quantile representations with $N$ quantiles. Using this definition, Algorithm 2 can be expressed as iteratively updating $ \hat{\eta}\_{k+1, l} = \Pi\_Q \mathcal{T}^{\mathcal{G}\_{l}} \hat{\eta}\_{k,l}. $ As previously noted, this process is analogous to the iteration in the QR-DQN algorithm, with two key differences: the incorporation of risk-sensitive greedy action selection and the use of an extended state-space. Consequently, we can leverage the steps outlined in [1][Section 7.3] to establish the convergence of $\Pi\_Q \mathcal{T}^{\mathcal{G}\_{l}}$. To begin, we will demonstrate that $\mathcal{T}^{\mathcal{G}\_{l}}$ is a contraction mapping. That is, the sequence of iterates defined by $\eta\_{k+1, l} = \mathcal{T}^{\mathcal{G}\_{l}} \eta\_{k,l}$ converges to $\eta^{\pi^*\_l}$ with respect to the supremum $p$-Wasserstein distance, $\mathrm{\bar{w}}\_p$, for $p \in [1, \infty]$. Here, we assume the existence of a unique optimal policy $\pi^*\_l$. For cases with multiple optimal policies in the risk-neutral setting, we refer to [1][Section 7.5] since extending this result to the risk-sensitive case is straightforward. With this assumption, we leverage the fact that the action gap, $\operatorname{GAP}(Q)$—defined as the smallest difference between the highest-valued and second-highest-valued actions across all states for a given Q-function—is strictly positive. By setting $\bar{\varepsilon} = \operatorname{GAP}(V^{\pi^*\_l}) / 2$ and using Lemma 5, we can see that after $K\_{\bar{\varepsilon}}\in\mathbb{N}$ iterations, where $K\_{\bar{\varepsilon}} := {\ln (\frac{\bar{\varepsilon}}{\phi(0)G\_{\mathrm{MAX}}})}/{\ln (\gamma)} - 1$, the greedy action in state $(x, s, c)$ becomes the optimal action $a^*$, and for any $a \neq a^*$, we have: $ \begin{aligned} V\_{k,l}(x, s, c, a^*) & \geq V^{\pi^*\_l}(x, s, c, a^*) - \bar{\varepsilon} \\\\ & \geq V^{\pi^*\_l}(x, s, c, a) + \operatorname{GAP}(V^{\pi^*\_l}) - \bar{\varepsilon} \\\\ & > V\_{k,l}(x, s, c, a) + \operatorname{GAP}(V^{\pi^*\_l}) - 2\bar{\varepsilon} \\\\ & = V\_{k,l}(x, s, c, a). \end{aligned} $ Thus, after $K\_{\bar{\varepsilon}}$ iterations, the policy induced by the return distribution becomes the optimal policy. Beyond this point, the distributional optimality operator transitions to the distributional Bellman operator for the optimal policy, which is a known $\gamma$-contraction with respect to $\mathrm{\bar{w}}\_p$. Using this result, we conclude that the combined operator $\Pi\_Q \mathcal{T}^{\mathcal{G}\_{l}}$ is a contraction with respect to $\mathrm{\bar{w}}\_{\infty}$, as established in [2][Proposition 2]. **References:** [1] Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. The MIT Press, 2023. [2] Will Dabney, Mark Rowland, Marc Bellemare, and Remi Munos. Distributional Reinforcement Learning With Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 2018
Require clearer explanation to the contradicting result
We have revised our rating based on feedback from other reviewers and the authors' response. Risk-sensitive MDPs are designed for applications in finance, healthcare, and robotics, where avoiding catastrophic outcomes is critical. Poor performance in these domains can lead to serious real-world consequences. The author response agree to us that the algorithms being “more prone to converging to suboptimal policies” refers specifically to those optimizing CVaR. Regarding the question of whether CVaR focuses solely on the left tail, their numerical results presented in the paper appear to contradict proposed theorem claims convergence to optimal or near-optimal solutions. It is also important to note that the SRM optimization approach in the paper relies on Kusuoka’s integral-based formulation of CVaR. If the proposed algorithm fails to correctly optimize a simple CVaR, it raises concerns about its ability to optimize the more complex SRM correctly. The foundational works [1, 2] which this paper builds on, demonstrate near-optimal CVaR optimization with predefined error bounds stemming from discretization. Additionally, the extended conditional decomposition method in [3] also achieves precise computation of CVaR under the extended conditional formulation. These prior works ([1, 2, 3]) compute CVaR accurately, it is not well discuss which portion of the algorithm that is challenging to compute accurately? Moreover, [4] provides an analysis and proposes a CVaR policy gradient method that requires additional $\alpha^{-1}$ sample efficiency. In contrast, the numerical results in this paper contradict the theoretical claims, and the discussion on sample efficiency and possible sub-optimality is absent. This raises significant concerns about the validity and robustness of the proposed approach. References: [1] Bäuerle, Nicole, and Jonathan Ott. "Markov decision processes with average-value-at-risk criteria." Mathematical Methods of Operations Research 74 (2011): 361-379. [2] Bäuerle, Nicole, and Alexander Glauner. "Minimizing spectral risk measures applied to Markov decision processes." Mathematical Methods of Operations Research 94.1 (2021): 35-69. [3] Pflug, Georg Ch, and Alois Pichler. "Time-consistent decisions and temporal decomposition of coherent risk functionals." Mathematics of Operations Research 41.2 (2016): 682-699. [4] Greenberg, Ido, et al. "Efficient risk-averse reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 32639-32652.
> The author response agree to us that the algorithms being “more prone to converging to suboptimal policies” refers specifically to those optimizing CVaR. Regarding the question of whether CVaR focuses solely on the left tail, their numerical results presented in the paper appear to contradict proposed theorem claims convergence to optimal or near-optimal solutions. It is also important to note that the SRM optimization approach in the paper relies on Kusuoka’s integral-based formulation of CVaR. If the proposed algorithm fails to correctly optimize a simple CVaR, it raises concerns about its ability to optimize the more complex SRM correctly. We would like to reiterate that the CVaR objective, in combination with the problem settings and the algorithm used to solve the problem, can make it more prone to converging to suboptimal policies. As mentioned in our response to Reviewer RikA, the specific characteristics of the MDP play a crucial role in determining whether the optimal policy is found, particularly the reward and transition models. In the stochastic cliff walking example, the agent receives a large positive reward (10 points) only upon reaching the goal, while receiving -1 points in cliff positions and 0 points elsewhere. This makes the positive rewards sparse. Additionally, the wind introduces significant stochasticity by moving the agent to nearby positions with a 50% chance. The episode terminates after 50 steps, increasing the likelihood that the agent does not reach the goal. Moreover, we use an off-policy algorithm to approximate the distributional value function. These factors collectively contribute to the CVaR objective's tendency to converge to suboptimal policies in this scenario. In contrast, Table 2 demonstrates that CVaR objectives can successfully find optimal policies under different MDP characteristics. This highlights how the performance of CVaR objectives depends heavily on the properties of the MDP. It is important to note that the shortcomings of the CVaR objective do not contradict our model's ability to optimize more complex SRM objectives. While SRM can be expressed as an integral of CVaR across various levels, these CVaR objectives are not optimized independently. The simultaneous optimization of weighted CVaRs at different levels contributes to the success of our algorithm, as confirmed by our results. For example, optimizing a simple SRM such as $w\mathbb{E} + (1-w)\operatorname{CVaR}_{\alpha}$ leverages information from the entire distribution, not just the left tail. We emphasize that CVaR is a subclass of SRM, and the disadvantages of CVaR do not imply disadvantages for SRM as a whole. > The foundational works [1, 2] which this paper builds on, demonstrate near-optimal CVaR optimization with predefined error bounds stemming from discretization. Additionally, the extended conditional decomposition method in [3] also achieves precise computation of CVaR under the extended conditional formulation. These prior works ([1, 2, 3]) compute CVaR accurately, it is not well discuss which portion of the algorithm that is challenging to compute accurately? Moreover, [4] provides an analysis and proposes a CVaR policy gradient method that requires additional α−1 sample efficiency. [1] and [2] focus on simple MDP settings with known transition and reward models, whereas our work introduces an off-policy model-free algorithm. Furthermore, [3] does not address MDP settings but rather focuses on the decomposition of coherent risk measures, with only a brief discussion of time-consistent decision-making. Similarly, [4] employs policy gradients, making its problem setup entirely different from ours. In contrast, our algorithm is an off-policy model-free approach that relies solely on sampled transitions to estimate the distributional value function. Given these differences in problem setups and methodologies, the works [1, 2, 3, 4] are not directly relevant to our work. > In contrast, the numerical results in this paper contradict the theoretical claims, and the discussion on sample efficiency and possible sub-optimality is absent. This raises significant concerns about the validity and robustness of the proposed approach. Our experimental results demonstrate that our algorithm can discover policies that are not achievable with comparable algorithms. Table 1 highlights that, under the same problem setup, SRM outperforms both risk-neutral and risk-sensitive algorithms using CVaR objectives across various metrics. Table 2 showcases the flexibility of our algorithm in optimizing different objectives, a feature with significant practical value in financial applications. Finally, Table 3 illustrates the strong risk-sensitive performance of our algorithm in more complex problems with larger state spaces.
Summary
This paper introduces a novel distributional reinforcement learning algorithm called QR-SRM, which extends beyond expected return by incorporating spectral risk measures. The authors provide convergence guarantees and enhance interpretability of policies by decomposing coherent risk measures.
Strengths
- The paper addresses a well-motivated problem by proposing an algorithm with asymptotically optimal regret bounds for scenarios involving trajectory-level feedback. - Up to Section 4, the authors clearly outline the motivations, objectives, and proof sketches, making it easier for readers to grasp the core concepts. - The authors included code for reproducibility and conducted experiments across diverse environments, adding practical value and robustness to the study.
Weaknesses
- A minor weakness is the need for improved visualization in the experimental results. The vertical lines are not immediately distinguishable, so adjusting the dash spacing, line thickness, or adding markers would enhance clarity. Additionally, using consistent labels for the same algorithm in the legend would help reduce reader fatigue. - My main concern is that the low performance of the experimental results makes it difficult to be confident that the algorithm was correctly reproduced. According to [1], QR-DQN performs at least 100 points on LunarLander-v2 after 0.1M steps. However, Table 3 of this paper shows much lower scores, suggesting the results may not be fully reproducible. Can the authors clarify? - Although the experiments were conducted in various environments, the aforementioned concerns about reproducibility make fair comparison with other baselines challenging. Reporting performance on some Atari environments, commonly used by algorithms that assume discrete action spaces, would provide a more reliable basis for apple-to-apple comparison. [1] Cho, Taehyun, et al. "Pitfall of optimism: distributional reinforcement learning by randomizing risk criterion." Advances in Neural Information Processing Systems 36 (2024). Typos - Line 197: "Coheret" should be "Coherent" - Line 230: $[h_l (G^{\pi})]$ should be $\mathbb{E}[h_l (G^{\pi})]$ - Line 286: "Output" should be in bold.
Questions
- In Line 232, shouldn't it be $h_{l+1} = \arg \max _h \mathbb{E}[h(G^{\pi^*_l})] + \int_0^1 \hat{h}(\phi(u)) du$? I'm wondering if $ \int_0^1 \hat{h}(\phi(u)) du=0$ is inherently guaranteed within the algorithm, or if there is a condition that ensures this which I may have missed. - In Line 383, $\alpha=0.6$ seems to be maximized at $CVaR_{0.8}$, and $\alpha=0.4$ at $CVaR_{0.6}$. Although the small vertical line intervals may be minor, the lack of alignment with targeted risk levels raises concerns. - In Table 2, aren't the cases with $\alpha=1.0$ essentially QR-DQN?
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Thank you for your valuable feedback. We appreciate your recognition of our work's strength. We address the highlighted weaknesses below. > A minor weakness is the need for improved visualization in the experimental results. The vertical lines are not immediately distinguishable, so adjusting the dash spacing, line thickness, or adding markers would enhance clarity. Additionally, using consistent labels for the same algorithm in the legend would help reduce reader fatigue. We will update the figures to ensure the vertical lines are more distinguishable. Could you kindly point out which legend contains inconsistent labels for the same algorithm so that we can address and correct it? > My main concern is that the low performance of the experimental results makes it difficult to be confident that the algorithm was correctly reproduced. According to [1], QR-DQN performs at least 100 points on LunarLander-v2 after 0.1M steps. However, Table 3 of this paper shows much lower scores, suggesting the results may not be fully reproducible. Can the authors clarify? Thank you for your insightful comment. The observed difference arises because, in all our experiments, including Lunar Lander, we report the discounted return $\sum \gamma^t r_t$ with $\gamma = 0.99$, rather than the raw sum of rewards. Upon reviewing our implementation, we confirm that our QR-DQN implementation can achieve 100+ points, consistent with the results reported in [1]. For your reference, we have provided the results of Table 3 without discounting. The difference in the final return in our case compared to [1] can be attributed to two factors: (1) we enable the wind option in this environment, which was not enabled in [1], and (2) we evaluate our policy using a different seed than the training seed to ensure the environment's stochasticity differs between the training and evaluation phases. | Model | $\mathbb{E}$ | $\operatorname{CVaR}_{0.5}$ | $\operatorname{CVaR}_{0.2}$ | $\operatorname{WSCVaR}^3$ | |-------------------------|----------------------|-------------------------------|-------------------------------|-------------------------------| | QR-SRM($\alpha$=1.0) | 100.21±50.03 | -3.12±52.34 | -60.23±60.36 | 20.63±54.32 | | QR-CVaR($\alpha$=1.0) | -155.14±119.02 | -254.20±118.09 | -328.23±116.45 | -241.09±111.26 | | QR-DQN | **130.55±53.76** | 7.21±90.24 | -134.27±98.50 | -0.82±74.54 | | QR-SRM($\alpha$=0.5) | 93.35±56.35 | -17.45±69.94 | -118.33±70.78 | -11.79±59.28 | | QR-CVaR($\alpha$=0.5) | -59.92±90.79 | -237.18±141.10 | -423.73±276.60 | -239.46±169.77 | | QR-iCVaR($\alpha$=0.5) | 106.79±149.09 | **37.63±161.23** | -48.88±143.50 | 29.89±144.85 | | QR-SRM($\alpha$=0.2) | 84.11±66.03 | -32.44±90.20 | -129.23±75.20 | -21.50±69.62 | | QR-CVaR($\alpha$=0.2) | -10.93±48.62 | -132.66±45.80 | -230.70±100.60 | -119.91±65.58 | | QR-iCVaR($\alpha$=0.2) | 22.53±148.31 | -37.30±133.44 | -99.55±104.63 | -37.94±124.80 | | QR-SRM($\alpha$=0.2,1.0)| 115.95±32.36 | 17.26±43.87 | **-39.01±44.72** | **39.13±36.95** | > Although the experiments were conducted in various environments, the aforementioned concerns about reproducibility make fair comparison with other baselines challenging. Reporting performance on some Atari environments, commonly used by algorithms that assume discrete action spaces, would provide a more reliable basis for apple-to-apple comparison. We hope that we have adequately addressed your previous concern regarding our empirical experiment. Due to the significant computational time required to report the performance of different variations of our algorithm with various seeds on Atari environments—currently taking weeks with our available hardware—we will complete these results for the camera-ready version of our work. **References:** [1] Cho, Taehyun, et al. "Pitfall of optimism: distributional reinforcement learning by randomizing risk criterion." Advances in Neural Information Processing Systems 36 (2024). [2] Alois Pichler. Premiums and reserves, adjusted by distortions. Scandinavian Actuarial Journal, 2015 [3] Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient Risk-Averse Reinforcement Learning. In Advances in Neural Information Processing Systems, 2022
We continue our response here to address the highlighted questions below. > In Line 232, shouldn't it be hl+1=argmaxhE[h(Gπl∗)]+∫01h^(ϕ(u))du? I'm wondering if ∫01h^(ϕ(u))du=0 is inherently guaranteed within the algorithm, or if there is a condition that ensures this which I may have missed. In the proof of Theorem 14 in [1], it is demonstrated that $\\int_0^1 h_{\\phi, Z}(\\phi(u)) du$ is guaranteed to be zero, where $h_{\\phi, Z}$ represents the closed-form solution of the optimization problem. We mention this fact in line 148. We present this proof using our notation here and will include it in the manuscript. Thank you for bringing this to our attention. Using the SRM definition from Equation 4, we have \\begin{aligned} \\operatorname{SRM}\_{\\mu}(Z) = & \\int\_0^1 \\operatorname{CVaR}\_\\alpha(Z) \\mu(\\mathrm{d}\\alpha) \\\\ & \\stackrel{(a)}{=} \\int\_0^1 F\_Z^{-1}(\\alpha)+\\frac{1}{\\alpha}\\mathbb{E}\\left[\\left(Z - F\_Z^{-1}(\\alpha)\\right)^{-} \\right]\\mu(\\mathrm{d}\\alpha) \\\\ & \\stackrel{(b)}{=} \\mathbb{E}\\left[\\int\_0^1 F\_Z^{-1}(\\alpha)+\\frac{1}{\\alpha}\\left(Z - F\_Z^{-1}(\\alpha)\\right)^{-} \\mu(\\mathrm{d}\\alpha)\\right] \\\\ & = \\mathbb{E}\\left[h\_{\\phi, Z}(Z)\\right] \\end{aligned} where step $(a)$ utilizes the CVaR representation provided in [2], and step $(b)$ applies Fubini’s Theorem. Next, we note that $ h_{\phi, Z}$, as defined in Equation 6, is differentiable almost everywhere, with its derivative given by \\begin{aligned} h_{\\phi, Z}^{\\prime}(z) & =\\int_{\\left\\{\\alpha: z \\leq F_Z^{-1}(\\alpha)\\right\\}} \\frac{1}{\\alpha} \\mu_\\phi(\\mathrm{d} \\alpha) \\\\ & =\\int_{F_Z(z)}^1 \\frac{1}{\\alpha} \\mu_\\phi(\\mathrm{d} \\alpha)=\\phi\\left(F_Z(z)\\right). \\end{aligned} Additionally, the infimum in the concave conjugate $\hat{h}\_{\phi, Z}(\phi(u)) = \inf\_z \left( \phi(u) \cdot z - h\_{\phi, Z}(z) \right)$ is achieved at any $z$ where $\phi(u) = h\_{\phi, Z}^{\prime}(z) = \phi\left(F\_Z(z)\right)$, which corresponds to $z = F\_Z^{-1}(u)$. Therefore, we obtain \\begin{aligned} \\int\_0^1 \\hat{h}_{\\phi, Z}(\\phi(u)) \\mathrm{d} u & =\\int\_0^1 \\phi(u) \\cdot F\_Z^{-1}(u) - h\_{\\phi, Z}\\left(F\_Z^{-1}(u)\\right) \\mathrm{d} u \\\\ & =\\int\_0^1 \\phi(u) \\cdot F\_Z^{-1}(u) \\mathrm{d} u-\\int_0^1 h\_{\\phi, Z}\\left(F\_Z^{-1}(u)\\right) \\mathrm{d} u \\\\ & =\\operatorname{SRM}\_\\phi(Z)-\\mathbb{E}\\left[h\_{\\phi, Z}(Z)\\right]\\\\ & = 0. \\end{aligned} > In Line 383, α=0.6 seems to be maximized at CVaR0.8, and α=0.4 at CVaR0.6. Although the small vertical line intervals may be minor, the lack of alignment with targeted risk levels raises concerns. Thank you for your detailed review of the results. The primary goal of this example was to demonstrate the gradual change in the distribution of returns and the option exercise boundary as the risk sensitivity is adjusted. For this reason, we presented the evaluation results based on a single seed. After evaluating the policies with multiple seeds, we found the above-mentioned values to be within one standard deviation of each other. Given that these α values are relatively close (e.g., 0.6 vs. 0.8 or 0.4 vs. 0.6), and considering the use of function approximation and the stochastic nature of the environment, the observed misalignments appear to be minor. However, when comparing policies with larger differences in α (e.g., 1.0 vs. 0.6 or 0.4), the differences in their behavior become more noticeable and align more clearly with their objectives. As noted in our response to Reviewer tbS4, another contributing factor to the misalignment between the final policy and its objective when optimizing CVaR is the phenomenon known as "Blindness to Success" ([3]). CVaR objectives focus solely on the left tail of the return distribution, neglecting valuable information from the right tail. This limitation can make algorithms optimized for CVaR more prone to converging to suboptimal policies. A key motivation for our work is the flexibility of Spectral Risk Measures (SRMs), which address this issue by incorporating the entire return distribution into the objective. For instance, assigning a small weight to the expected value ensures that the full distribution is taken into account, providing a simple yet effective modification that improves performance. > In Table 2, aren't the cases with α=1.0 essentially QR-DQN? Although the objective in all three models (QR-DQN, QR-CVaR(α=1.0), QR-SRM(α=1.0)) is to optimize the expected return, the key difference lies in the augmented state used in QR-CVaR and QR-SRM. This difference in the state accounts for the minor variations observed in the results. **References:** [1] Pichler. Premiums and reserves, adjusted by distortions. Scandinavian Actuarial Journal, 2015 [2] Rockafellar and Uryasev. Optimization of conditional value-at-risk. The Journalof Risk, 2000 [3] Greenberg et al. Efficient Risk-Averse Reinforcement Learning. NeurIPS 2022
Thank you for your detailed response, which has clarified my question regarding Equation 6. Regarding the labels in Figure 2, I noticed that $\lambda$ corresponds to ERM, $\nu$ to DPRM, and $\alpha$ to WSCVaR. It would be helpful to adopt a unified notation for consistency. In Lines 360–367, the paper introduces WSCVaR under the term "Blindness to Success." However, if the risk spectrum focuses on the left tail of the distribution, shouldn’t an agent still be optimal for its specific risk preferences while being suboptimal in terms of expected return? The fact that WSCVaR appears to be optimal across different risks seems somewhat inconsistent. Alternatively, are the authors attempting to demonstrate that WSCVaR outperforms other CVaR metrics? I am curious about the conclusion the authors aim to draw by introducing WSCVaR. Could you elaborate on the intended implications or advantages of this approach?
Dear Reviewer RikA, Thank you for your response and the clarification. We appreciate your insights and the opportunity to improve our work. > Regarding the labels in Figure 2, I noticed that $\alpha$ corresponds to ERM, $\nu$ to DPRM, and $\alpha$ to WSCVaR. It would be helpful to adopt a unified notation for consistency. To unify our notation for spectral risk measures (SRM) across different functional forms and parameters, we adopt the following approach. Additionally, we will relocate the introduction of these notations to the beginning of the experimental results section, allowing readers to become familiar with the notation in a single location before encountering the results. We define our model as QR-SRM($\phi$), where \phi represents the risk spectrum. This encompasses all possible functional forms of the spectral measure, ensuring consistency. For each specific type of spectrum function, we use a subscript: * **CVaR**: QR-SRM($\phi_\alpha$), where $\phi\_{\alpha}(u) = \frac{1}{\alpha} 1\_{[0, \alpha]}(u)$ * **Weighted Sum of CVaRs**: QR-SRM($\phi\_{\vec{\alpha}, \vec{w}}$), where $\phi\_{\vec{\alpha}, \vec{w}}(u) = \sum\_i w\_i \frac{1}{\alpha\_i} 1\_{[0, \alpha\_i]}(u)$ * **Exponential Function**: QR-SRM($\phi\_\lambda$) where $\phi_\lambda(u) = \frac{\lambda e^{-\lambda u}}{1 - e^{-\lambda}}$ * **Dual Power Function**: QR-SRM($\phi\_\nu$) where $\phi_{\nu}(u) = \nu (1 - u)^{\nu - 1}$ > In Lines 360–367, the paper introduces WSCVaR under the term "Blindness to Success." However, if the risk spectrum focuses on the left tail of the distribution, shouldn’t an agent still be optimal for its specific risk preferences while being suboptimal in terms of expected return? The fact that WSCVaR appears to be optimal across different risks seems somewhat inconsistent. > Alternatively, are the authors attempting to demonstrate that WSCVaR outperforms other CVaR metrics? I am curious about the conclusion the authors aim to draw by introducing WSCVaR. Could you elaborate on the intended implications or advantages of this approach? Thank you for your question. We would like to clarify that the term "Blindness to Success" is used in our work to describe a limitation of the CVaR objective, not to introduce WSCVaR. We present WSCVaR in this section to demonstrate how a more general spectral risk measure, such as WSCVaR, can effectively address this limitation. The success of CVaR objectives in finding the optimal policy largely depends on the specific characteristics of the MDP, particularly the reward and transition models. In this example, the agent only receives a large positive reward (10 points) upon reaching the goal, while receiving -1 points in cliff positions and 0 points elsewhere. This makes the positive rewards sparse. Additionally, the wind introduces stochasticity by moving the agent to nearby positions with a 50% chance. The episode is terminated after 50 steps, increasing the likelihood that the agent does not reach the goal. These factors collectively make CVaR objectives prone to converging to suboptimal policies in this scenario. In contrast, Table 2 shows that CVaR objectives can successfully find optimal policies under different MDP characteristics. This illustrates how the performance of CVaR objectives depends significantly on the properties of the MDP. As a subclass of spectral risk measures, CVaR allows for limited flexibility, with the alpha parameter being the only lever to adjust policies. Spectral risk measures, however, offer greater flexibility in defining objectives. For instance, WSCVaR allows us to combine multiple CVaR objectives with arbitrary weights. In this example, a simple weighted combination of the expected value and CVaR ( $ w\mathbb{E} +(1-w) \operatorname{CVaR}_{\alpha}$) leads to a policy that performs better across various metrics compared to policies derived from CVaR objectives alone. Importantly, this improvement cannot be achieved solely by tuning the alpha parameter of CVaR. We hope this explanation clarifies our approach and the advantages of using spectral risk measures like WSCVaR. We have uploaded a revised version of our manuscript that incorporates comments from all reviewers. To make the changes stand out, all additions are highlighted in blue. In response to your specific comments, we have: * Unified our notation for spectral risk measures in the experimental results section. * Updated Figures 1 and 2 to make the vertical lines more distinguishable. * Added the proof that $\int\_0^1 \hat{h}_{\phi, Z}(\phi(u)) \mathrm{d} u = 0$ to the appendix. * Fixed the typos you mentioned. We hope that our clarifications and updates have resolved the issues you raised. If you feel that our responses adequately address your concerns, we kindly ask if you would consider reevaluating your score.
Thank you for your detailed response and for actively incorporating feedback into the manuscript. I appreciate the clarification regarding the CVaR risk measure and the expanded flexibility of SRM, which have helped me better understand the motivations and contributions of the paper. However, after reviewing the discussion between the authors and other reviewers, I find that the response to the question regarding WSCVaR remains insufficiently addressed. Specifically, shouldn’t “success” and the "(defined) risk measure" correspond in a one-to-one manner? For instance, in the case of “Blindness to Success,” the critique of CVaR appears to rely on optimality being evaluated under expectation, while a proper evaluation under CVaR should instead reflect that CVaR yields favorable results according to the defined risk measure. In Table 1, this alignment would ideally result in diagonal entries that are bold or close to bold values, indicating the model’s correct optimization of risk measures. The results favoring WSCVaR, while intriguing, leave some ambiguity. If WSCVaR dominates other CVaR measures across scenarios, it raises questions about whether this dominance stems from inconsistencies in the alignment between risk measures and success definitions. Even if theoretical convergence guarantees hold for SRM or CVaR under specific MDPs, this ambiguity may lead to skepticism regarding its reliability in practice. Given these unresolved concerns, I do not find the response sufficiently persuasive at this time and therefore, will maintain my current score.
Thank you for your valuable feedback and for actively engaging with us during the discussion period, which has significantly helped us improve our work. We are pleased that our responses have addressed some of your concerns. As you rightly pointed out, aligning the algorithm’s objective with the evaluation metric ideally results in diagonal entries that are bold or close to bold values, reflecting the correct optimization of risk measures. This alignment is precisely what we observed and reported in Table 2, where the diagonal entries for QR-SRM are bold or close to bold values. However, it’s important to note that some favorable conditions in this environment may not hold in others. A key difference between the mean-reverting trading environment and the stochastic cliff-walking or windy lunar lander environments lies in their reward models. In the trading environment, the agent receives immediate rewards at each step based on its actions, whereas in the other two environments, significant positive rewards are only obtained at the end of an episode. Additionally, Table 3 demonstrates that QR-SRM($\alpha = 0.5$) performs better than QR-DQN with respect to $\text{CVaR}\_{0.5}$, and QR-SRM($\alpha = 0.2$) performs better with respect to $\text{CVaR}\_{0.2}$. Similar to Table 1, we also observe that using a simple WSCVaR objective instead of CVaR enables the discovery of policies that outperform others across various metrics. In our revised manuscript, we acknowledge that discrepancies between the objective and evaluated performance can arise from multiple factors, with “Focusing on the left tail alone” being one of them. Other factors include the use of an approximation of a distributional value function to derive policies. For instance, Greenberg et al. [1] discuss a policy gradient approach that uses an unbiased return estimate to update policies, whereas the value-based approach used in our work inherently introduces bias. Furthermore, the estimated return distributions introduce additional errors, particularly in the tails. We discuss these limitations and potential avenues for improvement in the conclusion section of our paper. We believe that improving the estimation of distributional value functions, especially with recent advancements in this area, could significantly address some of the misalignments between the objectives and evaluation metrics in our work. Despite these challenges, our results consistently show that, with all other factors held constant, WSCVaR helps identify policies that outperform QR-DQN in worst-case scenarios while having minimal impact on expected returns. Lastly, we would like to emphasize the significant benefit of SRM’s flexibility compared to CVaR. In many practical applications, the ideal risk-sensitive objective may not be clear to the user. Treating the parameters of the risk-sensitive objective as hyperparameters allows users to tune them and compare the resulting policies. This flexibility is particularly advantageous in environments where reward models are arbitrarily designed and lack clear real-world interpretation. For example, while a portfolio manager might have a clear objective in a trading environment, such clarity is often absent in environments like Cliff Walking or Lunar Lander. In these cases, tuning the objective parameters as hyperparameters is a practical solution. The flexibility of SRM, combined with the convergence guarantees discussed in our manuscript and the interpretability tools presented in Section 5, makes SRM an excellent choice for risk-sensitive policy optimization, as demonstrated by our results. **References:** [1] Greenberg, Ido, et al. "Efficient risk-averse reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 32639-32652.
Summary
This paper addresses limitations in current risk-sensitive Distributional Reinforcement Learning by introducing an algorithm that optimizes for a broader class of static SRM, moving beyond the commonly used CVaR. The proposed QR-SRM algorithm utilizes SRMs to adjust the agent's risk sensitivity dynamically, improving policy interpretability and adaptability. Extensive experiments on environments demonstrate that QR-SRM achieves superior performance and consistency with SRM objectives.
Strengths
- The application of SRM to DRL for more interpretable risk-sensitive policies is innovative, introducing valuable theoretical insights and practical tools for risk-sensitive control. - Theoretical grounding and comprehensive experimental evaluations - The problem formulation, motivation, and results are well-articulated, though some technical details could be simplified.
Weaknesses
- Certain theoretical sections, especially around SRM decomposition, may challenge readers due to dense terminology and complex proofs. More illustrative examples or simplified explanations could improve accessibility. - The paper assumes specific properties of SRMs and fixed initial preferences, which may limit the algorithm's flexibility in dynamic environments.
Questions
- The authors use an extended state space to solve the inner optimization problem. Can you provide rigorous justification - The proposed algorithm's computational complexity is not thoroughly analyzed. Given the bilevel optimization algorithmic framework, and the added complexity of optimizing SRM in a distributional RL framework, the computational complexity of the proposed algorithm may be a concern. some missing references about DRL for RSRL - Keramati, R., Dann, C., Tamkin, A. and Brunskill, E., 2020, April. Being optimistic to be conservative: Quickly learning a CVaR policy. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 04, pp. 4436-4443). - Liang, H. and Luo, Z.Q., 2024. Bridging distributional and risk-sensitive reinforcement learning with provable regret bounds. Journal of Machine Learning Research, 25(221), pp.1-56. - Chen, Y., Zhang, X., Wang, S. and Huang, L., Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation. In Forty-first International Conference on Machine Learning.
Rating
6
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Thank you for your thoughtful feedback. We appreciate your recognition of our work's innovation and the value of our theoretical insights and practical tools. We address the highlighted weaknesses and questions below. > Certain theoretical sections, especially around SRM decomposition, may challenge readers due to dense terminology and complex proofs. More illustrative examples or simplified explanations could improve accessibility. We recognize that the terminology introduced for the Decomposition Theorem may pose challenges for new readers. To address this, we included three examples in Appendix F to help familiarize readers with the meaning and intuition behind each term. In the first example, we provide detailed calculations to demonstrate the application of the Decomposition Theorem in a simple MDP. In the second example, we use Theorem 2 to illustrate how the risk measure evolves over time. Finally, in the third example, we apply Theorem 2 to one of our four experimental setups—the Mean-reverting Trading environment—to showcase the change in the risk measure in a practical context. > The paper assumes specific properties of SRMs and fixed initial preferences, which may limit the algorithm's flexibility in dynamic environments. In risk-sensitive RL algorithms, the policy optimization step typically assumes a fixed objective that reflects the agent's initial risk preference. In contrast to other methods, the decomposition of risk preference introduced in our work enables tracking the objective that the policy is optimized for at any time. This added interpretability is unique to our approach and is particularly valuable in dynamic environments. It allows the policy's behavior to be monitored continuously, and if it diverges from the user's preferences at any point, a new policy can be trained to realign with those preferences. > The authors use an extended state space to solve the inner optimization problem. Can you provide rigorous justification The non-linear nature of spectral risk measures makes it infeasible to solve the optimization problem directly using traditional dynamic programming methods. However, by augmenting the state space with additional variables that track accumulated costs and discount factors, the inner optimization problem becomes solvable through dynamic programming. The definition of the state-action value function is provided in the proof of Theorem 2 (Appendix A), and Lemma 3 demonstrates the recursive property of this value function, enabled by the extended state representation. > The proposed algorithm's computational complexity is not thoroughly analyzed. Given the bilevel optimization algorithmic framework, and the added complexity of optimizing SRM in a distributional RL framework, the computational complexity of the proposed algorithm may be a concern. The outer optimization leverages the closed-form solution presented in Equation 6, and updating the function $h$ using this method introduces negligible computational overhead. Compared to the QR-DQN algorithm, the only additions in our approach are state augmentation and risk-sensitive action selection. In Appendix G, we briefly discuss the computational complexity of action selection in our method and empirically evaluate the additional complexity introduced by risk-sensitive action selection. > Some missing references about DRL for RSRL Thank you for highlighting the missing references. We will ensure they are added to the manuscript.
Dear Reviewer RwY2, Thank you again for your valuable feedback. We appreciate your insights and the opportunity to improve our work. We have uploaded a revised version of our manuscript that incorporates comments from all reviewers. To make the changes stand out, all additions are highlighted in blue. In response to your specific comment, we have added the missing references to the manuscript. We hope that our clarifications and updates have resolved the issues you raised. If you feel that our responses adequately address your concerns, we kindly ask if you would consider reevaluating your score.
Thank you for the detailed rebuttal, which addresses many of the concerns raised.
Summary
This work studies the problem of incorporating static spectral risk measures (SRM into Distributional Reinforcement Learning (DRL) to enable more flexible and interpretable risk-sensitive decision-making. Unlike conventional Conditional Value-at-Risk (CVaR), SRMs offer a spectrum of risk preferences, allowing for more flexible risk-sensitive policies. The authors argue that using SRMs in DRL enables more flexible and interpretable policies, as SRMs allow for a spectrum of risk preferences rather than a fixed measure like CVaR. The authors propose an iterative DRL algorithm that utilizes a two-stage optimization process to optimize SRMs. They provide theoretical guarantees, proving convergence and characterizing the temporal decomposition of SRMs within the DRL framework. This decomposition enhances interpretability, as it captures how risk preferences evolve over time. The algorithm’s effectiveness is demonstrated through extensive numerical studies across four example environments, where it outperforms several baseline models, highlighting its potential for real-world applications.
Strengths
1. This paper is easy to follow and well-organized. 2. The theoretical analyses throughout the paper are technically sound and comprehensive, providing strong support for the proposed method. 3. The authors preform thorough numerical studies across four examples. The proposed algorithm outperforms several baselines, highlighting its potential for real-world applications.
Weaknesses
See Question section
Questions
1. In Table 1, when the objective is CVaR(0.1), why does QR-SRM with $\alpha=0.1$ not achieve the highest value? Could the authors clarify the reasons influencing this outcome? 2. The authors introduce the decomposition theorem for SRMs (Theorem 2). Could they use one of the four examples to illustrate how this theorem applies in a practical scenario? This would help readers better understand these concepts.
Rating
6
Confidence
2
Soundness
3
Presentation
3
Contribution
3
Thank you for your positive feedback and for recognizing the strengths of our work. We are glad that you found the paper easy to follow, well-organized, and technically sound and comprehensive. We address your questions below. > 1. In Table 1, when the objective is CVaR(0.1), why does QR-SRM with $\alpha=0.1$ not achieve the highest value? Could the authors clarify the reasons influencing this outcome? Several factors can contribute to discrepancies between the objective and the evaluated performance. These include early stopping during training, the inherent stochasticity of the environment, and the use of function approximation for value functions. A particularly significant factor in the CVaR case is the phenomenon known as "Blindness to Success" ([1]). CVaR objectives focus exclusively on the left tail of the return distribution, disregarding information from the right tail. This limitation makes algorithms optimized for CVaR more susceptible to converging to suboptimal policies. As a result, many risk-sensitive RL studies that focus on CVaR require additional modifications to enhance performance. A key motivation for our work is the flexibility offered by Spectral Risk Measures (SRMs), which provide a straightforward yet effective modification to the objective by incorporating the entire return distribution. For example, assigning a small weight to the expected value, as in QR-SRM($\alpha$ = [0.1, 1.0]), ensures that the entire return distribution is considered. The positive impact of this approach is clearly reflected in the results presented in Table 1. > 2. The authors introduce the decomposition theorem for SRMs (Theorem 2). Could they use one of the four examples to illustrate how this theorem applies in a practical scenario? This would help readers better understand these concepts. We provided three examples to illustrate the intuition behind the Decomposition Theorem and Theorem 2. However, due to space constraints, these examples have been included in Appendix F. The third example focuses specifically on one of the four environments—the Mean-reverting Trading environment—and includes visualizations that clarify the intuition behind Theorem 2. In the first example, we demonstrate the application of the Decomposition Theorem in an MDP with a known model. The second example highlights changes in the preference mappings without relying on the MDP model. **References:** [1] Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient Risk-Averse Reinforcement Learning. In Advances in Neural Information Processing Systems, 2022
Dear Reviewer tbS4, Thank you again for your valuable feedback. We appreciate your insights and the opportunity to improve our work. We have uploaded a revised version of our manuscript that incorporates comments from all reviewers. To make the changes stand out, all additions are highlighted in blue. In response to your specific comment, we have added clarifications to Section 6.1 to better explain the results presented in Table 1. We hope that our clarifications and updates have resolved the issues you raised. If you feel that our responses adequately address your concerns, we kindly ask if you would consider reevaluating your score.
Thank you!
We thank all the reviewers for their positive evaluations and constructive feedback. We underscore the key strengths of our paper mentioned by the reviewers: * Reviewer tbS4 praised its clarity, comprehensive theoretical analyses, and strong experimental results across four examples, highlighting the algorithm's real-world potential. * Reviewer RwY2 appreciated the innovative use of SRM in DRL for interpretable, risk-sensitive policies, alongside solid theoretical grounding and well-articulated motivations. * Reviewer RikA noted the well-motivated problem, clear presentation of concepts, inclusion of reproducible code, and robust experiments across diverse environments. * Reviewer tRxB commended our deep understanding of risk-averse RL, strong mathematical foundations, and concise theoretical presentation. We have uploaded a revised version of our manuscript that incorporates comments from all reviewers. To make the changes stand out, all additions are highlighted in blue. Notably, we have: * Added clarifications to Section 6.1 to better explain the results presented in Table 1. * Updated Figures 1 and 2 to make the vertical lines more distinguishable. * Added the proof that $\int\_0^1 \hat{h}\_{\phi, Z}(\phi(u)) \mathrm{d} u = 0$ to the appendix. * Added the contraction proof of Algorithm 2 to the appendix. * Unified our notation for spectral risk measures in the experimental results section. We hope that our clarifications and updates have resolved the issues you raised. If you feel that our responses adequately address your concerns, we kindly ask if you would consider reevaluating your score.
Meta-review
This paper proposes a novel distributional reinforcement learning algorithm that optimizes for static Spectral Risk Measures (SRM), extending beyond the commonly used CVaR risk measure. The reviewers appreciate the following strengths: - Considering SRM in distributional reinforcement learning is novel and important. - Theoretical analyzed convergence guarantee of the proposed algorithm. - Conducting experiments in four environments, demonstrating the proposed method outperformed baselines. The reviewers shared two major concerns: - The interpretation of the experiments is not consistent with and does not fully support theoretical results. - Although SRM offers greater flexibility compared to CVaR, the motivation and challenges of such an extension requires clearer justification in introduction and experimental design. There are other minor questions and concerns regarding confusing concepts and presentation issues, which have been addressed after thorough discussion during rebuttal period. After discussion period among the reviewers, two reviewers still recommend rejection due to the two unresolve major concerns. I recommend rejection, and encourage the authors to clarifying these questions in next version, as "clarifying these aspects would strengthen the overall contribution of the paper and make the results more compelling to the broader audience" quoted from a reviewer.
Additional comments on reviewer discussion
The reviewers shared two major concerns: - The interpretation of the experiments is not consistent with and does not fully support theoretical results. This concern is raised by Reviewer RikA and tRxB, and has been extensively discussed with the authors. However, both reviewers believe the concern is unresolved. - Although SRM offers greater flexibility compared to CVaR, the motivation and challenges of such an extension requires clearer justification in introduction and experimental design. The authors provided clarification which IMO partially addressed the concern. There are other minor questions and concerns regarding confusing concepts and presentation issues, which have been addressed during rebuttal period.