An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient

Summary

The paper points to limitations of the mean-variance criterion for risk-averse RL, and of the methods that optimize it. Instead, the paper proposes an alternative risk measure relying on Gini Deviation. The paper shows how to derive the PG for this metric, and demonstrates that it learns risk-averse policies more stably than methods that optimize the mean-variance.

Strengths

1. The idea of using Gini Deviation for risk-averse RL is original as far as I know. 2. The metric is demonstrated as more stable than mean-variance.

Weaknesses

Risk-aversion in RL is often handled by optimizing risk measures. Many risk measures have been studied, e.g., mean-variance, var, cvar, entropic, distortion measures (some of them are briefly mentioned in the paper). There are studies about both the properties of different measures and how to optimize them. In particular, commonly desired properties of the metric are captured by the notion of coherent risk measures. For example, mean-variance is not a coherent risk measure, but CVaR is, and many works have recently studied it. A paper that proposes a new risk measure and claims to its benefits cannot ignore these works and the notion of coherence. Considering the discussion above: 1. The presentation of the paper contribution lacks a lot of relevant context, hence the novelty and contribution are difficult to judge. 2. Relevant literature is hardly covered. There is no organized literature survey, and relevant works are not mentioned. 3. The experiments mostly demonstrate the weakness of the mean-variance risk measure, which is already known and cannot justify the suggested metric just by itself. Since the paper justification is essentially empirical, this is a quite critical issue. Finally, the maze benchmark used in Sec 2.2, Sec 5.1 and Fig 1 is taken directly from [Greenberg et al. 2022], with neither crediting its original nor citing the corresponding work. Notice that using benchmarks and code (even if you modify them) without explicitly citing them in both paper and code may be considered Plagiarism.

Questions

See the issues discussed above under Weaknesses. Additional question: As specified in the paper, the current method uses multiple trajectories for a single policy gradient. However, quantile regression (as in QRDQN) allows to learn quantile value representation without this limitation. Could such quantile regression be used to overcome the multi-trajectory limitation in mean-GD as well?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Algorithmic limitations are fairly discussed in Section 6.

Summary

This paper considers the problem of risk-sensitive reinforcement learning in the policy optimization setting, under the mean-variance risk criteria. They provide background on the problem and demonstrate the issues with learning an optimal policy due to the estimation of the variance requiring double samples. They demonstrate that previous approaches to this problem which use the immediate reward variance as a proxy are not desirable (Section 2). They introduce the Gini deviation as a surrogate for the variance, and demonstrate that through a Choquet integral characterization, it can be estimated without the double-sampling issue by leveraging quantile distributional RL. They incorporate this into a PPO-style algorithm and compare its performance against others across a range of risk-sensitive benchmarks.

Strengths

- Well presented motivation and contribution. - Novel idea, which achieves strong empirical performance.

Weaknesses

- There is little theoretical support for the algorithm, which would strengthen the contribution.

Questions

- What is the space $\mathcal{M}^\infty$ introduced in Definition 1? It is never defined.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss limitations of their work, and propose directions for future work.

Summary

The paper proposes to restrict the Gini deviation of the return for risk-averse reinforcement learning (RARL). This is motivated by a discussion of the shortcomings of variance-based RARL methods, such as scale dependence and instability due to large squared returns. A derivation of an expression for the gradient of the Gini deviation of the return serves as a basis for a practical algorithm, mean-GD (MG). It adds a penalty proportional to the Gini deviation to the reinforcement learning objective. The distribution of returns, which is required for the gradient of the Gini deviation of the return, is approximated with a finite number of sampled trajectories. An empirical evaluation of MG in the tabular and function approximation settings shows advantages of MG in learning risk-averse policies over return and reward-based baselines.

Strengths

The use of the Gini deviation as a new risk measure for RARL is motivated well by a concise discussion of the limitations of variance-based methods and is, to the best of my knowledge, new. The shortcomings of existing methods are illustrated with the help of a simple maze environment. The development of methods that are less sensitive to hyperparameters and environment properties such as reward scale is furthermore highly relevant for applications. The derivation of a tractable expression for the gradient of the Gini deviation is done carefully and required assumptions are stated clearly. The experimental evaluation furthermore considers a sufficiently broad range of environments (tabular maze over lunar lander to MuJoCo environments) and baselines.

Weaknesses

While the derivation of equations (15) and (16) are sufficiently clear, I would appreciate some intuition about the nature of the contribution of the Gini deviation penalty to the parameter update. The current version of the text does not really discuss the proposed algorithm but jumps right into the experimental evaluation. I think that for adoption of the algorithm by the community an intuitive understanding would be helpful. There is furthermore a gap between the assumptions for the derivation and the actual environments used for the experiments and the illustration. The maze environment, for example, uses a shortest path objective which implies a discrete return. Many real-world environments will have complicated return distributions with atoms etc. It would be interesting to learn whether this poses a problem or if the assumptions are technical in nature and only required for the proof. The significance of the experimental results is difficult to judge without further information. While I found the used hyperparameters in the supplementary material, I did not find information on how they were chosen. Was a hyperparameter optimization performed and, if yes, was it done manually, with a grid search or with some other algorithm? Without this information the frequent failures of the baselines are difficult to place. I would furthermore suggest to put vertical lines indicating the performance of MVPI in the figures (to avoid the problem of incompatible x axes). Right now it looks a bit like the most competitive baseline is only shown in the supplementary material where it is hard to compare it to the rest. Related to this point: Part of the motivation for the proposed algorithm was to mitigate the need to extensively tune the hyperparameter $\lambda$ for different environments/reward functions. It would therefore be interesting to see how sensitive MG as well as the baselines are to the hyperparameters controlling the risk-averseness. Typos: * In line 42 “We show that our method can learn risk-averse policy […]”, there is an “a” missing. * In line 69 the expression for the gradient of $M(\theta)$ contains an $a$ and a $u(\theta)$ which have not been introduced. Together with a different expression given in line 88, this looks like a typo.

Questions

* It seems to me that the equality in line 234 is incorrect as the probability density of sampling a trajectory is not necessarily the same as the probability density of obtaining its reward. An extreme counter example would be an environment with a reward identical to zero but many possible trajectories. I do not think this poses a problem for the further derivation of MG as the set of trajectories that result in the same reward is independent of the policy parameters. However, it would still make sense to correct this formulation for better readability. * As the policy is defined to output values in $[0, 1]$ (line 48), does this mean discrete actions are assumed? * I would suggest to better relate the paragraph on the baselines (starting in line 281) to the discussion of prior methods in section 2. With the current formulation, it is not clear to the reader which baselines correspond to which of the methods discussed in section 2. Adding some clarification here would make interpreting the results a lot easier. * I would be interested in learning why most of the environments were specifically designed for this paper. Are no standard environments for RARL available that would have been useful? * Would it be possible to use the standard deviation instead of the variance to get scale invariance? * Is the variance of the estimate for the CDF in equation (16) a problem? * Related to the last question: Would it be possible to learn an approximation of the return distribution to reduce variance and get better convergence of the policy? The return distribution is clearly non-stationary but tracking non-stationary functions with estimates is not uncommon in reinforcement learning. Maybe this could help with asymptotic performance. * Why is MG performing poorly on HalfCheetah while MVPI seems to do well?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations of the MG algorithm, mainly low sample efficiency, have been addressed in a separate paragraph.

Summary

This paper presents a new approach to measure the risk for risk-averse reinforcement learning problems. To be more precise, traditional methods usually use the variance of total return or per-step reward as the measurement of risk, since it is known that the variance is easy to calculate. Though easy to calculate, variance-based methods are observed to be sensitive to numerical scale and hard to optimize the policy. Based on the observation, the authors propose to use Gini deviation as the alternative to variance for measuring the risk. The authors study the properties of the new measurement and present how to estimate Gini deviation using empirical samples. Empirical experiments demonstrate that the new proposed methods can mitigate the limitation of variance-based methods while it can still learn high reward returns, compared with previous methods.

Strengths

- The paper is well written, and the authors did an excellent job of presenting the limitation of variance methods before introducing Gini Deviation. As a result, readers can get a sense of the motivation for using Gini deviation as an alternative. - The authors thoroughly introduce the properties of Gini deviation and present how to estimate this quantity using empirical samples. Besides the authors also introduce how to estimate this quantity under the setting of using importance sampling and PPO. And I think through this, authors can get a better understanding how to use the quantity in different variant of policy gradient methods. - The authors also demonstrate the advantage of Gini Derivation via empirical experiments, for both tabular, discrete, and continuous control experiments.

Weaknesses

- Though the authors present that Gini Deviation can be estimated via empirical samples, my primary concern is that it does not have close form and thus its estimation totally depends on the quality of the samples, if we do not have enough samples, then the variance or the bias of the estimation can lead to undesired optimization behavior. - Since Gini Deivation requires estimating the quantile, it may introduce additional complexity compared with variance-based method. - The authors claim that variance-based methods might be sensitive to numerical scales, but I have not seen any ablation study to directly compare this with Gini Deivation-based policy gradient method.

Questions

- Can you clarify if we use empirical samples to estimate Gini Deviation, what would be the draw back? For example, what's the variance and bias of the new estimator using empirical samples if it has? - Would you mind explaining the complexity of the new proposed method? - Also, is it possible to demonstrate that with the new proposed measurement, the new estimator would be robust to numerical scales? (Maybe no need for empirical experiments, some intuition explanation would also be okay).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

NA

Summary

This paper discusses the challenges of mean-variance RARL methods, and proposes the Gini deviation as a substitute to variance penalization.

Strengths

The paper is clearly written and well-motivated (by the disadvantages of return and instantaneous reward variance). The proposed method is evaluated in several experiments with comparator baselines, which seem convincing.

Weaknesses

It seems the proposed algorithm has to sort its samples in every iteration (L245-248) which seems computationally expensive, and which may be sensitive to the parameters chosen for $n$? While I recognize that this is a largely empirical paper, I suppose the paper could have been made more complete by providing some convergence guarantees for policy gradient, e.g., to a stationary point.

Questions

Included in “weaknesses”.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Response to rebuttal

I thank the authors for clarifying my confusion, and after reading the other reviews I am inclined to maintain my score.

Response to rebuttal

I thank the authors for the detailed response. I appreciate their explanations on how to close the gap between Assumption 1 to 3 and the benchmarking environments, on the choice of benchmarking environments, possible downsides of substituting the standard deviation for the Gini deviation, the variance of the estimator for the GD gradient, and the challenges associated to leveraging distributional RL for RARL, and the performance of MVPI on two versions of the HalfCheetah environment. It is a useful addition to the paper that the authors will discuss the process of hyperparameter tuning in the revised paper, and will integrate the MVPI results better in the main text. I would encourage them to also include the new experiments on the sensitivity with respect to the lambda parameter. Regarding equation in line 234, I thank the authors for pointing out that the counter example I gave violates Assumption 1. The example was intended to illustrate that, in general, the probability (density) of obtaining a specific trajectory is not the same as the probability (density) of obtaining it’s return as multiple trajectories could lead to the same return. For this reason line 234 appears imprecise to me. This is only a minor point, however, and does not affect the validity of the results. I would leave it to the authors to decide whether to change this formulation. I furthermore appreciate that the issue of a missing citation, pointed out by another reviewer, was resolved. I consider my questions and concerns sufficiently addressed, thank you.

Response

Thank the authors for answering my questions. I will keep my score and vote for acceptance. Please incorporate related discussion in your next version.