Summary
This paper addresses learning models that are sufficient to model the environment, in the sense that optimising for a risk-sensitive objective on that model is equivalent to optimising the risk-sensitive objective on the actual environment. The authors show that value-equivalent models (which match the real environment in expectation) are insufficient in the risk-sensitive case. The authors introduce the distribution equivalence principle, which defines the set of models that induce the same return distribution as the real environment (and therefore the same risk-sensitive values). To relax the assumption that the entire return distribution must be matched, the authors introduce statistical function equivalence, meaning that the models are are equivalent in terms of some statistics of the return (e.g. mean and variance). The authors define loss functions that implement these insights.
Strengths
* I think the paper addresses a novel topic: how best to learn models when the goal is to learn a risk-sensitive policy rather than a standard expected value policy.
* The writing is clear in sections 2-5, and the ideas are well-formalised. I think sections 2-5 are very strong.
Weaknesses
* The paper makes it clear that value-equivalent models fail in the risk-sensitive setting (Proposition 3.2 and experiments), however it fails to motivate why one would use distribution equivalence instead of the standard approach to learning a model - maximum likelihood estimation. Presumably, if the model has enough capacity and is expressive enough, we can expect a model learnt using MLE to learn the correct distributions for distributional equivalence. The paper should explain and demonstrate in which situations the model learnt using the proposed approach results in better performance than MLE model estimate. For example, in the case where model has limited capacity, there is limited data, or the model uses a simplified distribution (such as a Gaussian over successor states), we might expect the proposed approach to work better for risk-sensitive optimisation. However, the paper does not discuss or demonstrate these potential advantages.
* The description of the experiments is unclear - please see my questions.
* In the empirical evaluation, it appears that the authors do not compare against MLE model estimation, which is the most obvious baseline. In Four Rooms/Frozen Lake/Windy Cliffs, the only baseline is the value equivalent model (which is obviously a bad approach for risk-sensitive optimisation). In the option-trading environment, the authors compare against the VE model again, as well as a model-free approach. The authors outperform the model-free approach, but lines 318 and 329 hint that this is because their approach is model-based (and therefore obtains better sample efficiency). Thus, it is unclear if the approach of the authors is better because of the ideas introduced in the paper, or simply because it is model-based. The authors should compare against the MLE model baseline to demonstrate that in some situations their approach is better than the most naive MLE model-based approach.
* Proposition 3.2 uses this introduced notion of epsilon-strictly risk-sensitive. This definition is suitable for CVaR, but does not apply to many spectral risk measures, which apply non-zero mass to all quantiles (e.g. Wang risk measure). In the latter case, epsilon is zero, and therefore Proposition 3.2 does not support the arguments of the authors, as it shows the error is greater than or equal to zero. It seems like it should be possible to come up with a different bound for proposition 3.2 that is more general (and therefore shows that value-equivalence can be sub-optimal for any spectral risk measures other than expected value).
Minor comments:
* Reproducibility: the code provided has no Readme, and for the option trading environment it is completely unclear which of many files to run to generate the results in the paper.
* $\Pi$ is used to denote both a set of policies, and a projection operator. This notation is a little confusing.
Other than the lack of discussion about the potential advantages of the proposed approach (compared to MLE), the lack of a comparison to standard MLE baseline in the experiments, and uncertainty about some of the experiment details, I really like the paper. Thus, I am likely to increase my score to an accept score if I think these concerns are adequately addressed during the rebuttal period.
Questions
* Are you able to provide experiments demonstrating the difference in performance between between risk-sensitive policies optimised using your distribution-equivalent models vs an MLE model? Perhaps we might expect that models learnt using your approach result in better risk-sensitive performance when model capacity is limited (as demonstrated in Grimm 2020 for the value-equivalence case).
* Can you explain the situations (such as limited model capacity, or limited data) where your approach is more suitable than MLE model learning for risk-sensitive policy learning?
* Section 5 does a good job of explaining that models can be equivalent only for a certain set of return statistics that are of interest. However, in the tabular experiments the authors learn an equivalent model for the mean and variance, and then optimise for CVaR. This seems to contradict section 5, as mean and variance are not sufficient for estimating CVaR. What was the motivation for choosing the mean and variance as the return statistics of interest in this experiment?
* What return statistics/functionals were used for the distribution-equivalent model in the option trading experiment?
* Is the improvement in sample efficiency (Line 329 of the paper) compared to the Lim & Malik (2022) approach because of the distributional-equivalence approach you have proposed, or simply because any model-based approach (such as MLE) is more efficient than Lim & Malik (2022)?
* How is the model represented in the experiments? In the tabular environment, I assumed it was just a categorical distribution over successor states. However, the appendix says that GPU training was used - indicating that this is a neural network model? Likewise, what was the model architecture for the option-training case?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
I think the authors have done a good job of addressing potential limitations. In particular, by proposing the approximate version of distributional equivalence in Section 6.