Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning

Summary

This paper focuses on the scenario of continuous RL with high decision frequency. They show that it is hard for distributional RL agents to accurately estimate action values when the decision frequency increases, just like in the case of ordinary RL agents. They propose a distributional analogy of the action gap in ordinary RL called superiority. Based on this notion they try to mitigate this issue by the rescaling technique and propose a superiority-based algorithm. They also validate the proposed methods by numerical simulations.

Strengths

* The paper is overall well-written and easy to follow. * The idea of superiority distributions is interesting and intuitive, with solid theoretical justifications. * The work is quite complete. Besides the discussion about the action gap in distributional RL and the notion of superiority distributions, the author also proposed new algorithms and performed extensive empirical studies. Therefore, I choose to give a positive rating of this work. However, given that I am unfamiliar with continuous RL, I choose to assign a low confidence score to my review.

Weaknesses

* It seems that the basic idea of the superiority distribution is that for two distributions $\eta_1,\eta_2$, we define an object $\Delta$ that can be viewed as the difference between $\eta_1$ and $\eta_2$. And then given a risk-sensitive objective $\phi$, one may choose actions according to rescaled $\phi(\Delta/h^q)$, where $h$ denotes the decision frequency. But why not simply use the criterion $(\phi(\eta_1)-\phi(\eta_2))/h^q$ for decision making? I think the authors should discuss this issue in later versions of this paper. * I suggest the author add some explanations about the problem setting to the intro section, which would make the paper more readable for those not familiar with the specific field.

Questions

* I note that the superior distributions are actually rescaled by $h^{-q}$, where $q$ is a tuning parameter. In the empirical studies, $q$ is set to be $1$ or $1/2$. Are there any possible heuristic rules for the choice of $q$? * It seems that in the simulation studies, the performances of some methods (DAU, DSUP(1), DAU+DSUP(1/2)) are not monotone w.r.t. the decision frequency, but oscillate a lot as the decision frequency increases. Why would this happen? Can the authors give a more detailed explanation?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

It seems the authors do not explicitly addressed the limitations of their work.

Summary

This paper investigates the action gap in distributional RL, where the decisions are made at high frequency. The authors showed that the distributional RL is also sensitive to the decision frequency. In particular, they proved the action-conditioned return distribution collapse with different rates for the statistics. Also, they introduce the generalization of the advantage function in continuous-time MDP, i.e., superiority, based on which a novel algorithm is designed. The authors finally conducted extensive experiments in the high-frequency option-trading domain to demonstrate their theory and the efficacy of their algorithms.

Strengths

* The writing is rigorous with detailed and clear definitions and theorems. * Algorithmic practice in this setting is creditable. * Experiments are relatively extensive, considering both illustrated and comparative settings.

Weaknesses

* The motivation can be strengthened. What is an action gap in continuous-time MDP, and what is an algorithm's outcome when encountering high-frequency decision-making settings? Why do we investigate the distributional RL in this setting? More explanation would be better. * The clarity can be enhanced. Although I appreciate the rigorous writing, it would be better to emphasize the main conclusions of this paper. For instance, the statistics in distributional RL also decrease at different rates. How do readers learn from these observations? Some rigorous theorems may be put in the appendix and replaced with more straightforward explanations.

Questions

1. While the authors stated that DAU+DSUP(1/2) is inferior to DSUP(1/2), it is indeed a clear drawback of the proposed algorithm as this empirical result is consistent with the theory stated in Section 4.2, and hard to understand to me. Although the authors gave some explanations, they did not rigorously validate their hypothesis. Also, I am not convinced why DSUP(1/2) performs poorly in low-frequency domains. 2. Why only consider 30HZ and DSUP(1/2) in the risk-sensitive setting? My suggestion is to include DAU+DSUP(1/2) as well for an extensive comparison across different frequencies, which could be more convincing. 3. It is suggested that more examples be given to emphasize the significance of studying continuous-time MDP and the motivation to explore distributional RL. 4. It would be better to provide the pointers to the proof of each theorem to allow readers to check the proof instantly. 5. In what sense does the approximation hold in Eq. 4.2?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

The paper investigates the issues around continuous-time distributional reinforcement learning. In traditional RL, the advantage becomes less informative as the frequency of actions increases, vanishing at the limit and making it impossible to distinguish between actions. This work extends this result to distributional RL in several ways: it first proposes a framework based on Wasserstein distances between return distributions, then proves in this framework that a similar problem appears in DLR, and gives tight bounds on the asymptotic convergence rate of the distgap (an action gap analogue for continuous DLR), establishing that the rate differs from the traditional RL case. After that, the paper constructs a notion of superiority - a DLR analogue of advantage, and considers its different variants (transformations) that ensure that action gaps are preserved in the limit. Finally, two families of algorithms are constructed, and shown to outperform the baseline QR-DQN on a benchmark of high-frequency options trading task.

Strengths

The paper investigates a very interesting question, and gives a theoretically satisfactory answers. The formal analysis gives rise to a practical algorithm, which is empirically shown to outperform the QR-DQN baseline. The presentation is very readable while being maximally mathematically precise, which I really appreciated.

Weaknesses

The theoretical part requires quite a lot of background from the reader. While authors give a short intro to SDEs in the appendix, it is, in my opinion, nowhere near the depth required to understand the paper. While it's not really possible to present a comprehensive intro to stochastic analysis, the paper would benefit, in my opinion, from a slight expansion of the relevant appendix. Also in terms of presentation, I found the explanation of the algorithmic part of the paper really dense and confusing, and I cannot say I understood it particularly well from the main text.

Questions

1. The Wasserstein distances considered are for $p \in [1, \infty)$. What breaks if we put $p = \infty$? 2. Why is the behavior of $\vartheta$ mean non-monotone wrt frequency in the example considered in sec 5.1? 3. Apart form distributional RL, could a similar method be applied to continuous-time max-ent RL?

Rating

7

Confidence

2

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes.

Response

I thank the authors for their thorough response. Re: specific additional suggestions: I agree with the other reviewers, that adding heuristic, natural language explanations would benefit the paper. At the same time, given the space constraints, I understand it might be difficult to maintain the level of the mathematical rigour, which I value more highly. When reading the paper, I looked at the Algorithm 1 in the Appendix C, but could not follow the details - I would recommend either splitting it into sub-procedures, or otherwise giving a higher-level abstraction description, because right now it is a page-long wall-of-text, which makes comprehending it a quite daunting task. I decided to keep my original (positive) rating.