Policy Optimization for Continuous Reinforcement Learning

Summary

Reinforcement learning (RL) is a powerful tool for solving sequential decision making problems but has primarily been formulated for discrete-time Markov decision processes. However, many real-world systems are more naturally expressed in continuous time, and the proper choice of discretization time step may be challenging. Additionally, a controller operating in continuous time may be more suitable for high frequency applications. Prior work has formulated a continuous-time analog to the advantage function, called the q-value. Building on the concept of a q-value, the paper sets out to derive a continuous-time analog to the discounted occupation probability and performance difference lemma, two core concepts in RL. The authors then show how to compute a gradient of this performance difference lemma to give a continuous analog to the policy gradient. Next, the authors form a local approximation of the performance difference which takes an expectation over the current policy rather than the updated one. They provide bounds on the gap between the true performance metric and this local approximation. Then they use this local approximation to formulate a continuous analog to TRPO\PPO. The paper evaluates the continuous-time policy gradient and PPO algorithms on synthetic linear-quadratic stochastic control problems. They also consider a two-dimensional optimal pair trading problem and show their CPPO algorithm performs best.

Strengths

- Continuous-time RL is an under-explored area that has application in controlling systems at high frequency and using an adaptive discretization scheme with non-uniform time steps. - This appears to be the first work to propose a continuous-time analog in a stochastic setting to the performance difference lemma and derive a continuous PPO algorithm. - The paper is well organized and clearly written. It does a good job explaining the results and provides enough information to support its claims.

Weaknesses

- There is no discussion or experimental results which illustrate how the continuous-time formulation is advantageous over discrete-time RL. The proposed algorithms still ultimately require us to discretize. As such, there should be some comparison to existing discrete-time RL methods. The hope would be that formulating the problem in continuous time allows us to achieve better performance under certain choices of step size. Another potential benefit of continuous-time RL is the case of unevenly-spaced observations. Experiments which highlight these benefits would significantly strengthen the paper. - The problems considered are toy examples which do not tell us much about the scalability of the proposed methods. More complex scenarios would make the paper much more convincing. Especially ones in which the continuous-time formulation provides a clear advantage. - The discussion of related work could be better. It is still a little unclear to me how this work is positioned in the continuous-time RL literature.

Questions

- How do these continuous-time methods compare to their discrete-time counterparts when the time step discretization is uniform? How does the performance gap change with choice of discretization time step? What are the main advantages of this formulation? - How does this approach scale to harder, more realistic problems? - How exactly does the continuous-time policy gradient algorithm in this paper compare to previously proposed approaches?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

There is no discussion of limitations of the continuous-time formulation or their specific algorithms. The paper would be stronger if it discussed these. This could include assumptions made in the proofs or the fact that the work assumes the dynamics follow an Ito SDE driven by Brownian motion.

Summary

This paper investigates the continuous reinforcement learning problem. The proposed method is based on the notion of occupation time for policy gradient, which is analogous to the visitation frequency in discrete Markov decision processes. Empirical evaluations are conducted on two example scenarios for two versions of the algorithm (CPG and CPPO).

Strengths

- Analyze an Important formulation of continuous time and space in reinforcement learning. - Theoretical analysis is present to develop a policy gradient counterpart of TRPO/PPO. - The empirical evaluation is presented along with theoretical analysis.

Weaknesses

- Lack of comparison with discrete counterpart. - The lack of implementation details (missing hyperparameter, random seed) makes reproducibility challenging. - The evaluation of the proposed method is limited to only two hand-crafted examples, which hinders a comprehensive understanding of its implications.

Questions

In Figures 1 and 2, what does it mean to be the distance to the optimal to be some non-zero value? How does this translate to the reward performance? Reward performance is critical to know whether or not the achieved difference in distance is useful. The performance is compared with the proposed two methods, CPG and CPPO (Figure 3), and no baseline is considered. An essential choice would be to use the discrete counterpart (Policy Gradient - PG and PPO). A simple form can be to make the MDP discrete and run PG and PPO. This comparison is even important to understand what kind of problem the continuous RL is useful for. Is the achieved return by CPPO optimal (empirical best) in Figure 3? Implementation details need to be included; what are the PPO-specific hyperparameters (e.g., clipping), for Example 2? For reproducibility, a common practice is using several seed runs to account for variation in the results. How many random seeds are used for the experiments in Figures 1 and 2? These missing details make the presented results hard to reproduce.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper seeks to answer three research questions: 1) Is there a continuous time analog of the state occupancy measure, 2) Is there a convenient expression for the performance difference between two policies in the continuous time setting, and 3) Can PPO be adapted to fit the continuous time setting? In answering these questions, the paper presents a continuous time occupancy measure, a policy performance difference similar to that of conservative policy iteration and related algorithms, and continuous versions of REINFORCE and PPO are provided. Theoretical results for several properties are given, and experiments demonstrating that the algorithms were able to solve a couple of problems are provided.

Strengths

The paper provides a thorough theoretical treatment of defining policy gradients for the continuous time RL setting. The paper clearly defines and adequately answers its stated objectives.

Weaknesses

The biggest area for improvement in this paper is in the empirical results. While the main results of this paper are, theoretical new practical algorithms are presented. Thus, they deserve proper evaluation and experimentation to educate the reader on the challenges of using them. For example, there are no experiments illustrating that there were any special difficulties in applying these algorithms to the continuous setting. There should be experiments illustrating how the hyperparameters, particularly those specific to the continuous time setting, impact the optimization process. Currently, the results only tell us that the algorithms were made to work. Great, but we do not learn anything beyond this trivial result. It is important to develop the reader’s understanding of how they can make the algorithms work or at least what can cause failure. Additionally, the adaptive penalty term should also be explored since it deviates from the standard PPO implementation. Does it effectively constrain the distribution throughout learning? How does the penalty change over time? These are important questions to answer when introducing a new method. Minor quibble: the opening sentence of the introduction is boring. Many papers have used this form, and my eyes glaze over as soon as I read them. Try keeping this motivation brief and tie it immediately to the scope of the work discussed in this paper.

Questions

- How does the sampling at discretization impact the gradient estimate? - How does the step size need to be adapted to work in the continuous-time setting? Is it sensitive to the scaling of the time horizon or sampling rate?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Response to author response

The plots are not useful for comparing since they are all on different graphs. >As mentioned above, the performance of our algorithms appears to be quite robust to step size. What evidence do you have for this? It is not evident to me that one would expect the step size to be robust to changes in time discretization.