Efficient Exploration in Continuous-time Model-based Reinforcement Learning

Summary

The submission considers online RL, while the true dynamics is continuous-time. To deal with this issue, the authors provide a continuous-time model-based method. The proposed method is special by 1. iteratively fitting ode-based models and deriving/learning the control from the fitted models 2. having the option to adaptively selecting the sampling timing. The proposed method is able to well handle problems with continuous-time true dynamics, with both theoretical guarantees and empirical evidence.

Strengths

The submission is novel. The proposed method incorporates continuous-time models with adaptive sampling, which to the best of my knowledge is the first. The submission provides thorough results with both theoretical guarantees and also empirical experiments. The studied problem is well motivated: there are problems where the data is generated in continuous-time in need of specialized methodology like the submission. The writing is clear and easy to follow.

Weaknesses

The submission is solid, and the adaptive MSS seems to be an important and useful technique. However, there is one concern bugging me, which seems a little crucial: it is not clear whether the continuous modeling is indeed needed. My question is twofold: 1. whether it is necessary or practical to assume a continuous-time ground-truth dynamic; 2. whether it is necessary or helpful to use continuous-time models in the method for $f$ First of all, the continuous dynamics in section 2 and equation (1) may never be empirically achieved or used. There are very rare cases where one can really implement a policy in a continuous-manner as that equation. As a result, the policy functions considered will often time be piece-wise constant, which does not fit (1). It is not clear how this may affect the theory. When evaluating a policy, equation (1) may also need to be discretized. Therefore, in each episode, one can also fit a discrete-time model instead of a continuous-time one as suggested by the authors. I am wondering how the proposed method compares to this discrete-time version of the proposed method. Second, the theoretical results like proposition 1 also holds for the discrete dynamics setting. The adaptive MSS may also be feasible in the discrete-time setting, which itself now is implemented in a discretized version. Therefore, the observations above make me wonder whether the continuous-time part is really needed in the submission.

Questions

1. In the experiments, are the cost function collected in a discrete-time manner or continuous-time manner? More specifically, are the results summation of costs or the integral of costs over time? 2. What difference might be, if we replace the ODE model by a discrete-time model, for example, like iLQR. Please demonstrate the differences both theoretically and empirically. 3. In experiments why does the proposed method with equidistant perform better than discrete zero-order hold OC? Are the two policy implemented using the same time interval on the same grid? Or does the proposed method calculates the integral over time of costs? Or does it use a different time interval than the competing discrete-time policy? What is the definition of the discrete zero-order hold OC? 4. Is it possible to implement MSS with a discrete-time assumed model? For example, I can assume that the true model is a discrete-time model which is just same as the Euler discretized version of the ODE in the submission, and then conduct adaptive MSS like in the submission. What is the problem of this naive ablation? 5. Would it be possible to explicitly list the dynamics of the experiments? The cost function is provided in the supplements but not the dynamics. I am concerned that the considered experiments are too simple like all linear ODEs, which may make the method not general enough. As a summary, the submission definitely has novelty and contributions. However, I feel that the current method is not demonstrated clear enough, especially on what role the continuous-time modeling is playing in the method.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

This paper proposes a continuous time framework for model based reinforcement learning. Their algorithm OCoRL solves the optimal control problem eq (1) by: 1, selecting optimistic policy. 2, rollout to collect data. 3, update model estimation and statistics. Specifically, they study the measurement selection strategy (sampling state-action in the continuous time framework), and show both theoretically and empirically the effect of different measurement selection strategy.

Strengths

The paper models the optimal control or reinforcement learning problem in continuous time (eq (1)), which is elegant and also convenient for algorithm design and analysis. The realization of state-action at discrete time steps is naturally modeled as discrete-time sampling from the continuous time trajectory (the measurement selection strategy (MSS)). The proposed algorithm alternates between: optimistic planning given current statistical model and data collection for a more accurate model. They further proposed the adaptive MSS that samples state-action time points based variance of model estimate in the state-action space (rollout trajectory). Both theoretical and empirical analysis are provided and showcase the impact of different discrete sampling strategies.

Weaknesses

The algorithm enjoys elegance and nice theoretical properties, but it can be difficult to realize in practice. Especially, the first step in OCoRL that solves optimistic policy can be very time-consuming in practice. The authors mentioned in Appendix C that it is approximately solved by Iterative Linear Quadratic Regulator, but still it can be impractical. The experiments mainly focus on investigating the impact of measurement selection strategies. A comparison in terms of performance: time/computation complexity and over all cost with existing approach (PPO, etc) would better help readers evaluate this approach

Questions

See weakness

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weakness

Summary

This paper proposes a continuous-time model-based reinforcement learning method for controlling fully observed dynamical system environments where there is a cost to take a sample of the state. A Gaussian process dynamics model is used, and a novel adaptive measurement selection strategy is proposed to determine when to take a sample of the state, such that the overall optimization converges to the optimal policy in the limit of infinite trials. The proposed method (OCoRL) is theoretically analyzed to show a general regret bound holds for any measurement selection strategy and is empirically verified across a range of dynamical system environments.

Strengths

* The proposed method OCoRL and associated general regrets bound that holds for any measurement selection strategy appear novel. * The paper is well-written and clearly laid out. * The no-regret algorithm for nonlinear dynamical systems in the continuous-time RL setting seems widely applicable and relevant to the RL and ML community. * The code is reproducible and easily extendable, being well documented throughout.

Weaknesses

* How does the proposed method perform when the state differential is not observed $\dot{x}_n(t)$ and has to be inferred? Could you perform an ablation of this to show that the proposed method is still practical? * Why is the noise only added to the observed state derivatives $\dot{x}_n(t)$? Perhaps it could be more realistic to consider noise added to both the observed state $x_n(t)$ and the observed state derivative $\dot{x}_n(t)$? * Line 118: "predicted mean and epistemic uncertainty". How does the model guarantee that you only measure the "epistemic uncertainty" and not the "epistemic uncertainty and aleatoric uncertainty"? This was not clear, and I presume the model learns both. If so, how can you split the uncertainty to only use the epistemic uncertainty, as outlined in the method? * The adaptive MSS assumes that $m_n=\left \lceil{T/ \Delta_n}\right \rceil$, i.e., that a sample must be taken in each uniform interval $\Delta_n$ of time. This seems overly restrictive. Can OCoRL be adapted to skip taking a sample in some $\Delta_n$ of time, i.e., where they are not needed or informative? * On this, can a further ablation be performed where $m_n$ is varied across all the environments and baselines to empirically verify the adaptive MSS claims for wide ranges of $m_n$? * Unsubstantiated claim of "We compare the adaptive and equidistant MSSs on all systems and observe that the adaptive MSSs consistently perform better than the equidistant MSS". Table 1, shows that adaptive MSSs can perform better than equidistant MSS only in certain environments and that equidistant MSS achieves the same performance (final cost) within error bounds for the environments of Cancer Treatment, Pendulum, and Mountain Car.

Questions

* Could this approach be generalized to other environments that are not implicitly defined by an ODE, such as other types of differential equations, e.g., delayed differential equations, stochastic differential equations, etc? Furthermore, can this approach work for partially observable environments? * What is the trade-off of varying the number of measurements taken, i.e., an ablation of varying M for all experiments? Does the algorithm still hold under these settings? Can this be demonstrated empirically? * Could the OCoRL method be benchmarked against the closest related works of Yildiz et al. (2021) and Du et al. (2020)? * (From above): Can OCoRL be adapted to skip taking a sample in some $\Delta_n$ of time, i.e., where they are not needed or informative? * Typo: Line 339: "Figure 1" -> "Figure 2".

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are addressed with the assumptions outlined in Section 2.1.

Summary

This paper introduces a novel algorithm for efficient exploration in continuous-time model-based reinforcement learning. The algorithm represents continuous-time dynamics using nonlinear ODEs and captures epistemic uncertainty using probabilistic models. The analysis shows that the approach achieves sublinear regret with significantly fewer samples, making it a promising solution for various applications. The paper also develops a practical adaptive measurement selection strategy that reduces the number of measurements per episode while retaining regret guarantees. The benefits of continuous-time modeling and planning with optimism are demonstrated in several environments. The authors aim to catalyze further exploration within the RL community regarding the potential of modeling dynamics in a continuous-time framework.

Strengths

This paper appears to be one of the pioneering works in the field of continuous-time reinforcement learning (RL) applied to nonlinear dynamical systems. Furthermore, the utilization of epistemic uncertainty for measurement selection, a technique commonly employed in active learning but rarely explored in RL, adds a unique and valuable dimension to this study. In addition to these notable contributions, the paper introduces several innovative techniques and investigates their efficacy across various experimental environments.

Weaknesses

The paper encompasses a wide range of techniques, including model-based RL, continuous-time RL, and aperiodic strategies. However, the abundance of new content may make it challenging to write and comprehend the relationship between each technique and the overall idea. To enhance clarity and understanding, it would be beneficial to reorganize the paper and provide clearer connections between the different techniques employed. Regarding Table 1, it appears that, under a known true model, discrete-time control outperforms continuous-time OC without the inclusion of MSS. Consequently, it may not be fair to directly compare continuous-time OC with MSS and discrete-time control since the conditions and components differ significantly between these approaches. A more appropriate comparison could be made between continuous-time OC and discrete-time control without MSS. To provide a more comprehensive evaluation, it is advisable to compare this method with a broader range of other proposed RL methods. The current results may not fully demonstrate the performance of OCoRL, and including additional comparisons would enhance our understanding of its capabilities. In Figure 3, it would be beneficial to consider incorporating more environments for experimentation. Relying solely on a single environment may not provide enough evidence to draw robust conclusions. Including a broader set of environments would strengthen the validity and generalizability of the findings.

Questions

In which scenarios are measurements considered costly? Have you discussed in the Introduction whether there are enough scenes that truly require continuous-time systems due to costly measurements? In the problem setting, the computation of cumulative regret relies on an optimal policy. However, if the optimal policy is unattainable for a particular problem, can your method still be applied? Please provide an explanation. Regarding Line 122, is it possible for re-calibration techniques to provide accurate uncertainty estimation while maintaining a low time complexity?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Follow up on rebuttal

We hope we could address your concerns adeptly. We would further like to emphasize another benefit of continuous-time modeling over its discrete-time counterpart. Discrete-time algorithms such as H-UCRL only work for the setting where the measurement frequency and control frequency are the same. Continuous time modeling can separate these two elements. We leverage this in our work, by proposing MSS that only collects data that benefits in learning the ODE. In summary, due to continuous-time modeling, our proposed method has more control over the system’s measurement and control frequencies, incorporates a general class of policies such as piece-wise constant policies with any choice of control frequency, and *outperforms the true discrete-time model*. We have updated the paper to further discuss the benefits of continuous-time modeling. It is much appreciated if you could please reconsider your assessment, or respond with questions/suggestions so that we can improve the paper in this regard.