When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL

Summary

The paper focuses on continuous-time reinforcement learning. Typically the control and the measurements of continuous-time systems occur at discrete time points. These interventions generally come with a cost. The paper proposes a time-adaptive approach to reduce the number of these interventions in an optimal manner. Therefore they formulate the problem as an MDP, for which standard RL algorithms can be used. Additionally the authors propose a model-based algorithm for their setting.

Strengths

1. The paper describes an interesting setting. The problem of choosing the optimal measurement and control time-points for continuous-time systems is relevant for many processes in nature and industry. 2. The reformulation of the original problems to discrete-time MDPs is well described. 3. The various experiments visualize and describe the effects of the hyperparameters nicely.

Weaknesses

1. The references in the paper lack consistency in formatting. Conference names are sometimes written in full, sometimes abbreviated, and occasionally omitted altogether. 2. One major issue is the lack of details on the learning process. It would be interesting to compare the learning curves of the equidistant and the time-adaptive approach. 3. The experimental section leaves some questions open -> See Questions Minor Weaknesses: - Figure 3 comes before Figure 2 - Page 5, Line 134 says time discretization splits the whole horizon in T/K discrete time points. I think it should be K time points with length T/K - It is difficult to discern the decline in interactions for the greenhouse task in Figure 3 first row, which coincides with the drop in episode reward.

Questions

1. Regarding the experiments for Figure 2: After what time fraction are the greenhouse and pendulum swing-down usually in the stable equilibrium? As described, if the systems are in the stable equilibrium, the discretized approach continues with interactions, while the time-adaptive approach needs less. It would be interesting to compare the interactions for the time until the stable equilibrium is reached. 2. Figure 2, Pendulum Swing-up: Why does the equidistant approach perform so poorly? In Figure 1, the task could be solved with K=5 number of interactions, without high-frequency changes. Intuitively the equidistant approach should perform well with more than 10 interactions. 3. How many iterations were used for learning the equidistant policy and the SAC-TaCos policy in Figure 2? It appears that learning the equidistant policy should be easier, as it has smaller input and output spaces. 4. In the first row of Figure 3, the interaction cost increases, yet the reward does not decline significantly. It would be interesting to identify the point at which the reward starts to drop.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitations have been adressed adequately.

Summary

The paper introduces a framework for reinforcement learning named Time-adaptive Control & Sensing (TACOS). The TACOS framework reformulates the problem of continuous-time RL into an equivalent discrete-time Markov decision process (MDP) that standard RL algorithms can solve. Additionally, a model-based version of TACOS, named OTACOS, is proposed to reduce the sample complexity.

Strengths

1. The introduction of the Time-adaptive Control & Sensing (TACOS) framework is a novel approach that creatively combines the challenges of continuous-time dynamics with the necessity of minimizing interactions due to cost considerations. 2. The paper provides a strong theoretical foundation for the TACOS framework, with clear reformulation of continuous-time RL problems into an equivalent discrete-time MDP. 3. The empirical results support the theoretical claims, with demonstrations of TACOS and OTACOS outperforming traditional discrete-time counterparts

Weaknesses

1. Although the problem posed by the paper is novel and interesting, the solution simply involves incorporating time $t$ into the action space for learning, which a little lacks novelty. 2. The empirical validation is somewhat limited in diversity, primarily focusing on controlled synthetic environments. This limitation might affect the generalizability of the results to more complex or noisy real-world systems.

Questions

The model learned here differs from the typical models in model-based reinforcement learning, as it incorporates time as a transition variable, making the learning process more challenging. Could the authors demonstrate the learning performance of the model itself?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper does not show a discussion of limitations. Authors could enhance their discussion by explicitly addressing the potential limitations in real-world scenarios that involve complex, noisy, or non-stationary environments.

Summary

This paper proposes a novel time-adaptive RL method framework (TaCoS) for continuous-time systems with continuous state and action spaces. The framework shows that the settings of interactions having costs or a budget of interactions, can be formulated as extended MDPs, that can be solved with standard RL algorithms. The paper theoretically demonstrates this, and empirically verifies this. The paper also empirically demonstrates that TaCoS works across a range of interaction frequencies, and proposes OTaCoS a model-based RL approach.

Strengths

* The paper is well written, and the intuitive explanations help the reader. * The problem of time-adaptive RL for continuous-time systems with continuous state and action spaces is of high significance to the community and is well-motived throughout. * The paper contributions are novel to the framework proposed by TaCoS and provide theoretical and empirical evidence. * The method works surprisingly well, with the surprise that TaCoS achieves better control performance even with a small budget of interactions.

Weaknesses

* L154: “Intuitively, the more stochastic the environment, the more interactions we would require to stabilize the system.” the argument should either have a reference or be empirically verified. * Minor: Missing related work reference of (Nam et al. 2021). References: * Nam, HyunJi Alex, Scott Fleming, and Emma Brunskill. "Reinforcement learning with state observation costs in action-contingent noiselessly observable Markov decision processes." Advances in Neural Information Processing Systems 34 (2021): 15650-15666.

Questions

* Is the action applied from the policy kept constant until the next interaction? I presume the action is kept constant. Have you considered parameterizing the action and enabling non-constant actions or even continuous trajectories of actions between interaction times as a form of open-loop planning?

Rating

8

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, they are discussed in Section 5.

Summary

Reinforcement learning (RL) is effective for discrete-time Markov decision processes (MDPs), but many systems operate continuously in time, making discrete-time MDPs less accurate. In applications like greenhouse control or medical treatments, each interaction is costly due to manual intervention. To address this, we propose a time-adaptive approach, Time-adaptive Control & Sensing (TACOS), which optimizes policies that also predict the duration of their application. This results in an extended MDP solvable by standard RL algorithms. TACOS significantly reduces interactions while maintaining or improving performance and robustness. We also introduce OTACOS, a model-based algorithm with sublinear regret for smooth dynamic systems, offering further sample-efficiency gains.

Strengths

- handling continuosu control is sort ot interesting since this also bridge the gap between real-world RL and RL in simulation.

Weaknesses

My questions are listed as below

Questions

Thank you for the opportunity to review your paper. I have a few questions and observations outlined below. 1. Real-world applications and global frequencies. - The discussion around the requirement for different global frequencies in real-world applications is intriguing (line 32), as it highlights the limitations of discrete-time sampling (lines 30-32). However, it seems this limitation is not fully addressed by continuous-time control (or the TACOS algorithm). The fundamental difference between discrete-time control and continuous-time control lies in the amount of environmental information included to compute the policy. For instance, within the total time interval $[0,T]$, discrete-time control computes the policy based on $K$ sampled data points {$t_1, t_2, \cdots, t_K$}, resulting in an optimal policy only for those specific sampled times. In contrast, continuous-time control computes an optimal policy over the entire continuous duration $[0,T]$ and then applies it at discrete times (constrained by Equations (2) and (3)). Utilizing the policy derived from a continuous-time formulation in a discrete-time setting would provide better generalization ability to obtain higher rewards for $t \notin$ {$ t_1,t_2,\cdots,t_K$} but ***does not necessarily address the limitation mentioned in lines 30-32.*** 2. Definition of Notations: - The paper lacks precise definitions of certain notations. For example, the policy $\pi: \mathcal{X} \to \mathcal{U}$ maps states to control inputs, but the exact meaning of $\pi_{\mathcal{T}}$ is unclear. In line 80, it is stated that "$\pi_{\mathcal{T}}$ is a policy that predicts the duration of applying the action." This suggests $\pi_{\mathcal{T}}$ is a prediction variable rather than a policy, as the term "policy" typically refers to decisions or actions taken. Additionally, while $t_{i} = \pi_{\mathcal{T}}(x_{t_{i-1}}) + t_{i-1}$ is mentioned, the role of $\pi_{\mathcal{T}}$ in the objective function needs further clarification. Also have some minor issues.. - Line 65-66 and 68-69 discuss "real-time inference" or "adaptive control approach," which is a known challenge in real-world reinforcement learning. Including references to related literature would help readers better understand these contributions as follows. [1] Dulac-Arnold, G., Mankowitz, D., and Hester, T.Challenges of real-world reinforcement learning.arXiv preprint arXiv:1904.12901, 2019. [2] Hyunin Lee, Ming Jin, Javad Lavaei, Somayeh Sojoudi, Pausing Policy Learning in Non-stationary Reinforcement Learning. ICML 2024 [3] Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, I., and Abbeel, P.Continuous adaptation via meta-learning in nonstationary and competitive environments.In ICLR, 2018. - Also please discuss what SDE stands for.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

No potential negative societal impact.

Follow up on rebuttal

Dear Reviewer, We hope this message finds you well. We have noticed that our detailed rebuttal, addressing each of the concerns raised in your review, has not yet received any feedback. We understand the demanding nature of the review process and appreciate the time and effort invested in evaluating our work. We kindly urge you to consider our responses to your questions, as we believe they adeptly address your concerns. With some days left, we hope we can still have a fruitful rebuttal period. Thank you, Authors

Follow up on rebuttal

Dear Reviewer, We hope this message finds you well. We have noticed that our detailed rebuttal, addressing each of the concerns raised in your review, has not yet received any feedback. We understand the demanding nature of the review process and appreciate the time and effort invested in evaluating our work. We kindly urge you to consider our responses to your questions, as we believe they adeptly address your concerns. With some days left, we hope we can still have a fruitful rebuttal period. Thank you, Authors

Official Comment

Thanks for raising our score and the active engagement in the review process!