Neural ordinary differential equations (ODEs) are widely recognized as the standard for modeling physical mechanisms, which help to perform approximate inference in unknown physical or biological environments. In partially observable (PO) environments, how to infer unseen information from raw observations puzzled the agents. By using a recurrent policy with a compact context, context-based reinforcement learning provides a flexible way to extract unobservable information from historical transitions. To help the agent extract more dynamics-related information, we present a novel ODE-based recurrent model combines with model-free reinforcement learning (RL) framework to solve partially observable Markov decision processes (POMDPs). We experimentally demonstrate the efficacy of our methods across various PO continuous control and meta-RL tasks. Furthermore, our experiments illustrate that our method is robust against irregular observations, owing to the ability of ODEs to model irregularly-sampled time series.
Paper
Similar papers
Peer review
Summary
This manuscript presents GRU-ODE, a new ODE-based recurrent model designed to tackle POMDP control tasks. By utilizing a numerical ODE solver, GRU-ODE effectively incorporates past observations, actions, and rewards into its context. In comparison to previous RNN-based methods, GRU-ODE offers several advantages such as the ability to handle irregular observations and infer unfamiliar physical environments. Experimental results demonstrate the superior performance of GRU-ODE compared to baselines across standard POMDP tasks, meta RL tasks, and tasks with temporally irregular observations. **After rebuttal**: The authors addressed most of my concerns and the clarification is valid. I have increased my score to 5.
Strengths
1. The experimental results in this work adequately verify the proposed GRU-ODE method, which demonstrates competitive evaluation performance. 2. The paper is well-organized and easy to read.
Weaknesses
1. My main concern is the applicability of the proposed method. The authors assume certain conditions, such as the structure of the reward function and the underlying physics of the task, which may limit the applicability of GRU-ODE. Furthermore, ODE-based encoders fall short in handling discrete game environments such as Atari, as mentioned in line 337. 2. The baselines considered are not sufficiently recent, which weakens the persuasiveness of the evaluation. It is recommended to include results from published papers from the past two years, such as [1, 2]. 3. Using ODEs would inevitably incur more training time and inference time. Although the authors compare runtimes with different numerical solvers, it is recommended to also include comparisons to baseline methods. [1] Ni T, Eysenbach B, Salakhutdinov R. Recurrent model-free rl can be a strong baseline for many pomdps[J]. arXiv preprint arXiv:2110.05038, 2021. [2] Morad S, Kortvelesy R, Bettini M, et al. POPGym: Benchmarking Partially Observable Reinforcement Learning[J]. arXiv preprint arXiv:2303.01859, 2023.
Questions
1. Could you elaborate on the distinction between the GRU-ODE module and the model used in prior studies [1, 2]? 2. How are baseline methods used for tasks with irregular temporal observations? [1] Rubanova Y, Chen R T Q, Duvenaud D K. Latent ordinary differential equations for irregularly-sampled time series[J]. Advances in neural information processing systems, 2019, 32. [2] Lechner M, Hasani R. Learning long-term dependencies in irregularly-sampled time series[J]. arXiv preprint arXiv:2006.04418, 2020.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors adequately acknowledge the limitations of the proposed method.
Summary
The paper proposes GRU-ODE, which encodes historical observations, actions, and rewards into a latent variable. This model is based on recurrent networks and modifies the GRU topology by separating its memory cell into time-continuous states. The integration of variables is accomplished through neural ODEs. Given the differential process, this formulation can extract more accurate unobservable information from partially observable settings. This approach outperforms baselines in several partially observable and Meta-RL control tasks and irregular partially observable tasks.
Strengths
- The use of ordinary differential equations (ODEs) allows the model to handle continuous time dynamics, which is a more accurate representation of many real-world systems. This can lead to more accurate and realistic models. - The proposed GRU-ODE model is designed to handle partially observable Markov decision processes (POMDPs), a challenging problem in reinforcement learning. By encoding historical observations, actions, and rewards into a latent variable, the model can extract unobservable information from the environment. I found this quite interesting specifically, the results presented in Fig~3 are quite convincing. - The paper demonstrates that the proposed method is robust against irregular observations, which is a significant advantage in real-world applications where observations may not always be regularly or uniformly sampled. - The authors show that their approach outperforms baselines in several regular partially observable control tasks. - Its nice to see that the encoder and the policy network together from the same data buffer. Simplifies the training process.
Weaknesses
- Because networks were trained from the same data buffer – does this cause exploration problems? Potentially why this approach is unable to handle sparse rewards? - The models performance should be tested on robotics tasks e.g., Franka kitchen(https://github.com/Farama-Foundation/D4RL/tree/master/d4rl/kitchen/third_party/franka). It's unclear how well the model would generalize to other slightly more complicated dynamics. - The model makes several assumptions, such as the use of ordinary differential equations (ODEs) to model the dynamics of the system. If these assumptions do not hold in a particular application, the model's performance will be impacted i.e., the stationarity and time-dependence.
Questions
- Is this the first combination of GRU-ODE – or the first applied to control? - Could you compare this recent SOTA POMDP formulations e.g., Dreamer with the same partial inputs? - How different is the performance if the same latent context variable (i.e., same GRU-ODE encoder) is used for both the action and critic? - Currently, only positions / velocities are removed? How does performance change if multiple observation information is removed. Would it be possible to evaluate this for some environments? - For $x_t$ why does it have to be a concatenation of $o_{0:t}, a_{0:t}$ and $r_{0:t}$? How much does the performance vary when you remove this extra signal? - How is the incremental $dt$ varied in the numerical ODE solver for irregular observations? If the $\delta = 0.05$ is varied, does it cause issues when the time is different between the generative process and model? - Can you relax the deterministic policy function assumption? Would this cause issues with differentiating $a(t)$ wrt to time. - How is discount formulated? - How does this formulation perform in stochastic settings? - Would it be include a metric that ranks performance conditioned on the time budget? - Was there any pretraining?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
- The authors have adequately addressed the concerns about the Model Complexity. This is because the proposed GRU-ODE model, while powerful, is likely more complex than traditional models. This could make it more computationally intensive and harder to implement. - Furthermore, these setting are relevant for lower-level control e.g., the Hopper tasks. However, for environments with discrete dynamics these might not be relevant.
Summary
The authors propose a neural ODE-based GRU for use in partially observable RL. They effectively "filter" GRU recurrent states using an ODE gate, before feeding the filtered state back into the GRU. They use a variational approach, where their model $g$ spits out a mean and variance. They find that training with just reward results in a highly-variable latent state $z_t$, so they constrain $z_t$ via an additional KL loss term with the prior recurrent state. They go on to evaluate their method across partially observable and meta RL tasks. They examine cartpole with masked velocity/position, as well as various PyBullet/MuJoCo masked control tasks, comparing against SAC+LSTM and a few other methods. They go on to evaluate tasks with irregularly-spaced observations in time. The idea is that a continuous temporal representation using the ODE is more amenable to continuous time than a discrete RNN.
Strengths
- Their method is novel -- I was not able to find other papers using ODEs within RNNs for POMDPs. - Their half-cheetah experiment is very promising, showing that their method matches oracle performance while others cannot - The paper is generally well written
Weaknesses
- Although their method seems interesting, it seems impractical given current hardware. They must solve an ODE at each timestep (even during inference). Given that interesting RL tasks require collecting tens/hundreds/thousands of millions of transitions, their method is likely to be restricted to toy tasks, at least for now. - Many of their experiments are limited to 1M frames, which seems low. This is likely due to the high computational complexity of their method. For most environments, it is not clear that their conclusions would hold at 10M or 50M steps as their policies do not appear to have converged. In their defence, the half-cheeta experiment appears very promising, and the Ni et al. paper they reference trains the PyBullet tasks for the same number of environment steps. - They do not ablate components of their approach
Questions
- If you are going to modify the internals of a GRU, I think it makes sense to write out the full GRU equations. - "We found that the reward signals could help the agent learn unobservable information, leading to improve performance of the agent in PO environments." - What does this mean? What is "unobservable" information in this context? Generally POMDP observations are modeled as $o \sim O(s)$, in that the state is always indirectly observed. - Line 135/136: Are you providing the true Markov state to your model via the loss function? This limits your approach to tasks where we know the underlying state, like toy tasks. After reading the training section, I don't think this is the case but it might confuse readers. - I would be interested in further information/experiments about highly-variable recurrent states and how a KL regularizer helps - I do also wonder how responsible this is for your results. How much is the ODE that is helping compared to the variational state constraints.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
1 poor
Contribution
2 fair
Limitations
- The authors claim that their method performs well in discrete tasks, such as atari. This is to be expected, as a continuous-time representation does not make sense in these scenarios. - They also find that solving ODEs is computationally expensive
Summary
The author present a method for doing deep reinforcement learning for POMDPs with uneven time between observations. Their method uses TD3 with Neural-ODE+GRU encoders for the actor and the critic. They test on various partially observed control tasks, as well as tasks with uneven observation times.
Strengths
The approach is sound and the results are good even though limited. To the best of my knowledge this is a novel approach for dealing with continuous time observation in RL and generally for using Neural-ODEs in RL. The problem of dealing with non-uniform time between observation is interesting for a lot of problems, especially robotics. The presentation is good (except a few minor things which I will highlight below). I find the integration of continuous-time actions with equation (5) clever, I would have gone with a constant action in between timesteps if I was going to tackle this problem. There will be a lot of small complaints below, but keep in mind that my overall opinion of the paper is positive.
Weaknesses
I think the evaluation environments are a bad choice for POMDPs. Just removing part of the state is a very basic way to simulate partial observability. In the case of observing position and not velocity, think of all of the typical control benchmarks (Deepmind control suite for example) with pixel observations. They're equivalent (you can't observe velocities from a single frame) and yet people mostly treat them as MDPs and ignore the partial observability. How? Because you simply have to stack a couple of frames and use regular deep RL approaches. The partial observability aspect is easily addressed and is simply not interesting. There's no information gathering happening. This is a wider complaint because that's the way that a lot of POMDP papers do evaluation, so I won't take off to many points for this. But as a research community, we really need to sit down and find better benchmarks for POMDPs which highlight their ability to combine information gathering and reward maximization, which is the beauty of the framework. The way I read the first result section is this way: we do well on the easy problem, position only (but I'd like to see basic TD3 with a concatenation of 3-4 previous observations as a baseline), but on the harder, actually interesting problem, we do just as bad as the other approaches. I find the use of non-standard deprecated environments odd, even though you mention they're "more difficult" (how so?). Because all of the evaluation seems to be on non-standard environments, it's difficult to assess the significance of the results. My intuition is that pendulum from position only is actually easily solved, so it raises suspicion that's you are able to beat everyone else by a large margin. No information on how other algorithms were tuned was provided. Convergence speed, which is what your algorithm seems to be better at, varies wildly depending on hyperparameters. Some sentences I found hard to read: line 201 to 203, 170. 52-54. Line 56 is a bit hand-wavy. What do you mean by extract more accurate unobservable information? What the experiments show is faster reward maximization. No evaluation of the method's ability to gather information is shown. The sentence about POMDP starting at line 107 is not correct. You're confusing meta-RL and POMDPs. Please give a formal mathematical definition of a POMDP and provide a citation. You will see that for most the the literature the reward function is the same across episodes. Yes, you can transform a meta-RL problem into a POMDP, by adding a hidden task variable as input to the reward function. Equation (2) $x_t$ is defined too far from it's appearance in the equation, I was confused at first (minor) Line 178, the comment about "physical principles" feels a bit hand-wavy to me. The type of Neural ODEs that are used here are no more physically inspired than typical RNNs; no physics knowledge is given to them. Yes a lot of mechanics is usually written down by engineers and physicists as ODEs, but I don't think that means Neural ODEs have "underlying physical principles", unless some actual physics guide the learning.
Questions
Why are there no experiments on pixels observations? It seems the algorithm is scalable. Why did you use the deprecated environments? What makes them harder and why does that matter as long as the comparison is fair? Model implementation, item 2 and 3. Why do you add back o_t as input to the network. It's already encoded by g, no?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
In their limitation section, the authors mention atari. I don't think this is a very important limitation; it's OK, nobody expected ODEs to do well here. What worries me more is the ability to scale to pixels at all. They mention that their method is slower and I suspected that.
Some notes on the authors' reply
W6: OK, is this going to be changed in the paper? Q1: Dreamer, DreamerV1 are all well known RL methods that have recurrent memory in them and can deal with POMDPs. They perform well in pixel-based environments.
Replay to Reviewer FbJ4
Thanks for your reply. About **W6**: Yes, we will add this definition to our paper and delete the previous version. About **Q1**: Upon revisiting the relevant literature, we acknowledge our oversight and apologize for the misunderstanding in our initial response. However, upon closer examination, methods like Planet[1] or Dreamer[2] define POMDP as "Since individual image observations generally do not reveal the full state of the environment, we consider a partially observable Markov decision process (POMDP)," where images themselves are treated as partially observable states. This distinction sets it apart from our experimental setup. Consequently, we consider these two approaches to exist on different baselines. We will clarify the distinction between these environment settings and provide references to the relevant pixel-based methods in the related work section. The challenge of learning tasks in pixel-based environments lies in improving image representation ability, an aspect we intend to address in our future works. We are thankful for the valuable insights you've provided. [1] Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2019, May). Learning latent dynamics for planning from pixels. In *International conference on machine learning* (pp. 2555-2565). PMLR. [2] Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019). Dream to control: Learning behaviors by latent imagination. *arXiv preprint arXiv:1912.01603*.
Thank you for your response. I appreciate the additional information provided. But I still have some questions regarding the proposed method. - I have some doubts about the runtime comparison results. According to [1], the ODE-RNN takes 60% more time than the standard GRU, and the Latent ODE requires roughly twice as much time to evaluate compared to the ODE-RNN. Since there are similarities between the proposed method and ODE-RNN [1], I would appreciate more information on details such as network architectures, model sizes, computation resources, training frameworks, etc., in order to better understand and assess these results. - Could you provide more details on why changing the computation order of the ODE and GRU updates is important? I am not clear on how this change "has proven beneficial for forward-in-time encoding in our scenario". I also notice that forward-in-time encoding has been discussed in [1], which to me seems like a task-specific design choice rather than major model modifications. [1] Rubanova Y, Chen R T Q, Duvenaud D K. Latent ordinary differential equations for irregularly-sampled time series[J]. Advances in neural information processing systems, 2019, 32.
Replay to Reviewer kjqZ
Thank you for your response. Sorry for not solving your confusion in the rebuttal, we hope the following answers can solve them. 1. We believe you are an expert in the field of ODEs, and your insights into the computational time issue are highly relevant. First, within our implementation of *ODESolve()*, we employ the "Euler" method for computation. This offers substantial reductions in computation time when contrasted with the default "dopri5" approach (Runge-Kutta of order 5 of Dormand-Prince-Shampine) utilized in ODE-RNN[1]. We observed that employing complex *ODESolve()* methods doesn't notably enhance performance(as shown in Appendix section 10), instead, it increases computational time significantly. Meanwhile, let us demonstrate that we are working within the context of RL tasks. When training RL algorithms, it's not just the ODE-GRU that needs training, other networks such as policy networks and value networks also require training. During the training process, a significant amount of time is consumed by sampling transitions from the data buffer and updating the data buffer. We believe that models like ODE-RNN do not necessarily require a long time to train for fitting time series data (compared to the training time for RL). Therefore, in terms of the overall training time of our approach, it doesn't significantly exceed the time required by the previous method that used an RNN as an encoder (RMF)[2]. In fact, our network parameters and computational resources have already been listed in the supplementary materials we previously submitted (Section 9 and Section 10). Our network architecture is also identical to the one depicted in Figure 1 and our training procedure follows the setting in [2]. Additionally, we will release our code if the paper is accepted. 2. For changing the computation order, we acknowledge that this may seem like a task-specific design. During our experiments, we found that performances of encoders with computational order like ODE-RNN are quite similar to simple GRU encoders (just slightly better) and the visualization results are not reasonable compared to the current version. We analyzed that the distinction between these two orders is which function receives $h_{t-1}$ as the input. We conjecture from the experimental results that using GRU to receive $h_{t-1}$ is much more stable than using *ODESolve()*, as in the POMDP setting inferring unobservable state firstly and then computing continuity can obtain more environment information without losing continuity. The inclusion of the variational process and the utilization of the KL divergence are also seen as modifications to better train the RL algorithm in the POMDP setting. Since our objective is to address POMDP tasks, we reckon that any modifications that can enhance our performance are indeed meaningful. [1] Rubanova, Y., Chen, R. T., & Duvenaud, D. K. (2019). Latent ordinary differential equations for irregularly-sampled time series. *Advances in neural information processing systems*, *32*. [2] Ni, T., Eysenbach, B., & Salakhutdinov, R. (2021). Recurrent model-free rl can be a strong baseline for many pomdps. *arXiv preprint arXiv:2110.05038*.
Thank you for your response. I appreciate the valid points you raised, and I recommend including them in the revised version. As a result, I would like to increase my rating to 5.
Replay to Reviewer kjqZ
We will include relevant explanations in the revised version. Thanks for your suggestions. We are pleased to earn your recognition.
What if you were to apply the KL divergence regularization to the other methods like SAC-LSTM/SLAC/VRM? Would they outperform your ODE-based approach?
Replay to Reviewer 6E9a
Thank you for your reply. Let me first give you our conclusion, these three methods will not outperform our method. Actually, the SLAC and VRM methods use the evidence lower bound (ELBO) as part of the optimization target, the ELBO can be written as: $\textit{ELBO}(q)=\mathbb{E}_q[\log p(x|z)]-\textit{KL}(q(z)||p(z))$, which contains the KL divergence. For the SAC-LSTM, at the beginning of our research, we tried to combine this algorithm with KL divergence optimization. However, we found the performance is just slightly improved and even inferior to VRM. Thus, all these three methods will not outperform our method with KL divergence.
So that ELBO loss pushes all recurrent states to be unit-normal. In your Eq. 8, it is my understanding that you are enforcing a different constraint: the current state must be similar the previous state, right? > We set the prior to our previous posterior This seems like a stronger and more useful constraint, as it ensures that the state's rate of change remains low, where as the ELBO loss says nothing about how quickly the state can change. With your SAC-LSTM experiments, was the prior based on the previous state?
Replay to Reviewer 6E9a
Thank you for your further reply. We have revised our response because some of our understandings were not clearly conveyed before. Yes, we use KL loss with the prior based on the previous state in SAC-LSTM and our aim is to avoid significant changes between hidden states of adjacent time steps. Actually, at first, we found the hidden state of the LSTM does not change hugely (we regard the computation of hidden state update as simple). However, with our GRU-ODE, due to the unavailability of a supervised learning framework, there were instances where the hidden state exhibited significant changes, especially when compared to LSTM in the context of RL. Within the RL framework, where the GRU-ODE serves as an encoder to represent historical information, rapid changes between the context variables $z_t$ may compromise its representational capacity and potentially affect the performance of the RL algorithm. Thus, we made this improvement and used KL divergence to regularize the adjacent outputs. You could find some similar regularization methods in [1,2]. [1] Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., & Whiteson, S. (2019). Varibad: A very good method for bayes-adaptive deep rl via meta-learning. *arXiv preprint arXiv:1910.08348*. [2] Deng, F., Jang, I., & Ahn, S. (2022, June). Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representations. In International Conference on Machine Learning (pp. 4956-4975). PMLR.
Thanks for the clarification. To be fully convinced, I would need to see ablations for the previous-state-prior KL constraint applied to the other methods. It seems unfair to apply the previous-state-prior KL constraint to your method and not the others, as it is a general method and not specific to the ODE-based approach. As such, I will keep my score a weak accept.
Thank you for your response and including the additional experiments. I believe the approach has merit (despite the concerns about computational constraints) and as such I will be increasing my score by 2 points.
Decision
Accept (poster)