Summary
This paper studies the problem of learning from histories in partially observable MDPs where a key question is how to design an architecture that is general enough so as to work on a large set of problems, yet specific enough to be sample efficient.
Inspired by the success of PID in the more classical control literature, the idea here is to design an architecture that is solely comprised of summation, attention, and linear operations.
Experiments show that on tracking problems, where PID is originally proposed for, as well as more complicated control problems, this simple architecture is good enough to achieve competitive results.
Strengths
The paper shows that using a complicated architecture is an overkill for many of the domains that the RL community considers as benchmarks. On this note I'd like to mention that, while the contexts are somewhat different, a similar conclusion can be made from "Towards Generalization and Simplicity in Continuous Control" Rajeswaran et al, where they show that simpler architectures such as linear models are equipped to deal with Mujoco Tasks. The difference is that in their paper they were considering reward-maximization in presence of the full state, whereas here the goal is learn a memory-like function in conjunction with performing well in the given task.
In any case, I generally like results that show simple stuff work, because simple things are easy to understand, implement, and execute.
Weaknesses
Overall I like this paper but I also have some concerns:
Fundamentally, I feel like the results in this paper are pertinent in terms of a bigger question, namely how rich we want our hypothesis class to be for learning a certain target. The solution adopted by the ML community nowadays seems to be: we really do not know how complicated a task is, so by choosing a very complicated architecture and the fact that we are at the mercy of universal function approximation capability, we can always in principle find the right fit. Moreover, the double descent phenomenon ("Deep double descent: where bigger models and more data hurt", Nakkiran et al) tells us that empirically we are very likely to find a good fit with larger and more complicated models and that, despite more classical belief, such an overparameterized fit happens to also be quite robust. ("A Universal Law of Robustness via Isoperimetry" Bubeck and Sellke)
Now, moving to the RL setting, these insights from supervised learning do not really carry over because we usually learn by bootstrapping off of Bellman targets, and by increasing the complexity of our function approximator, we also have to learn from increasingly more complicated targets (unlike supervised learning where targets are fixed).
So I think, at least with our current RL algorithms, unlike SL we always need to choose our function class carefully and so adding inductive bias, such as ones provided in this paper, can usually be helpful. My worry is that adding such biases are helpful in a limited sense: It is not clear how much inductive bias should I add for a given task when I don't know too much about that task. Sure, I can always choose environments and problem settings where I know that my inductive bias is conducive to solving the problem, but do we really want to bring back the burden to ML practitioners to carefully think about the kind of inductive biases that are useful for their task?
Grounding my concerns more so in the experimental results provided here, for example we see that the inductive bias provided is already giving small (or negative) gains as we move to some of the more complicated domains such as HalfCheetah and Ant (based on Figures 18 and 19 in the Appendix), so I ask authors, are you not worried that as we move to much larger domains then it becomes increasingly more difficult to show performance gains for GPIDE compared to other approaches? In other words, does the inductive bias scale reasonably with problem complexity or there is only so much this could be useful?
To further play devil's advocate, to say that GPIDE is capable of beating the transformer architecture is sort of a weird claim, because as authors say GPIDE could be thought of as a special case of transformers, and further as you show the attention layer is often not very useful. This tells me that with proper hyper-parameter tuning and regularization, the transformer architecture should in principle be able to always do better than GPIDE, and we can still hope to find the transformer useful as we move to larger problems.
Questions
I would like to ideally see an acknowledgment that as we move to much more difficult domains, GPIDE may be incapable of doing well simply because the actual function to be represented in not in the function class. But such an example is currently absent. This may give a false sense that the paper is claiming that GPIDE is a general-purpose memory learner but that is not true, and I am not saying that authors say it is true, but having examples of failure cases would really clarify this point.
If I may say, I would also like to see experiments on different types of problems. For example, can we expect GPIDE to perform reasonably on image-based domains? Like the paper mentions, we can hope to apply the GPIDE on the embedding space learned by some more capable function approximator, but it seems to me that to learn good embeddings that are useful for downstream memory learning, one needs to leverage some kind of GRU/Transformer/LSTM architecture, and so it is not immediately clear to me how to leverage GPIDE in this setting.
My other question is also that as we know RNNs struggle to learn important events from the distant past because the vanishing/exploding gradients. LSTMs and Transformers are less prone to this issue but I wonder if we could better compare this capability of GPIDE in contrast to the existing architectures. In the Half-Cheetah experiment, for example, do we have a good sense of how far back one needs to go to be able to accurately learn the position and/or the velocity signal?
Also, from the intro the paper reads "Another hurdle stems from the fact that policies are often trained in an imperfect simulator, which is likely different from the true environment. Combining these two challenges necessitates striking a balance between extracting useful information from the history and avoiding overfitting to modelling error."
My thoughts reading this sentence was that we are going to either have actual sim2real experiments or some kind of a transfer learning benchmark, where we learn the memory function in some settings, and test the learned memory function in another test setting. This would have been more convincing in terms of claiming that GPIDE is not prone to overfitting. Did I get it right that this experiment is absent?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.