Summary
The paper proposes a set of metrics to measure and isolate the memory dependency of partially observable environments.
The authors illustrate that some currently existing benchmarks do not sufficiently isolate the memory of an agent.
Based on their analysis they propose two versions of a T-Maze environment, one of which appropriately isolates memory.
Further, the authors investigate the effect of memory architecture on tasks that require long- and short-term memory.
Transformer-based policies outperform recurrent policies on tasks that require long-term memory dependencies,
while there seems to be no benefit on tasks that require short-term memory dependencies.
Strengths
**Significance:**
I fully agree with the authors that prior works often conflate different effects and do not sufficiently isolate the memory component.
Therefore metrics that help disentangle memory from other environmental effects are crucial.
**Originality:**
The authors propose novel metrics that evaluate and isolate different aspects of POMDPs in RL, such as credit assignment and long-term memory.
The proposed metrics provide an important measure on the effect of memory and will be useful to facilitate future benchmark design and evaluation of new algorithms.
**Quality:**
Quality of theoretical contributions is high.
Weaknesses
**Scalability:**
I credit the authors for mentioning the complexity for computing c^M as limitation, but it seems that the computation of other metrics is also limited.
For example, l_{mem}(\pi^\asterisk) requires access to the optimal policy.
For the minimalistic TMazes it is straightforward to obtain this policy, how is it obtained for more complex tasks such as Psychlab? Does it involve human experts?
How is it computed for procedurally generated environments, where the optimal policy varies between levels?
How would this scale to more complex environments, given that human demonstrations may not necessarily be optimal?
**Relevance of proposed environments:**
The authors propose two environments, namely Passive T-Maze and Active T-Maze to disentangle credit assignment from memory.
The Minigrid benchmark suite already provides a T-Maze environment that should exhibit the same characteristics as the proposed Active T-Maze, namely MiniGrid-Memory [1].
Passive T-Maze could be useful, but it is very minimalistic and only evaluates whether a small bit of information can be stored and carried across long timespans in an agent's memory.
There is no mention of the observation space, but providing, say, image-based observation for Passive T-Maze, would increase its complexity and enable application of vision-based methods.
**Interpretation of results:**
The authors mention that LSTM starts to falter at a memory length of 250 for Passive T-Maze.
Figure 2, however, shows that LSTM can solve the same environment with a memory length of 750.
This suggests that some other effects might influenced LSTM training, such as hyperparameter tuning.
Maybe the authors can improve the results of LSTM by further tuning of the method?
**Clarity:**
The authors should clearly state that the paper focuses on long-term memory and does not consider other aspects, such as memory capacity, or robustness of memory to noise.
**Missing relevant work:**
There is prior work that derives theoretical bounds on approximation error of history-based methods [2].
Further, other history-based approaches such as a hierarchical Transformer memory [3] and pretrained models as a memory module in RL [4], are not cited.
If applicable, adding these method in their experiments would strengthen the paper and might yield some more interesting findings.
[1] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic Gridworld Environment for OpenAI Gym, 2018. Publication Title: GitHub repository. https://minigrid.farama.org/environments/minigrid/MemoryEnv/
[2] Gandharv Patil et al., On learning history based policies for controlling markov decision processes. 2022.
[3] Andrew Lampinen et al., Towards mental time travel: a hierarchical memory for reinforcement learning agents. NeurIPS 2021
[4] Fabian Paischer et al., History compression via language models in reinforcement learning. ICML 2022
Questions
- The authors conclude that Transformers exhibit worse sample efficiency than LSTM on short-term memory tasks.
Is there an intuition why that would be the case?
On Passive Visual Match, which requires short-term memory according to the authors, a GPT reaches higher return compared to LSTM.
Could it be that the drop in performance is due to different observation/action spaces (continuous vs discrete) instead of long-term vs short-term memory?
- In line 304 the authors claim that the success on long-term memory tasks for GPT2 is consistent with the tendency to perform well in the supervised learning setup on large datasets.
The authors draw an analogy here which lacks support.
How do long-term memory tasks relate to the supervised learning setup on large datasets?
Do the authors imply here, that there are only long-term dependencies present in large datasets used for supervised learning?
Also, Morad et al. 2023 specifically advises against drawing comparisons between supervised learning and RL.
- What is the observation space for the Active/Passive T-Mazes?
- In line 270 it would be good to explain what the actual task is in Passive Visual Match, prior to discussion on the reward function.
- Figure 4 left shows superiority of GPT2 over LSTM, for Passive Visual Match the difference is not as pronounced, a significance test would benefit the interpretation of these results.
- Figure 5: it would be good to clarify what is meant by "optimal policies that lack (long-term) credit assignment"
I assume it refers to a markovian policy, since the return on T-Maze is 0.5.
- Do all figures show mean and standard deviation?
This should be mentioned in the figure captions.
- The last three sentences in Definition 1 could be moved to Definition 2, since they are about memory length and not context length.
- Equation in line 103: I think there should be $t - l_{ctx}(\pi)+1:k$ instead of $t - l_{ctx}(\pi)+1:t$ in the subscript after the conditional independence, right?
- The symbol k is reassigned repeatedly which is a bit confusing, the authors may consider using different symbols.
- Will the authors make the code for reproducing their results and computing their metrics publicly available?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
The authors have adressed limitations regarding computation of the metrics in more complex environments.
However, it is not clear from the paper, how well the other metrics scale with complexity of the environment, or how the optimal policies are obtained, i.e. does that require human demonstration?
In the case of procedurally generated environments (which are commonly used nowadays) would that require extensive "labeling" by humans?