Semantic HELM: A Human-Readable Memory for Reinforcement Learning

Summary

This paper addresses the problem of partial observability by proposing a semantic-enhanced method. This approach converts environmental observations into human-readable language tokens and incorporates them into the hidden state embeddings. Relative to other methods, this technique exhibits resilience against representation collapse and is sufficiently explainable, making it viable for high-stakes domains.

Strengths

(1) The presented method is concise and straightforward to implement. Moreover, its potential for broad applicability across diverse domains stems from the inherent generalizability of the foundational model (CLIP). (2) The paper provides a comprehensive review of foundational knowledge, enhancing its readability for varied audiences. (3) The extensive experiments conducted show the effectiveness of the Semantic HELM approach.

Weaknesses

The proposed method, which embeds a subset of predefined tokens as semantic descriptions of each observation frame, appears to be a rudimentary solution to the partial observability problem. Put simply, it may only be suitable for simplistic environments that rely on recognizing specific objects. In comparison to a basic model that labels the current frame and concatenates the observation with its labels—thus serving as input to a memory-based policy—I don't see this method will stand out. If the paper aims to show that a language-based memory adequately serves as policy memory, it would be more beneficial to delve into the specifics of description design rather than utilizing a sequence of independent objects. The paper should also engage with environments where past events significantly influence current decisions. I further question whether a singular language-modality memory is sufficient for more complex environments. For example, in three-dimensional video games like Minecraft, the absence of a visual-modality context may lead to inconsistent behavior sequences.

Questions

As stated in the weakness part.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This paper contains the limitation part including the modality gap and captions diversity. I believe the author should dive more into the caption design of history observations for more complicated environments.

Summary

This work introduces a a memory mechanism for RL agents that relies on foundation models but does not require training for correct functioning. The main contribution of this memory architecture is that memory tokens are preserved in a "natural language" space, making easier for humans to interpret the decision process of the agent

Strengths

This work addresses long-term memory and interpretability, two challenging yet core research areas in the RL field. Authors provide a very deep and broad empirical evaluation and analysis of their method and the baselines, highlighting clearly why each of them robust or weak on the different tests. The fact that the conclusions are drawn over 4 very different environments with 30 independent runs makes the reader confident on the results obtained. The ablation study included also reinforces the results. The work does also a broad coverage of the existing literature and related works

Weaknesses

The main weakness of this work and the reason why I cannot give a stronger mark with the current version is the clarity of the method and it's differences with HELMv2 which this work is heavily based on. Specifically I would extend section 2.2: * I would expand figure 2, where the method is explained, to highlight what are the key pieces that change from HELMv2. The paper does refer to HELMv2 and the main differences at a high level, but I think that including a depiction would greatly help the readers as I had to go several times through 2.1 and 2.2 to understand the differences * Lines 100-101, you state "Instead of merely fitting the statistics of the respective embedding spaces we introduce 101 a token-retrieval mechanism that preserves the semantics extracted by CLIP" I think this requires detailed explaining * Lines 108-113 the steps are clearly explained but I miss the "why" are the authors introducing those changes, what each of them is trying to achieve * Line 119, you were explaining memory compression level then follow with "by performing this procedure at every time step, we build up a human-readable representation of the history of the current episode" I would reiterate here what is making thins human readable (the representation, not the compression) and again contrast it with HELM. To allocate space for the depictions and explanations I mention above I would merge parts of the experimental discussion or move some pieces to the appendix. I also think that authors give a lot of weight on interpretability of the method but then the work lacks doing a proper analysis of it as I note in the limitations box below

Questions

I didn't see mention of it in the paper but authors provide the code in the suplemental material, is there intention to publish it whit the paper? ------ Post rebuttal I believe authors have correctly addressed my concerns and I am updating my score

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Authors address some limitation at the end of the work but I missed some further discussion and study on the interpretability of the method and it's limits, specially since authors state interpretability as one of the core goals of the work. Specifically, I miss authors including some in-depth analysis on how interpretable the decision making really is rather than relying so heavily in the assumption that it is interpretable because language is used in the memory stored.

Summary

This study presents Semantic History-Embedded Language Model (SHELM), a new interpretable memory mechanism for reinforcement learning (RL) agents in partially observable scenarios. Current memory methods are often uninterpretable for humans, which hampers their use in critical areas like autonomous driving or healthcare. SHELM addresses this by using human language to clarify the decision-making process. It employs CLIP to convert visual data into language tokens and a pre-trained language model to create an interpretable past record. Despite limitations in synthetic environments, SHELM performs well in photorealistic settings and shows robustness to environmental disturbances.

Strengths

The paper's key strength is its development of the Semantic History-Embedded Language Model (SHELM), an innovative memory mechanism that brings improved interpretability to reinforcement learning. This is crucial for deploying RL agents in high-stake applications where human understanding of the decision-making process is vital. SHELM's ability to link visual inputs with language tokens makes the agent's memory easily understandable for humans. Moreover, the method demonstrates state-of-the-art performance in photorealistic environments and superior robustness to environmental noise compared to existing approaches. In addition, the narrative of the paper is very clean and clear, making it easy to follow.

Weaknesses

- There are minor typos in the paper that should be corrected for revision. E.g. in line 56: "a interpretable" -> "an interpretable" - In the last paragraph of the Introduction, there is a claim saying: "On Avalon we find that a semantic memory does not yield any benefits. In fact, we observe that agents with memory tend to learn faster, but reach lower final performance than a markovian baseline." I am wondering if there is any explanation for this. I also could not find where it has been discussed in the Experiments sections. Could you please provide a pointer for this? - How can we ensure that the actor-critic head does not ignore the features passed to it from the TrXL? This means that the agent may make decisions solely based on features passed to it from the CNN module. Even if this happens at very few decision steps, then this could lead to serious questions about the idea of the paper and the extracted insights. - This work endeavors to mitigate the black-box nature of deep RL agents through the utilization of a black-box tool (LM). I would use the metaphorical concept of "fighting fire with fire" to describe this paper. As discussed in the paper, the ultimate aspiration is to ensure interpretability throughout the decision-making process when applying such methodologies in practical scenarios. However, the lack of interpretability and transparency in the LM raises profound doubts regarding the fundamental premise of the paper. - From Figure 3, SHELM is only outperforming baselines on the MiniGrid Memory task. On other tasks, Either HELMv2 or Dreamerv2 is better. Is this a correct observation? - In line 145, there are references to Figures 8 and 9. But these figures are not in the main paper. I would suggest moving the discussions of those figures to the appendix where they are. - From the results of the paper, they are not as convincing as promised in the abstract. Could you please clarify this?

Questions

See weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

See weaknesses.

Summary

This paper introduces Semantic HELM, a memory mechanism for encoding an agent's past observations into a semantically meaningful space and recalling from the memory. It is an extension/modification of HELM, a previous work that uses a language model to encode past observations, but unlike prior methods, Semantic HELM's tokens are interpretable (i.e. semantically related to the observations) rather than random. The paper shows comparisons to prior methods on four environments, including ones requiring strong memory to complete successfully.

Strengths

I think the paper is extremely thorough in its experiments. It evaluates on four different environments and against reasonable baselines such as the previous HELM works and Dreamer. It even shows experiments where it is about par with other methods rather than cherrypicking environments where theirs does better. On the Memory and PsychLab environments, which seem the most explicit about requiring memory, this method does perform extremely well compared to baselines. The method does have some natural advantage on PsychLab because it has the benefit of the knowledge from CLIP, but HELMv2 also has CLIP encodings, so this isn't an unfair advantage. The idea is interesting and makes sense. I think the motivation for having a memory that is (somewhat) interpretable is pretty strong. The ablation studies are really interesting and alway very useful and careful.

Weaknesses

The interpretability claims should I think be somewhat caveated in the paper. As the authors acknowledge, SHELM can in fact end up learning associations that are not quite semantically correct (as stated in line 224). I get what the authors are saying here that overall it is much more aligned semantically than prior methods, but I think some of the statements about interpretability in the intro should be somewhat hedged. I did not quite understand from the paper how the vocab was determined and it seems potentially limiting. Reading section 2.2, it looks like it finds the token overlap between the LM and CLIP and appends a custom prompt. First, does this imply that you cannot encode anything longer than a single language token (not including the prompt?) That seems like a big limitation if true. Also, how are the prompts decided in each environment (is there a table somewhere)? This is some additional information that needs to be hand defined per envionment.

Questions

See above for questions about vocab.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

See above - possibly some limitations about the vocab that I think the paper could address better.

Summary

The paper looks at using a compressed representation of recent observations in the form of top-k CLIP-extracted text entities. They look at training policies in a few different simulated environments with this representation. They cast this as an extension of the prior "HELM" method, although their key idea does not need to necessarily be paired with HELM.

Strengths

The overall discussion, paper writing, introduction, and experimentation, all seem pretty "solid". The authors seem to be devoted to a genuine scientific exploration and they build rapport with the reader such that they believe the experimentation and investigation is earnest.

Weaknesses

There are a couple main limitations in mind for me of the overall method: 1) In the first place, the entire reason used to motivate the investigation was to add interpretability. However, I'm not sure that a small list of short words, as extracted by CLIP, is a huge value-add for interpretability, since the downstream policy is still black-box. It is possible that the downstream black-box part of the policy model learns correlations with the text features that are not as a human would interpret them to be. This limitation is not really investigated empirically. Instead, after the introduction, all of the results are about performance on the tasks and not interpretability of the performance on the tasks. 2) Another huge limitation that is not discussed is that the proposed method would only be expected to work well in environments where there is a "single or few things" observable at a time in an image, and the mere existence of these objects is all the downstream policy needs to know. The reason is that CLIP can be used okay as a zero-shot object detector, but it is not good at fine-grained understanding of environments. For example, robot manipulation environments would not likely be well helped by the method. I consider the above 2 to be primary limitations of the method and my recommendation is primarily driven by those 2. Here are a few other points to consider for the authors as they revise and for future work: - The term "memory" is repeatedly used here. However, I'm not sure this is the best word choice for the author's method. The authors are not creating any type of persistent memory mechanism. Instead, the method only serves to compress past K observations. The method works by operating on the past K observations, and compressing them all down to the top-k CLIP words. Since nothing persistent is created, it's not necessarily a memory mechanism, I would argue. - "Markovian baseline" -- I know what Markov means, but I'm not sure exactly what they mean by Markovian baseline. I think they mean that the baseline is memoryless and operates on only the current observation, which would assume that the environment is Markov, even though it's not. But it also could mean that they give the agent full observability so that it is indeed Markov. It's not clear and should be clarified, moreover "Markovian baseline" is likely not the best choice to use as shorthand for it, even though it seems like the authors might have done this before. - I was about to suggest that an interesting baseline would be to, instead of using the top-k CLIP text entities, to instead just train a policy on the underlying CLIP vision embeddings themselves, but then I realized this seems to essentially be what HELMv2 was.

Questions

Do each of the discussed weaknesses make sense?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations section in my opinion is incomplete since the authors do not address the primary limitations I pointed out in the "Weaknesses" discussion. There is no discussion of broader impacts.

Summary

Paper presents a method to augment a feed-forward RL policy with an “interpretable” memory. In general, it is based off GTrXL and HELM, which uses transformer-XL (TrXL) to encode current visual observation (or further map it to some embedding through embedding look-up, CLIP encoder, etc) while paying attention to some past observations (through the memory mechanism in TrXL). Finally, both the output of TrXL (reterived memory) and current visual observation embedding are fed to actor and critic heads. The proposed method replace CLIP embedding with a bottleneck in explicit text, which is claimed to be more “interpretable”. The proposed method is evaluated on several domains again some baselines and ablations.

Strengths

+The paper is generally clear and well-written. Although there seems to be many legacies in this direction (HELM, HELMv2, etc), the authors clearly explains how their method is different from these prior arts, especially on what motivates the proposed changes, making it friendly to readers who might not quite familiar with memory in RL agents. +The study problem is relevant to the interest of NeurIPS. The use of language model, CLIP and other foundation models in RL should be of interest to many audiences of this conference, which could generate possible impact. +The proposed method generally makes sense. Instead of sending CLIP embedding directly to TrXL, introducing an explicit text bottleneck does bring more interpretability, and the results does not seem to indicate significant drawbacks from the previous model using implicit embeddings. +Results on lots of challenging domains (minigrid, MiniWorld, Avalon, PsychLab, etc) with ablations seems to justify the claims well.

Weaknesses

Having said those above, I do have some concerns on baselines and results, listed below: -Although the authors mentioned four domains, no all baselines (PPO, PPO w/ LSTM, DreamerV2, HELM v1/v2, etc) are tested on all these environments. Ex., there is no results with DreamerV2 on Avalon. Can the authors comment on that? Also, I couldn’t find more justifications on the selection of these baselines – is the main goal of experiments to validate the effectiveness of memory mechanism in RL agents, or simply demonstrate that the proposed memory suites the tasks better? What is the state-of-the-art agent in the four environments? These are the points that needs to be further elaborate to help us understand the significance of the results. One specific point: since DreamerV3 has been released, is there a certain reason to only include results from its precedent? -One main contribution made by this paper is to upgrade the CLIP + TrXL memory in HELMv2 to a more “interpretable” version. This raises two issues that do not seem to be addressed in the paper: 1. Replacing the CLIP embedding in HELMv2 with some explicit text indeed introduces a semantic bottleneck in the memory pipeline, as the visual observations are now compressed into even lower bit discrete text instead of dense embeddings. In theory, this could bring some drawbacks on providing information of the past. Some of the results do indicate this, ex. Slower convergence rate on MiniGrid, worse results on MiniGrid-Memory but some do not, ex. PsychLab. Can the authors explain more on how could such differences emerge? In fact, description of each domain could have been more precise to allow more space for analysis. Specifically, analysis on domain-specific impact on convergence and end performances on different memory-based (w/ or w/o language bottleneck) or memory-free agents should be included. 2. Interpretability seems to be a major matter of concern in this paper, however, I couldn’t find any results, metrics, or analysis in the main text dedicated to demonstrating the interpretability of the proposed method (with only an illustration in Figure 1, but I don’t think it count as a result). How interpretable can the proposed memory be? Does it indeed exhibit some interesting interpretable behavior? What is inside the memory when run on these four domains? Compared to baselines, does user trust this system more due to interpretability rather than better results? These are all questions that need to be answered to justify the interpretability of the proposed method. -Here are some ablations/baselines that could be missing: - GTrXL (transformer-XL instead of LSTM for RL) - HELM/SHELM w/ Different CLIP models (RN50, ViT, etc) - Fine-tune TrXL (it is frozen in SHELM no?) - Simply concat t-1 observation (CLIP embedding or text) with current visual observation Minor: -Which env is used for ablations?

Questions

See [Weaknesses] --- ### Post-rebuttal Thank you for the detailed reply and the additional results. My concerns on baselines and limitations are mostly addressed. I will raise my score to 5. I will consider raising my score even further if the remaining questions can be addressed. --- Raised my score to 6 after further discussions with the authors.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

To ensure interpretability, the authors have to pre-define a set of sentences, which effectively limit the scope of visual observations that can be stored in the memory. Maybe using a generalist caption model like BLIP-2 could help mitigate this?

Thank you for your response

I want to thank the authors for their response and further clarifications. I believe the additions further clarify the contributions of this work, I am updating my score accordingly