Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

Summary

RL has a notorious sample efficiency problem, and the authors propose to tackle this problem by having RL agents read instruction manuals. The authors propose a novel framework called Read and Reward (R&R), which is composed of two modules. The first module is QA which extracts and summarizes relevant information in the manual, and the second module is reasoning which evaluates events in the game and provides appropriate rewards for the RL agents to learn with algorithms such as A2C and Agent 57. The authors found R&R to significantly improve the agent's sample efficiency in several challenging games, such as Skiing.

Strengths

The paper is well-written and has the following strengths: * **Interetsing approch**: as the authors mentioned, this is the first work that demonstrates improving RL performance by reading instruction manuals. Many pure RL methods suffer significantly in Skiing, so seeing other approaches that improve training is encouraging. * **Further contribution to NLP + RL**: the authors further explore the space of using NLP to improve RL agents. This line of research will help reduce the need to train agents from scratch and provide useful priors to the agent. * **No labeling required**: the amount of human labor required in the proposed framework is minimal — they only provide some generic questions, such as "what is the objective of the game?"

Weaknesses

Despite its strengths, the paper has the following weaknesses: * **Adhoc setting**: the setting seems ad hoc, stitching together various modules such as QA and object detection. A multi-modality model such as GPT-4 can probably serve as a general replacement for these modules, unlocking new capabilities, such as learning from image/video tutorials, in addition to instruction manuals. That said, there has not been an excellent open-source multi-modality model available. * **Slightly insufficient evaluation**: the authors only tested their approach on four games, but maybe it's worthwhile to evaluate some other games, such as Montezuma Revenge, which RL algorithms usually struggle with.

Questions

I'm curious about the delayed reward schedule. Would it not artificially create a sparse reward problem? Misc: * The citation format seems slightly problematic. `However, in many real-world scenarios (Kolve et al., 2017)` feels more common and readable compared to `However, in many real-world scenarios Kolve et al. (2017);` * Lines 44 and 48 seem repetitive.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Summary

The authors introduce Read and Reward framework, where RL agents accelerate their learning of a new environment by interpreting user manuals. Specifically, the framework consists of a QA Extraction module that extracts and summarizes relevant information and a Reasoning module that evaluates object-agent interactions based on the information. The algorithm accelerates learning by using auxiliary rewards, whenever the interactions are detected by the reasoning module. The method empirically boosts the efficiency and performance of many Atari games.

Strengths

The topic in focus is very relevant because of the recent surge in the abilities of LLMs. The solution of utilizing LLMs to map auxiliary rewards in RL is novel. The performance of speeding up RL training for the four games investigated is apparent. The ablations performed are thorough and complete.

Weaknesses

- document length and unstructuredness are overstressed in the paper as a method/contribution. However, a) many NLP papers have already addressed them, and the method employed in the paper is simple summarization; b) long context models are already available to ingest more than the text needed to reason - the overall idea is quite nice; however, due to the difficulty of grounding knowledge in the environment, the paper only considered "hit" interactions and required unsupervised object detectors (which might be unreliable) OR game states, and the interaction is quite primitive in many scenarios, therefore cannot describe most tasks/useful interactions. - Table 1 needs error bars/std

Questions

- have you tried using VLMs to do the reasoning? instead of object detectors. This way the set of interactions can also be expanded - does the scale of the aux reward matter in the games investigated? - is there an intuition on the speedup ratio (Table 3) Vs. game? (Breakout appears to really benefit from R&R)

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Both limitations and broader impact sections are included in the draft

Summary

This paper proposes Read and Reward (RR), a method to incorporate prior human knowledge about the environment to achieve performance and efficiency gains in RL environments. The paper instantiates RR in several Atari environments by reading the information from the instruction manual. A full end-to-end pipeline is demonstrated on Skiing and a pipeline without object detection is demonstrated on Tennis, Pacman, and Breakout.

Strengths

*Originality*: While incorporating prior knowledge into RL training is a previously investigated topic, this paper provides an end-to-end pipeline and demonstrates its usefulness on Atari Skiing. The method introduced is more general than methods previously investigated. *Quality*: There are thorough experiments with ablation studies, such as training the QA module on a different set of instructions. *Clarity*: the paper is well-written. *Significance*: as RL agents are eventually deployed in real-world scenarios, it will be increasingly important to have them learn efficiently and incorporate prior knowledge, which is often in the form of text. This paper provides a complete demonstration of a method leveraging natural language data to improve performance. More generally, it sketches out a framework for exploiting external data that future algorithms may build upon. Such agents may be rapidly deployed to learn in an unsupervised fashion.

Weaknesses

Although RR is an interesting proof-of-concept of automated reward shaping, it is unclear whether it scales to more complex environments or works on noisier sources of textual knowledge. In particular: 1. Does RR generalize to noisier sources of textual data? While the Wikipedia ablation is interesting, the data itself is rather clean and contains similar information. One interesting experiment to try might be with text-based RL environments, such as the Jericho [1] or Machiavelli [2] environments. Oftentimes these environments have publicly available [walkthroughs online](https://forum.choiceofgames.com/t/guides-for-all-games/15569/4), although they are often incomplete and noisy, as the instructions are intended for humans. Is RR able to extract the rewards and guide the agents in this setting? (I realize that this would be a significant undertaking so am not expecting it in the rebuttal) 2. The reward shaping only surrounds object detection. As a result, the QA prompt, even though it is hand-designed, still generalizes. However, there are several RL environments where the rewards are more heuristics rather than strict rules. For example, dialogue in the Diplomacy environment [3] does not have clear positive or negative reward but instead depends on the context. One could imagine gaining better understanding of dialogue in Diplomacy by leveraging knowledge on the internet, but it is unclear how RR might be applied in that scenario, as the reward shaping must center on text and the QA prompt likely could not be hand-designed. 3. It seems like it was quite difficult to train a bounding-box detector for most games. However, this would imply that RR's capabilities are bottlenecked by limitations of such bounding-box detectors. This might make RR difficult to scale to more complex environments with more ambiguous objects. [1] Hausknecht, M., Ammanabrolu, P., Coté Marc-Alexandre, & Yuan Xingdi (2019). Interactive Fiction Games: A Colossal Adventure. CoRR, abs/1909.05398. [2] Pan, A., Chan, J., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., & Hendrycks, D. (2023). Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark. ICML. [3] Meta Fundamental AI Research Diplomacy Team (FAIR) et al. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science 378,1067-1074 (2022). DOI:10.1126/science.ade9097

Questions

1. Would it be possible to have more results on other Atari games? I understand that the bounding-box detection was challenging, but are Tennis, Pacman, and Breakout the only other Atari games that allow for ground truth calculation of the objects from the RAM state? 2. How robust is the automated reward shaping to the quality of the bounding box detector in Skiing?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

While some limitations are addressed, it would be interesting if there were additional discussion about the limitations of extracting information from natural language data. For example, humans understanding of the physical world may not be adequately encoded in natural language. Does this limit the application scope of agents using the RR method? Additionally, if the LM's reasoning abilities are erroneous, will this irreparably break RR? Finally, it would be helpful to directly note that as RR is fundamentally a reward shaping method, it is difficult to apply it to situations where the reward itself is unclear. This happens often with delayed reward. In particular, RR would likely not improve performance on Go or Chess.

Summary

This paper presents a novel method for Single Agent Reinforcement Learning which utilises computer game instruction manuals to enhance learning efficiency and performance. The Atari game manuals are used to accelerate RL algorithms in learning to play four different games. The framework comprises a Question-Answer Extraction module and a Reasoning module. The QA Extraction module summarizes salient information from the manual. The Reasoning module evaluates in-game events using the extracted information from the manual and then assigns auxiliary rewards when such interactions are detected. These auxiliary rewards are provided to standard RL agents. The paper's results show improvements in both performance and training speed for various RL algorithms when aided by this novel framework. It is asserted that this is the first successful attempt to use instruction manuals in a fully automated framework for solving the Atari RL benchmarks. It is noted that even small open-source QA language models could effectively extract useful gameplay information from human-written documentation for RL agents.

Strengths

The results in this paper are certainly extremely promising, and the ability to use large language models to create additional rewards from the instruction manual is very promising. That such methods are so much more sample efficient is clearly going to be important in realms beyond simple game play.

Weaknesses

The main, and very problematic issue with this paper is the lack of robust evaluation. There are no statistics given, no averaging over random seeds provided and no code made available. It is thus entirely unclear how much the results themselves can be trusted under scrutiny. Thorough and transparent evaluation is absolutely vital in all RL scenarios, and because this has been left out completely, this paper is, I believe, not ready for publication.

Questions

Without a full analysis of the variation in results over a large number of random seeds, how can the results be trusted?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

2 fair

Limitations

I believe that the limitations provided are fair.

No attachment

There appear to be no revisions as yet. Clicking on the button on the rebuttal does not link to the paper with additional experiments.

Thank you for the response

While the method is a bit problem-specific, I still think it is an interesting demonstration of incorporating prior knowledge into RL environments. I would like to keep my score

response

I thank the authors' clarification on the choices of Atari games.