NetHack is Hard to Hack

Summary

This paper seeks to study and better understand the large performance gap between neural and symbolic agents in the NeurIPS 2021 NetHack Challenge. The main hypothesis is that symbolic agents advantage derives from hierarchical reasoning, which was not an element in participating neural agents. To test this hypothesis, a new dataset is generated using the winning symbolic agent that records both actions and higher-level strategic labels and a neural behavior cloning agent was trained with this augmented data. Beyond this, the paper also explores the impact of increased model size and dataset size, changes to the neural architecture, and the addition of a policy fine-tuning step using a reinforcement learning algorithm. The results suggest that hierarchical training improves the neural agent's performance significantly more than increased model capacity but that more powerful model architectures (i.e. a transformer-based model) could overfit to the augmented data.

Strengths

Quality/Soundness The hypotheses and claims of the paper were laid out clearly and the experiments were well-designed to evaluate them. Clarity/Presentation I found the paper to be clearly presented and well-written. At each stage I had a clear understanding of the question under investigation and the methodology for studying it. Originality/Contribution The paper is explicit that its main contribution is not algorithmic but scientific. Since the scientific questions posed here are grounded in the performance of agents from a specific competition that happened last year, I expect that this analysis is original. Significance/Contribution Increasing understanding of the performance gap in the NetHack problem, as a proxy for complex, long-horizon problems in general, could be important. Symbolic approaches make use of quite a lot of domain expertise applied to constructing the symbolic structure, making them effective in their target problem but inflexible. Neural networks seem to be quite flexible and capable of learning without a lot of structure engineering, but don't seem to be able to take advantage of structure in the environment. It seems sensible to study the primary factors preventing neural approaches from building the same long-range structures. I think it is worthwhile to see the comparison between this structural change and the alternative interventions of more data, more parameters, or more sophisticated/expensive architecture. The finding that, in a time-constrained setting (true of many decision-making problems), model structure may be more important than model capacity seems likely to at least spark interesting conversations within the community. The fact that there is plenty of performance gap still to cover, even with this built-in domain knowledge may also inspire further investigation into what measures could come closer to closing the gap and how those insights might be applied to more general practice. Overall Overall, I find this to be a clearly written paper with a reasonable scientific question and sound methodology to address it (modulo some missing statistical analyses). The findings are not revolutionary but, to my eyes, they do provoke further questions about neural approaches to learning in complex, long-horizon problems and may inspire follow-up work either in the NetHack testbed specifically or in studying these questions more generally.

Weaknesses

Quality/Soundness My main concern in this area is the small number of independent samples per model class (6). I do understand that these results are generated at great expense and that it may not be feasible to generate more trials, but the small number of samples diminishes the statistical power of these analyses. I can see that the error bars are quite small; assuming those are showing the standard error, that's encouraging. Nevertheless, whether more samples can be generated or not, I think it's important that the paper include the findings of low-sample hypothesis testing (e.g. t-test) on these results; without that, we can't confidently distinguish between noise and meaningful differences. Originality/Contribution The paper acknowledges that behavior cloning, hierarchical policies, and transformer-represented policies have all been studied in prior work. Significance/Contribution NetHack is not, in and of itself, an intrinsically important problem to solve. The results presented here are not conclusive or enormously surprising. The main result is fairly predictable: adding explicit supervision about high-level strategy and explicit hierarchical structure in the model helps the model take advantage of hierarchical structure in the environment. Overall Overall, I find this to be a clearly written paper with a reasonable scientific question and sound methodology to address it (modulo some missing statistical analyses). The findings are not revolutionary but, to my eyes, they do provoke further questions about neural approaches to learning in complex, long-horizon problems and may inspire follow-up work either in the NetHack testbed specifically or in studying these questions more generally. ---After discussion--- I have considered the other reviews and the authors' responses. I continue to feel confident about my overall assessment.

Questions

n/a

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Since the paper does not propose significantly new algorithmic ideas, the main source of limitations would be in the methodology and analysis. I generally found the paper to avoid overclaiming. I've already discussed one area where this aspect of the paper could be improved: acknowledgement of the small sample sizes and proper statistical analysis to inform the conclusions. The other area might be in the conclusions where perhaps the summary of the findings might be a bit too general and could be toned down and/or stated clearly as hypotheses (e.g. "Hierarchy hurts overfitting models" is an overly broad conclusion from a limited set of experiments but seems like a reasonable hypothesis given these results).

Summary

# Problem Statement The paper addresses the challenge of neural policy learning methods struggling in long-horizon tasks, particularly in open-ended environments with multi-modal observations, such as the game NetHack. It was observed that symbolic agents significantly outperformed neural approaches in the NeurIPS 2021 NetHack Challenge. # Main Contribution The paper's main contribution is an extensive study on neural policy learning for NetHack. The authors analyzed the winning symbolic agent and extended its codebase to generate one of the largest available demonstration datasets. They examined the advantages of an action hierarchy, enhancements in neural architecture, and the integration of reinforcement learning with imitation learning. Their investigations resulted in a state-of-the-art neural agent that surpassed previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, they also demonstrated that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players. # Methodology and Experiments ## The Hierarchical HiHack Dataset The authors create the HiHack dataset, which is a hierarchically-informed version of the NetHack Learning Dataset (NLD-AA), containing 3 billion recorded game transitions from over a hundred thousand games played by the AutoAscend agent. ## Hierarchical Behavioral Cloning The authors extend the ChaoticDwarvenGPT5 (CDGPT5) model, a top-performing open-source neural model for NetHack, by introducing a hierarchical decoding module. The model consists of three separate encoders for different types of observations and an LSTM core module. The hierarchical extension replaces the linear decoder of the CDGPT5 model with a hierarchical decoder that predicts the strategy label and selects the appropriate low-level MLP for action prediction. The hierarchical LSTM policy and the baseline non-hierarchical LSTM CDGPT5 policy are trained using a simple cross-entropy loss. The results show that the introduction of hierarchy significantly improves the performance of LSTM policies trained with behavioral cloning, yielding a 40% gain over the baseline in mean NLE score and a 50% improvement in median score across seeds. The authors confirm that this improvement is due to hierarchy and not simply a result of the increased parameter count of the hierarchical LSTM policy. ## Architecture and Data Scaling The authors explored scaling as a potential solution to improve the performance of the model, which was significantly behind the symbolic policy used to generate the HiHack demonstrations. They developed a novel base policy architecture for NetHack that introduces a Transformer module into the previous CDGPT5-based architecture. They also conducted data scaling experiments using subsets of the HiHack dataset to examine the relationship between dataset size and the test-time performance of BC policies. The results showed that both the non-hierarchical and hierarchical variants of the combined transformer-LSTM policy architecture yielded gains, but the larger model performed worse than the smaller one due to overfitting. This suggested that scaling of model capacity alone would not be sufficient to close the neural-symbolic gap. Additionally, brute force scaling of the dataset alone could not viably close the gap to symbolic methods. ## Combining Imitation with Reinforcement Learning The authors explored combining imitation learning with reinforcement learning (RL) to bridge the performance gap with AutoAscend. They used a combination of behavioral cloning (BC) and asynchronous proximal policy optimization (APPO) for training. The results showed that RL fine-tuning significantly improved the performance of all models. The best-performing approach was APPO + BC using the hierarchical LSTM model, which achieved a new state-of-the-art for neural policies on NLE, surpassing the previous best result by 48% in mean NLE score and 25% in median NLE score. The Transformer-LSTM models performed worse due to their slower training speed and the fixed training time budget. The authors also observed that fine-tuning with RL improved the error-correction capability of models across all model classes compared to their purely offline counterparts.

Strengths

# Originality The problem is interesting and the approaches are insightful. # Quality The analysis and experiments are comprehensive. # Clarity The article is overall well written and clear.

Weaknesses

1. The current focus of the study is quite narrow, being primarily centered on the application of imitation learning for NetHack, limiting its influence. In the context of mastering the game, while this approach is interesting, it is unlikely to exceed the performance of experts that generate demonstrations, not to mention that the experts are already algorithms that can scale well. Furthermore, NetHack, despite being an excellent game, is somewhat niche and its real-world implications are relatively minimal. The techniques proposed in this study are specifically tailored for this game, which limits their potential for inspiring more universally applicable methods that could have a broader impact. - The availability of hierarchical labels is a strong assumption that does not often hold, which further limits the applicability of the proposed methods. 2. Even just for bridging the performance gap between neural models and AutoAscend, there is no promising direction revealed by the work as the various augmenting components seem to contradict each other.

Questions

1. When introducing Transformer to augment the capacity of the neural model, why did authors choose the architecture as shown in the article? Specifically, transformers are best known for their NLP and CV capacity, which could make them good replacement for the CNN and MLP encoders. 2. Why do the authors enforce the 48 hour training time cap instead of training all models till convergence? Given that this study does not appear to prioritize data efficiency or training efficiency, the necessity of such a computational time constraint is unclear. It would be beneficial to understand the rationale behind this choice, as it may not directly align with the study's primary objectives.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors note that possible avenues for future exploration include: (a) methods for increasing the Transformer context length to give the agent a longer memory to aid exploration; (b) addressing the multi-modal nature of the demonstration data (i.e. quite different trajectories can lead to the same reward), which is a potential confounder for BC methods. Some forms of distributional BC (e.g. GAIL, BeT) could help alleviate this issue. The aforementioned two points do not address the limitations raised in the "Weakness" section.

Summary

The paper improves the existing solutions in the NetHack Learning Environment (NLE). This is done by taking earlier solutions from a competition around NLE, collecting more data with the best available (symbolic) agent, and using that data to improve a neural only solution. The paper provides experiments with imitation learning (with or without RL tuning), larger models, hierarchical memory setup (LSTM + Transformers) as hierarchical behavioural cloning setup, using labels of the newly collected dataset. While there are improvements, it is still below the demonstrator results, which is then studied by scaling the model sizes and amount data collected. Paper concludes by providing the state of the art results in the task, but also noting that scaling alone is not enough to reach the expert demonstrator level (symbolic agent).

Strengths

- Provides more detailed dataset than the previous works (with hierarchical action labels) - Sets an interesting premise/task for trying to reach the demonstrators' (AutoHack agent) performance with neural solutions. - Different ablations to try to answer questions (data/model scaling, model architecture with hierarchy) - Proposed hierarchical approach to imitate the demonstrator agent.

Weaknesses

While I enjoyed reading the paper, overall I think the results are interesting or applicable to most of the NeurIPS audience, even in the limited scope. The paper presents many results and provides some explanations for them, but does not verify these explanations with further experiments. I think proper answers to these issues would be insightful to many, and others could then use these insights in their work (e.g., where the trained agent failed to imitate the demonstrator? What was the cause of poorer performance? Why did bigger model perform worse?). Creating such insight in one environment would be sufficient, as by focusing on a single environment, you can create very specific scenarios to tease out these answers. - Limited scope of the work: experiments done in a single environment. Most of the paper is framed in a way that this is not a huge issue (e.g., ablations), but proposing new method just for playing NLE has limited impact. If a new method is proposed to generally improve RL/IL performance, it should be tested at least in two distinct environments. - Limited improvement in the context of SOTA solutions: 2x over the baseline used in the paper with RL and proposed architecture included, but other neural agents in the NetHack Challenge had higher score. To be interesting in terms of performance, it should at least outperform the NetHack Challenge Neural solutions. - Proposed method is limited in novelty, as evident by the previous work listed in the paper. If the hierarchical BC figured out the hierarchy automatically (or, if it was an emergent behaviour of the model), that would be more interesting. - Paper outlines some assumptions on why things failed (e.g., "model overfitted" or "learned to self-correct"), but these claims were not verified with results. The paper would be much stronger if you can give solid, verified answer that indeed, overfitting was to blame or that RL trained the model to "self-correct".

Questions

Questions: 1) In multiple occasions paper says that the lower performance of bigger model is due to overfitting (e.g., line 229). However there are no results/experiments to show that this indeed was the case. A simple way to find this out is to do train-validation (or even train/validation/test) split, and testing on held out data as training progresses. 2) Regarding data scaling experiments: did you change any other settings of the training setup when increasing data amount? Previous work has demonstrated that the optimal model size and/or training compute depends on the amount of data (Hoffmann et al. 2020). 3) Regarding model scaling experiments: I assume only the number of layers in the transformer was changed? The bottleneck of the network may be elsewhere, e.g., one of the input layers or output layers. I would recommend scaling the whole network, similar to what OpenAI VPT work did, where ResNet blocks were "widened" in terms of filters, as well as increasing transformer size (Baker et al. 2022). Also, Hoffmann et al. (2020) changed number of layers, number of attention heads and transformer dimensionality when scaling models. This might be something you want to try. 4) Instead of LSTM + Transformer model, did you experiment with transformer model only? E.g., akin to VPT work (Baker et al. 2022), embed all inputs into one vector, stack vectors over timesteps, apply causal transformer, and predict actions from the transformer outputs. This type of model might scale better, as it reduces the amount of components that might interfere. #### Comments (not questions) - Fig1 right: weird scale. Any chance to get more points? - Line 205: grammar error at the start of the line - Explain/rename "Dlvl" and why "Turns" is good metric - Figure 3: "LSTM + XXL Dec" is bit confusing naming, since "decoder" is not commonly used term in the paper. I'd recommend using something like "LSTM (bigger)" to simply reflect that it is the LSTM baseline but with bigger network - Figure 3 (and others): add explanation to caption what is the error bar of the bar plots. Is it standard deviation or standard error (or something else)? - Table 2 caption: starts with weird "[V4]" #### References - Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022). - Baker, Bowen, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. "Video pretraining (vpt): Learning to act by watching unlabeled online videos." Advances in Neural Information Processing Systems 35 (2022): 24639-24654.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

No explicit sections for limitations or broader/societal impact was given. Authors bring up the future work ideas in the conclusion. While I think the work does not require societal impact section (no immediate impact), I urge authors still think through of any cases where the work or the insights could impact others. Or alternatively, what impact would _not_ including some results do (e.g., skipping some analysis). ## Rebuttal acknowledgement I have read authors' rebuttal which did address my concerns, and I increased my rating from 4 to 7 to signal my vote to accept this paper (change was done before discussion period closed).

Summary

This is an emergency review, and I regret that the paper is out of my expertise, which is why my review will rather stay at the surface level. The paper is concerned with the NetHack challenge, a complex AI challenge that in 2021 reached headlines, because symbolic agents considerably outperformed neural agents. I see three main contributions in the paper: - The construction of a large-scale dataset, based on the best symbolic agent and its policy choices, that can enable training better neural agents - The training of better neural agents based on this dataset, and other improvements - A systematic analysis of the effect of different technical improvements (hierarchical BC, larger Transformer models, larger datasets, online fine-tuning with RL), notably finding that scaling training sets or model size alone will not bridge the gap to the best symbolic agent. The problem is of very high interest to the AI community, and the technical investigation, results, and discussion appear thorough and insightful. The dataset might also enable further research. I find especially the results regarding scaling interesting, i.e., that performance increases logarithmic, and so more data or bigger models alone will not enable achieving parity with the symbolic approach. Quality of writing is very good, and so the paper is easy to follow (subject to my lack of technical background). Minor notes: - The paper appears to be missing a link to the dataset - The related work is not easy to access for someone not close to the field. E.g., paragraphs on "imitation learning" and "hierarchical policy learning" give too little detail about the basic ideas (do not start with descriptions of what they are for, but what they do) - "The full observation space of NLE is far richer and more informed than the view afforded to human players of NetHack, who observe only the more ambiguous “text-based” components of NLE observations" - I do not fully understand this sentence, please expand. What can systems observe in NLE, that humans don't receive in the original interface? Or do you mean that NLE aggregates the Ascii terminal characters into something more high-level? - Showing an excerpt from the dataset would be helpful, especially, as it is not quite clear what is added there, both strategies and substrategies? Or the more specific one only?

Strengths

See above.

Weaknesses

See above.

Questions

See above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Yes, the authors critically discuss that scaling alone will not bridge the gap to symbolic agents on this challenge.

Summary

The paper explores reasons for this performance gap between neural and symbolic methods in NetHack: Symbolic agents use hierarchical policies and parsers to extract high-level features Symbolic agents have handcrafted heuristics and error correction Neural agents lack inductive biases like hierarchy that may be needed for sparse rewards Experiments show hierarchy, scale, and combining imitation and RL help improve neural agents: Hierarchical behavior cloning improves over flat BC Larger Transformer-based architectures improve over LSTMs RL fine-tuning provides gains, especially for underfitting models But significant gaps to symbolic agents remain

Strengths

The experimental design is very clever, the chart is very clear, and the experimental effect is obvious. The paper explores a novel problem domain of applying neural networks to master the game NetHack, where current methods struggle compared to symbolic AI. The authors introduce a new large-scale dataset of NetHack demonstrations called HiHack to facilitate this analysis. The idea of using demonstrations to help neural networks learn better policies in sparse, long-horizon environments like NetHack is creative.The methods are detailed appropriately to replicate experiments. Results are presented logically and incorporate useful visualizations. The conclusion summarizes takeaways concisely.Mastering complex environments like NetHack with sparse rewards and long time horizons remains an open challenge for deep RL. This paper provides significant evidence and analysis characterizing the limitations of current neural network methods in these settings, and points the way towards progress, whether via incorporating stronger inductive biases like hierarchy or combining neural and symbolic approaches. The insights will broadly impact research in sparse reward RL, imitation learning, and integrating neural and classical AI.

Weaknesses

This model is based on the nethack, and the results hold up on the above models, and whether the above results can still hold up on the other models。The authors recognize the limited generality so far of methods tested on NetHack to other complex environments.No obvious harmful biases or problematic data sources are introduced in this work. The NetHack environment itself seems relatively innocuous.

Questions

Can you add some experiments, add some theoretical derivation, whether the contribution of this article is more.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The model is not so representative, can switch a more popular model。Overall, the authors demonstrate good care and thoughtfulness regarding the limitations and potential negative impacts of this research direction. The discussion seems sufficient without being overreaching or distracting from the primary technical contributions. I do not have any major suggestions for improvement.

Response to rebuttal

Thanks for your response! I think we are generally in agreement about both the value of using NetHack and the limitations of gathering findings in a single domain. Similarly, I think we generally agree about the potential illustrative value of the results and the dataset as well as the limited surprise factor in the findings.

Thanks for the rebuttal

Thank you for the comprehensive response. Although I admit that comparisons with the training time cap do provide unique insights and practically makes the experiments more computationally manageable, I want to point out that the training efficiency can vary by a large margin depending on the implementation details, which are non-intrinsic to the algorithmic methods and can obscure analysis. Training all models till convergence allows the influence of the architecture and data factors to manifest more clearly, which are the article's main focus, and in my opinion surpasses the benefits enlisted by the authors of using the training time cap. Other than this point, my questions are properly addressed thanks to the authors' rebuttal and I will update my rating from 6 to 7.