Summary
In this work the authors fine tune a DistilGPT2 model to produce diverse, largely playable Mario levels. They incorporate a novelty search to favor diversity in levels, and also explore conditioning level generation on natural language user prompts. Their novelty search proceeds as follows: Given an initial set of levels, a level is sampled from the set and randomly mutated. The mutation involves deleting a random contiguous segment of the level and sampling a new one in its place from their model, then using a fine tuned BERT model to do inpainting on the border regions to smooth the transition between this and the rest of the level. The mutated level is accepted if it is sufficiently different from the other levels, using a distance metric comparing the player path through the level.
Strengths
- The use of LLMs for game level generation certainly makes sense, and I think it's valuable to publish work in this area for the NeurIPS community to see and build on. The technical approach taken here is original, to my knowledge.
- The combination of genetic algorithms with LLMs is interesting and novel, and I found the use of BERT for inpainting to smooth out the transition between the mutated segment and the rest of the level to be a particular clever approach.
- The paper also does a great job of visualizing the levels produced by the model which is very helpful in getting a feel for the sorts of outputs that their model produces. The playability and temperature analyses were quite interesting as well.
Weaknesses
- Section 4.5: It's hard to tell that the novelty search is actually helping, without a comparison to an ablated version that doesn't do the novelty search. In particular it's hard to tell if the darker points in Figure 9 seem to be filling in the space in between because of the novelty search, or if it's just that there are more sampled points so they happen to fill in some of the space in between the earlier points. Likewise with the rightmost graphs in figure 7 looking denser than the leftmost ones – taking more samples will always add some density, so in order to really conclude something you need a comparison to an ablation. Showing something like 300 levels from the novelty search plotted on the same tSNE as 300 levels from a version without novelty search (eg as a different color) would make it much clearer that there's actually an improvement in diversity (and likewise some analogous experiment for the other figure).
* In a similar vein, in Table 3 it's hard to know how much the prompt is helping without seeing the scores without prompting (for an idea of what scores a fairly random approach would achieve); though I think is a less major issue.
* After seeing the LSTM from prior work used in the Table 1 comparison, I was surprised that LSTM comparisons weren't given for any of the future evaluations, which would be helpful for understanding how much better this LLM setup is than prior work. For example, we learn that 88% of levels are solvable, but how does this compare to the LSTM approach? Likewise in the diversity analysis, a naive LSTM baseline would be interesting (but no need for all the novelty search machinery). This would just help me get a sense of where this work stands compared to prior work on these metrics that are being measured. That said, I wouldn't reject this paper on that basis – 88% playable is a great statistic regardless of what you compare it to, and I don't actually expect that the LSTM would make particularly good levels, but having the full comparison would strengthen the evaluation.
- As a last, minor point, It's surprising (in a way that makes me feel that an explanation is needed) that such poor performance is obtained by from-scratch-MarioGPT on the tile prediction task given it was trained on 200,000 datapoints. The "tile accuracy" is presumably a measure of how well the system does next-tile prediction (analogous to next-token prediction in language modeling, and presumably this is the objective that all these models are trained on). I'd imagine something as simple as "always predict an air tile" since most levels look like they're more than half air, or "predict the same tile that you saw to the left" would do quite well, and the inability to pick up on even simple trends like this after 200,000 datapoints is surprising to me. Perhaps there is additional complexity I'm missing?
Overall, this is an interesting paper that I end up borderline reject on because of the evaluation points mentioned above, but I don't have a huge amount of prior experience in this area or what typical evaluations are for PCG, so I am open to revising through discussion.
Questions
* Relating to the weakness point above, in Table 1 I'm a little surprised that everything except MarioGPT does so poorly (LSTM does okay at least though). It'd be helpful to have a little intuition for why this is so hard?
- The concurrent work cited by this paper (Todd et al 2023) finds that pretraining (vs from-scratch) GPT2 has little to no impact on a range of metrics. That is of course a somewhat different task and setup, but I'm curious if you have thoughts on why you seem to find the opposite result in this domain?
- What embedding is used for the tSNE? Is it something like the last layer of MarioGPT?
- Do you guarantee that each tile is parsed as a separate token? Without this it seems like the network could potentially get confused about reasoning about lengths/distances, like if one token represented two blocks while another represented just one. Seems like it works either way, but I'm curious if this is something you're doing or might look at.
- The language conditioning is cool! Of course, a natural extension of this would be moving towards level generation from more unstructured natural language. For example, flexibility with synonyms to "many" or more complex things like "a series of narrow pillars with an enemy on top of each one". That sounds like enough for a separate paper but I'm personally curious if you've done any preliminary work in that direction or what you think of it. To be clear, I don't think this is needed in this submission I just think that the work is cool and am curious!
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
I think that this paper does an adequate job of presenting its limitations.