Unbounded: A Generative Infinite Game of Character Life Simulation

Summary

This work combines various aspects of Generative AI to develop a LLM-based game engine. The design principles refer to open-world, continuous interaction games with limited simultaneous characters but significant opportunities for situation generation and interaction. Capitalizing upon recent research in the field, the system introduces a number of technical innovations both in terms of architecture (in the form of a dynamic regional prompt adapter) and in term of implementation (distillation of Small LLM). With an existing implementation, the paper includes elements of comparative evaluation, both quantitative and qualitative.

Strengths

This is an ambitious piece of work, tackling difficult technical issues in balancing generation and interaction, as well as various training and fine-tuning protocols. It appears to have innovated on a number of aspects, for instance the dynamic mask to balance character and environment generation. It gives some interesting, detailed, insights such as those of Figure 4. The improvement over IP-Adapter thus appears more than incremental. Another strong points is the Small LLM distillation framework. The comparative LLM evaluation has staged an appropriate number of LLM, capturing size and diversity. Finally, the paper demonstrates a good command of recent relevant work and compares its approach to really recent alternative methods (2023, 2024), which is welcome. Having considered the authors' rebuttal and additional data, I have raised my score for contribution to 3 ('good').

Weaknesses

From a fundamental perspective, there is a lack of awareness of some game design issues. The introduction is relatively naïve considering how these issues are addressed in game/play theory (Caillois, Huizinga). With the fine line between simulation and entertainment, and notwithstanding the reference to The Sims and Tamagotchi, contemporary audiences tend to have gameplay expectations that can only be met with sustained and meaningful (in the narrative sense) interactions. This feeds into the overall perspective of the evaluation methods which in the paper are primarily focused individual ‘event’ generation. While sustained UX evaluation may be at odds with the system’s current level of maturity, some of this could have been addressed through even slightly longer exchanges or chains of events construed as minimal narrative experience (which is claimed in the abstract but does not really materialize in the examples). The small examples given, including on Figure 1, do not convey a very strong sense of gameplay. There is no evidence that the number of user-simulator examples is at all sufficient for the intended purpose of multi-topic data collection. Nor is there any discussion of some hierarchical organization of topics that would come close to scenarios, game genres or elementary forms of plot backbones. Despite the statement on replicability, the level of implementation details is unclear. Implementation details are rather cursory and uneven with some in-depth details but no systematic discussion, and minor duplications from other sections. Evaluation has both positive (see above) and negative aspects. Unless I am mistaken, Table I shows a couple of inconsistencies for results. On Environment Consistency, how can 0.322 be the best score when it is in-between 0.381 and 0.257? How can 0.675 be the best score in-between 0.595 and 0.832? It is difficult to find the Qualitative component of evaluation very convincing in the absence of a more systematic approach for generating test situations. From a similar perspective, the example Prompts do not contain advanced mechanisms for controlling response, or specific examples. One would thus wonder how LLM response can be kept consistent across runs, and what type of statistical sampling has been associated to LLM use?

Questions

To which extent are results from Figure 6 consistent with results from Table I? How have you ensured the quality and fairness (for evaluation) of the text prompts sample? How can the current approach maintain narrative consistency over a number of interactions?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Summary

The authors present "UNBOUNDED" a virtual pet game based on LLMs. The authors present two novel technical elements to support this game: a regional IP-Adapter for character-environment consistency and a domain-specific distillation approach for finetuning a smaller LLM to allow for real time gameplay. The authors then present experiments comparing their approach to baselines in terms of visual quality and consistency, and overall game quality as scored by GPT-4o.

Strengths

The main strengths of the paper are (1) the originality of entirely basing a virtual pet game on LLMs, (2) the significance of the two primary technical contributions to figure coherent image generation and real time LLM application uses, and (3) the quality of the image portion of the evaluation/experiments. The paper is also overall well-written, especially in terms of the technical aspects. These primarily justify the positive aspects of my above scores.

Weaknesses

The paper has two major weaknesses in its present draft, the way it positions itself in terms of prior work and the evaluation. ### Prior Work The authors do not appear to have engaged with the field of technical games research at all. This is a shame, as their game can be understood as an AI-based game [1] or specifically a NN game [2]. More broadly, the authors' work in terms of generation would fit into the Procedural Content Generation paradigm [3]. There is a 3+ decades long history of work in this space that is relevant to the authors' work [4], with significant recent work at the intersection of LLMs and game generation [5], and it is crucial that the authors update the introduction and related work to situate their work within this field. ### Evaluation The authors primarily evaluate their work in terms of image quality and I have no major concern with these results (though the DreamSim results seem to incorrectly attribute the best score to their system). However, I have major concerns with the "Evaluation of LLM Generations", which is functionally the closest thing presented to a holistic evaluation of UNBOUNDED. Firstly, the norm within the area would be to make use of a user study for evaluation purposes [6,7]. Particularly since the authors make such an effort to allow for real time control, this would seem to follow naturally. Secondly, the evaluation is somewhat concerning due to the training process. Specifically, since the distillation approach makes use of GPT-4 to simulate user interaction data and the authors sample for uniqueness/variety, its fairly likely that the newly sampled user interaction data approximations will be very similar to these. Further, the largest model the authors compare to is again a GPT-4 variant. Given that GPT-4 is being used for both generation of data, as the system, and as the evaluation, there's a clear risk for bias. This could be addressed by any number of ways, including automated metrics for measuring game quality developed in prior game generation-like work [8,9]. But at present, it is difficult as an outside reader to get a sense of the actual quality of UNBOUNDED as a game. Failing this, it might be beneficial to include longer examples of gameplay in the supplementary materials at least. ### Figures While these are not of major concern, I wanted to note two aspects of figures that I felt could be improved. First, the authors make use of a large number of fonts in their figures, and their primary font is a bit difficult to read. I'd suggest updating the figures to use a single, more legible font. Second, Figure 4 is largely superfluous. If the authors wish to retain it, it might be a better fit for an appendix. 1. Treanor, M., Zook, A., Eladhari, M. P., Togelius, J., Smith, G., Cook, M., ... & Smith, A. (2015). AI-based game design patterns. 2. Zhu, J., Villareale, J., Javvaji, N., Risi, S., Löwe, M., Weigelt, R., & Harteveld, C. (2021, May). Player-AI interaction: What neural network games reveal about AI as play. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-17). 3. Shaker, N., Togelius, J., & Nelson, M. J. (2016). Procedural content generation in games. 4. Pell, B. (1992). METAGAME: A new challenge for games and learning. 5. Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., & Yannakakis, G. N. (2024). Large language models and games: A survey and roadmap. arXiv preprint arXiv:2402.18659. 6. Anjum, A., Li, Y., Law, N., Charity, M., & Togelius, J. (2024, May). The Ink Splotch Effect: A case study on ChatGPT as a co-creative game designer. In Proceedings of the 19th International Conference on the Foundations of Digital Games (pp. 1-15). 7. Guzdial, M., & Riedl, M. O. (2021). Conceptual game expansion. IEEE Transactions on Games, 14(1), 93-106. 8. Khalifa, A., Green, M. C., Perez-Liebana, D., & Togelius, J. (2017, August). General video game rule generation. In 2017 IEEE Conference on Computational Intelligence and Games (CIG) (pp. 170-177). IEEE. 9. Guzdial, M., & Riedl, M. (2018, September). Automated game design via conceptual expansion. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (Vol. 14, No. 1, pp. 31-37).

Questions

1. How do the authors position their work in the field of technical game design? 2. What do longer gameplay examples look like? 3. How do the authors respond to my concerns around their "Evaluation of LLM Generations"?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

General Response

We thank all the reviewers for their thoughtful feedback. We’re glad that they acknowledge our originality of entirely basing a virtual pet game on LLMs (reviewer 4EBp), and acknowledge our two main technical contributions in coherent image generation and real-time LLM application uses (reviewer 4EBp, Nnkj), and recognize that our paper has good image generation evaluations (reviewer 4EBp) and compares to really recent alternative methods (reviewer Nnkj). We address their questions below and have updated the submission paper accordingly, and hope for a re-evaluation of the paper score if their concerns have been addressed.