FactorSim: Generative Simulation via Factorized Representation

Summary

This work presents FACTORSIM, a framework that converts any language specification into a complete simulation for training RL agents. FACTORSIM decomposes the input prompt into steps and uses a factored Partially Observable Markov Decision Process (POMDP) to minimize the context needed for each generation step. It also introduces a method to benchmark generative simulation and demonstrate the capability for FACTORSIM to be used in robotics setting.

Strengths

1. Develop a robust pipeline for constructing game environments from language descriptions, which could significantly enhance the scalability of training generalist agents. 2. Formalize the framework as a Partially Observable Markov Decision Process (POMDP), reducing the need for full context during generation and improving outcomes. 3. Demonstrate the potential of this method to generalize to embodied scenarios.

Weaknesses

1. The evidence for generalizing to embodied scenario is limited. 2. The successful rate in Table 1 and Figure 3 is low. Could there be some potential way to improve it?

Questions

I have one primary concern: how could this be applied to the field of robotics and embodied AI? **From this concern, several questions arise:** - How is the background (or scenario) generated within this pipeline? In a 2D game setting, detailed descriptions generate the scenarios, but this can become extremely tedious as scenes become more complex, such as in an embodied environment. While language-guided scene generation could be a solution, how will it fit into the POMDP framework? - The framework addresses three main components: controller, model, and view (rendering). In robotics, these aspects are typically handled by a physics simulation. How will this framework further contribute to the field of robotics? Currently, the paper shows potential for task generation in tabletop scenarios only. - I am still unclear on how the Robotics Task Generation part is achieved by this pipeline. **Some suggestions:** - While it might be challenging to address this concern with experiments during the rebuttal period, more discussion on approaches and challenges would be beneficial. - Any additional experiments that could demonstrate the pipeline's usage in robotics or embodied AI could help.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are addressed adequately.

Summary

The paper proposes a LLM prompting method to generate full game / robot simulations in code based on text descriptions. Given a long text description, the method first utilizes an LLM to decompose it into multiple sentences, and then use them to iteratively generate and update simulation code. For each iteration, the code is generated and updated separately as three modules, i.e., controller, model and view. The update happens in a factorized way - the authors use the LLM to identify relevant state and context to avoid feeding the full generated code into LLM. In experiments, the method is evaluated on game and robot simulation code generation benchmarks. The method shows superior results against other LLM prompting baselines in generating valid simulation code that aligns with text description.

Strengths

- The proposed method exploits the structure of simulation to modularize and factorize code generation. This strategy significantly improves LLM's capability to generate full simulation code. - The method is comprehensively evaluated on game and robot benchmarks. - The paper is well written and easy to follow.

Weaknesses

The major contribution of the paper seems to be a prompting technique crafted for the specific task of simulation code generation. While such a technique does improve performance on the task, my concern is it is neither fundamental nor sufficiently novel. The proposed prompting technique highlights two key designs: - modularize simulation code generation manually, which aligns with the common practice to manually decompose a complex task into sub-tasks for LLMs to handle more effectively. - extract relevant portion of code for LLM to consume and update, which is also an implementation-wise design that many works have already incorporated. While the paper writes factorized POMDP formulations, they don't seem to make a difference on how the prompting method is implemented. So I'm concerned that the contribution of this paper is more as a practical application rather than a general method or framework.

Questions

I'm curious what the failure modes of FactorSim is like.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

See Weakness

Summary

The paper proposed a factorized approach to generate simulated games via LLM code synthesis. The code idea is that one doesn't need to generate the entire code at once, but rather generate different part of a POMDP game, such as controller, model, and view. The generated simulation allows RL policies to train on top. The authors introduced a benchmark to evaluate the proposed framework and show good results in terms of prompt alignment, transfer ability and human evaluation.

Strengths

The paper investigates an important problem, simulation generation. The evaluation over the mentioned environments is solid, spanning from automated tests to human evaluations.

Weaknesses

1. The paper is poorly written. I have hands-on experience with almost all important concepts mentioned in the paper, yet still have a hard time understanding the paper, and have to read again and again including some code. Rather than talking about abstract terms like POMDP / factorization first, I think the authors can start easy with intuitions and explanations. The figure can also be improved. The main method figure shall spend more time showing what's special about "Factored POMDP" compared to prior methods. The benchmark claim should have its own section. The motivation is not clearly narrated either. The world model section in related work doesn't seem to fit there. 2. On of the main contributions the authors listed in the introduction section is a benchmark. However, I think this benchmark seems to lack the technical depth I was expecting as a standalone contribution. I feel it's just a set of small metrics rather than benchmark. 3. The paper just lacks the level of technical contribution that meets my criteria for a Neurips paper. While there are many other prompting papers like CoT, ToT, the problem the paper is trying to solve is also very specific. 4. While I have experience with both LLM and robotics, I believe the authors should not put Robotics as primary area, but NLP or code synthesis community.

Questions

In figures like figure 6, is the human pass rate based on the previous stage e.g. only executable code. It seems that on open source model like llama 3, gensim with CoT is very close to factor sim. Can you explain the insights?

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The author discussed the limitations of automated evaluation by conducting a human study. No outstanding negative societal impact.

Summary

The paper introduces an LLM-based method for generting code for simulations. After generating the simulation of famous games based on their manual and description, the authors show that policies trained in these environments transfer well to the real games.

Strengths

- **S.1 Great results.** I think that the results from Fig.3 are very impressive. Zero-shot transfer is very hard, and doing so much more reliably than vanilla GPT-4 is impressive. - **S.2 Overall good idea.** The idea to deconstruct the game development task into M-V-C makes a lot of sense to me. I just thought that's already kind of captured in the POMDP formulation. - **S.3 Good presentation.** The overall presentation and writing are good, although there is much left to be desired in terms of implementation details.

Weaknesses

- **W.1 Implementation details and relationship to formulas.** I'm happy that there was code provided with the submissio and I hope that it will be released publicly because based purely on the main body of the paper, the method is not reproducible. Including the prompts in the appendix helps but I wish they were commented a bit more on why certain phrases and sections are there. And I'm wondering if the theory that's presented in the paper holds water wrt the actual instructions in the appendix. Because as far as I understand, there aren't any restrictions on what code the LLM can generate for each component, right? Also, the paper mentions graphs every so often and I don't know how they fit into this. I also think the context selection is crucial to your method and from the main body of the paper, it's completely unclear how that's implemented. - **W.2 Missing examples.** Along a similar idea, I'd have loved some examples throughout the paper to illustrate what some of these instructions actually mean. - **W.3 Unclear robotic experiments.** The robotic experiments seem to be more of an afterthought in the paper, and it's unclear what existing assets there are, what control code is assumed given, what camera parameters are assumed, etc. - **W.4 Unclear input mapping.** The appendix mentions that the controller is given or that button presses always mean the same thing. I completely don't understand what's meant by that. Overall, I think the paper shows a great idea and is probably beneficial to the community, but some work should go into tweaking the main body of the paper and making the method more clear and reproducible.

Questions

I don't have any major questions wrt the work. There were a couple of points throughout the paper in the methods section, where I asked myself why this is relevant, but then this was cleared up a paragraph later.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately address the limitations of their work.