While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household activities. The limitation arises from the fact that LMs are trained only on written text and miss essential embodied knowledge and skills. In this paper, we propose a new paradigm of enhancing LMs by finetuning them with world models, to gain diverse embodied knowledge while retaining their general language capabilities. Our approach deploys an embodied agent in a world model, particularly a simulator of the physical world (VirtualHome), and acquires a diverse set of embodied experiences through both goal-oriented planning and random exploration. These experiences are then used to finetune LMs to teach diverse abilities of reasoning and acting in the physical world, e.g., planning and completing goals, object permanence and tracking, etc. Moreover, it is desirable to preserve the generality of LMs during finetuning, which facilitates generalizing the embodied knowledge across tasks rather than being tied to specific simulations. We thus further introduce the classical (EWC) for selective weight updates, combined with low-rank adapters (LoRA) for training efficiency. Extensive experiments show our approach substantially improves base LMs on 18 downstream tasks by 64.28% on average. In particular, the small LMs (1.3B, 6B, and 13B) enhanced by our approach match or even outperform much larger LMs (e.g., ChatGPT).
Paper
Similar papers
Peer review
Summary
This paper proposes a new collection of data and tasks from an embodied environment to enhance the embodied reasoning ability of pre-trained language models. The paper also designs a fine-tuning strategy that combines the advantages of EWC and LoRA for stable fine-tuning. The experiments show that the proposed method can efficiently fine-tune GPT-J and outperform much larger language models.
Strengths
This paper designs two ways of data collection in the embodied environment that address goal-specific and general cases in real-world applications. This paper designs a comprehensive reasoning task set based on the collected data to evaluate the reasoning performance from different aspects (plan generation, object tracking, etc.) This paper adopts a suitable parameter updating strategy (EWC_LoRA) for language model fine-tuning, which demonstrates time and memory efficiency.
Weaknesses
The paper lacks experiments on why EWC is needed. The motivation for including EWC is to avoid overfitting downstream tasks or catastrophic forgetting on the pretraining task, but LoRA has a similar purpose. From the experiments in Table 2, EWC_LoRA has a lower performance than LoRA except for a slight improvement in perplexity.
Questions
1. Related to weakness: It will be good to see whether there are any other benefits gained from EWC. 2. How do you ensure that the answers of the negation Housework QA are truly irrelevant? 3. The activity recognition performance of all methods is quite similar. Does this indicate that the activity types are easily distinguishable without requiring a full description of the experience, for example, just keywords at the end are sufficient for the task? Have you tried to include confusing activities in the evaluation set? 4. How do you introduce the confusion term (counting task, confusing unseen of plan generation)? Do you have any guideline or do you just randomly inject phrases?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The author mentioned the limitations of only collecting embodied experience from one world model. societal impact: did not find negative impacts, probably if the data collection is extended to the real world in the future, it should avoid collecting personal or sensitive experiences.
Summary
The paper introduces a training paradigm termed "finetuning with Embodied Experiences from World Models (E2WM)" to enhance the abilities of language models (LMs) in reasoning and planning tasks associated with physical environments. The authors argue that LMs trained solely on large-scale text corpora lack the embodied knowledge necessary for robust performance in such tasks. To address this, they propose leveraging world models, specifically the VirtualHome simulator, to collect diverse embodied experiences and use them to construct fine-tuning tasks, such as plan generation, activity recognition, counting, and object path tracking. The proposed fine-tuning method incorporates Elastic Weight Consolidation (EWC) regularization with low-rank adaptation (LoRA) to preserve the models' generality and avoid catastrophic forgetting and make the training process more efficient.
Strengths
- The paper introduces an approach to enhance off-the-shelf LLMs with embodied knowledge. The experiments involve both goal-oriented planning and random exploration. in collecting embodied experiences. While EWC and LoRA are existing methods (hence the paper may be perceived as with limited technical novelty), this work shows that their incorporation into the fine-tuning process helps retain the models' general knowledge and capabilities while adapting them to new tasks. - The related work seems sufficient, albeit a section on catastrophic forgetting, e.g., [1] would be a plus. - The paper provides an evaluation of the finetuned LMs on both seen and unseen tasks, and several ablation studies. The generalizability of the models is also tested, as well as their performance on the original pretraining data to ensure core language modeling abilities are retained. [1] Korbak, Tomasz, Hady Elsahar, German Kruszewski, and Marc Dymetman. "Controlling conditional language models without catastrophic forgetting." In International Conference on Machine Learning, pp. 11499-11528. PMLR, 2022.
Weaknesses
- Lack of comparison: Only two GPT-based LLMs are used. Would be valuable to consider a more diverse set of recent LLMs. The paper also does not compare the proposed E2WM paradigm with existing methods for enhancing LMs with embodied knowledge, such as [1]. - Scalability to larger models: Although the EWC-LoRA approach is designed to improve efficiency, it is not clear how well it scales to larger LM architectures. Further investigation into the scalability of the proposed method would be beneficial. - Limitations on benchmarks: Evaluation is performed on bAbI but there are several embodied benchmarks and simulators that could be considered. Doing so would allow evaluation in realistic embodied settings, and compare other embodied models proposed (such as the Episodic Transformer [2] or [1]). - Experimental Results: The number of tasks and models evaluated is relatively small and the presentation of the results could be clearer, e.g., adding numbers to bars in Figs. 3 and4. It seems that ChatGPT outperforms the fine-tuned models, so overall the model capacity appears to be an important factor, and the proposed fine-tuning shows marginal improvements in some of the tasks. [1] Lin, B.Y., Huang, C., Liu, Q., Gu, W., Sommerer, S. and Ren, X., 2023, June. On grounded planning for embodied tasks with language models. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 11, pp. 13192-13200). [2] Pashevich, Alexander, Cordelia Schmid, and Chen Sun. "Episodic transformer for vision-and-language navigation." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15942-15952. 2021.
Questions
Have the authors considered an evaluation on actual embodied benchmarks? What is the motivation for choosing Babi as the benchmark dataset?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
There is a very brief discussion on limitations as part of the conclusion section.
Summary
This paper proposes to incorporate world models into large language models to enable understanding of object permanence and planning capabilities that are missing in text-only models. Specifically, the authors utilize embodied environments (VirtualHome) to collect training examples including goal-oriented planning (plan generation, activity recognition) and random exploration (counting and object plan tracking). To avoid overfitting, this paper finetunes GPT-Neo-1 3B and GPT-J-6B models using LoRA and EWC. Evaluated on seen and unseen tasks, results show that the proposed method after finetuning outperforms the corresponding text-only baselines, and match or outperform ChatGPT performance under few-shot setting.
Strengths
1. This paper is motivated by an interesting limitation in text-only language models, which is the lack of embodied knowledge in pre-training. The proposed fine-tuning dataset, along with EWC-LoRA method, improve the model performance on challenging embodied reasoning and understanding tasks, without dramatically increasing the original language model perplexity. 2. The paper includes some interesting ablation studies to discuss how much improvement is observed from each fine-tuning task.
Weaknesses
1. The contribution and findings of the paper is limited. Although the limitation of the LLMs is well motivated in the paper, fine-tuning on similar distribution to improve model performance and thus outperform baselines is expected. Furthermore, fine-tuning with LoRA to improve efficiency, as well as EWC to reduce overfitting have been well studied. Given that the (relatively weak) baseline is the original text-only model (and a few-shot prompted ChatGPT model), it is not convincing that the proposed method (either finetuning data or method) is generalizable to broader embodied tasks (e.g., compared to an embodied or multi-modal model). 2. Some important details are missing and the evaluation is not very convincing. The paper is phrased to incorporate world model into LLMs, but it is confusing (without explicit explanation) that both training and evaluation are not with the embodied environment. Rather, only partial of the data is collected using VirtualHome. More importantly, it is not clear how the data are constructed (for example, how to sample "irrelevant actions" for counting QA and statistics of the training and eval sets) and what the held out set is for training the model. For evaluation, I understand the automatic eval is the most convenient metic, but have been widely adapted, metrics such as Rough-L for tasks like plan generation may not reflect model performance on complex text generation tasks. See more details in the questions below.
Questions
1. Have you done other evaluation on the base language model (i.e., NLP benchmarks such as superglue) apart from perplexity? 2. Why do you think EWC only outperform LoRA (in table 2)? As a regularization method, EWC constraint parameter updates and thus should understand LoRA by intuition. 3. Do you have a naive fine-tuning baseline to compare to? 4. Do you have detailed analysis on when and why the model improves performance on both seen and unseen tasks?
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The paper briefly mentioned limitations.
Summary
This paper proposes a new method to improve the embodied planning capability of large language models (LLM) via adding both goal-oriented planning and random exploration data from a world model/simulator as well as the EWC-LORA regularizer that prevents catastrophic forgetting of pre-training tasks. The new regularizer EWC-LORA proposed by the authors is not only time and memory-efficient compared prior KL constraint-based regularization and also leads to negligible perplexity increase and better downstream task performance. The authors show superior performance of the method over base LLMs in constructed embodied planning and activity description tasks based on the VirtualHome simulator and also the bAbI a dataset for testing multiple types of knowledge and abilities including embodied knowledge, logic reasoning, linguistic knowledge, etc. Note that the method can only outperform much bigger LLMs such as ChatGPT in many of the scenarios.
Strengths
1. The authors presents a simple yet effective way to improve LLM's embodied planning capability without sacrificing the abilities during pre-training under a small/negligible compute/memory cost. Such idea is neat and of great practical importance. I think the method will be quite significant for the future directions of making LLMs better at embodied tasks. 2. The authors have done extensive experiments in both customized and existing benchmarks/datasets to show that the proposed method can outperform base models without fine-tuning with embodied data. It is impressive that the authors show that the small models with embodied planning data fine-tuning can outperform large LLMs such as ChatGPT in many scenarios. 3. The authors also performed detailed ablation studies to show the importance of the proposed EWC-LORA regularizer via comparing to KL divergence and EWC only. The ablation studies on including various data mixture also make the paper more complete.
Weaknesses
1. I think the authors should compare to stronger baselines that also consider improving language model's embodied decision-making and reasoning capabilities such as [1,2]. Improvement over base models is good but not that surprising and convincing. 2. The authors should consider other embodied decision-making benchmarks such as ALFWorld, which is used in [1. 2]. This would provide a clearer picture of the comparison between the method and prior approaches. It would also add more tasks in the empirical evaluation, which can further validate the method. [1] Shinn, Noah, Beck Labash, and Ashwin Gopinath. "Reflexion: an autonomous agent with dynamic memory and self-reflection." arXiv preprint arXiv:2303.11366 (2023). [2] Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).
Questions
Please address the comments listed in the section above.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
Yes.
Summary
The paper proposes to enhance language models by finetuning them on “embodied experiences”, which are textual data generated by household activity simulator, which is referred to as “world models” in the paper. Through evaluations on several tasks (e.g., planning, object tracking), it is shown that finetuning on these “embodied experiences” leads to better performance compared to larger models (e.g., ChatGPT) that are not trained on these data. To minimize catastrophic forgetting and improve training efficiency, the paper also proposes to combine “elastic weight consolidation” and “low-rank adapters” to finetune the language models.
Strengths
- The premise of the paper is interesting, novel, and promising — because language models are not trained on embodied data, they might be less robust to those scenarios concerning interactions with the environments (i.e., the “embodied settings”). - The idea of using household activity simulator as a “world model” is interesting and likely significant in the context of enhancing language models. - Moreover, the authors conduct thorough experiments that support the central claim. - The writing and presentation of the paper are also clear.
Weaknesses
Despite the strengths of the paper, below are some concerns for the problem settings and evaluations: - While it is shown on GPT-Neo and GPT-J (which are relatively small language models in today’s standard) that the proposed approach improves their capabilities on results such as bAbI, as also indicated in the paper evaluations, larger models (i.e. ChatGPT) which are not trained on these embodied data also attain similar performance. Because the constructed tasks only require scene context in the text form, it is unclear whether similar delta will be seen on larger models. Or put this in other words: will this improvement diminish with larger scale training even without embodied experiences? The reviewer would like to note that due to practical reasons, it is understandable that experiments like these may not be done for larger LMs, but it is worth further discussions or experiments in the paper to support the claim. - Another weakness is that the training tasks need to be hand-curated for the “embodied experience”, which includes planning, activity recognition, counting, and object path tracking. This is unlike how autoregressive LMs are trained, which only requires one main self-supervised task of next token prediction. This brings to the question of how scalable the proposed approach is, as the broader “embodied experiences” include tasks of much higher diversity, in addition to including other modalities such as vision.
Questions
An important baseline is ChatGPT, which is not trained on “embodied experiences”, but it doesn’t seem like the prompt for querying ChatGPT is provided. In contrast, the paper notes that they used few-shot prompting for the smaller LMs. If they are given the same prompt, would the performance differ?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
The limitations are not adequately discussed. See the comments in the “weakness” section above.
Thank you for the response and the effort for the additional experiments. I have raised the "soundness" score to 4, but I would like to maintain my overall rating given the scope considered in this work, i.e., I would still recommend acceptance for this work. [2] While I appreciate the effort for the extra clarification, I'm not convinced by the argument that the designed training tasks, planning and object tracking (at least in a simplified environment like VirtualHome), are sufficient to cover "embodied experiences". For example, physical or visual experiences such as those experienced by a physical robot are also core to "embodiment" (and likely would lead to better world understanding for LLMs), but these are simply not present in the setup explored in this work. The example brought up by the authors, regarding the objectives in LLM training, is also different from the setup in this work, as those objectives are unsupervised at its core, while the training data used here are more like labeled data. However, I still like the premise proposed in this paper, and this can serve as a stepping stone for future work.
Thank you so much for your supportive review! We fully agree with your points and didn’t claim planning and tracking in our work have covered all relevant “embodied experiences”. We agree that there are more diverse types of experiences (like physical and visual ones as suggested). They present enormous new opportunities for incorporation and further improvement of LMs. We hope the idea and approach presented in this work can inspire more studies in this exciting direction.
Thank you for the explanation and the results of your model scale-up experiments in the rebuttal. I would like to maintain my rating. I believe a fine design of the evaluation tasks is more important than carefully tuned models. Therefore, I still recommend this work to the community. Additionally, I have the following comments: 1. For Negation QA, if the irrelevant items are just a sample of things not mentioned in the ground truth plan, it seems that they cannot avoid being generally relevant. For example, "spoon" in the example of "which object is irrelevant to making coffee?". Since your evaluation set is relatively small (2 questions will affect 1% accuracy), it is better to have fewer ambiguous answers. As a benchmark task set, I would expect to see a more quantitative and systematic design, for instance, using n-hop relationships in the knowledge graph to construct the QA pairs. 2. Regarding the necessity of EWC, I have some doubts about it. It might be good to see the mean and standard deviation of multiple-round experiments.
Thanks for your positive feedback and valuable insights! Your suggestions regarding the quantitative and systematic design of the benchmark set, as well as the multiple-round experiments for verifying the necessity of EWC, will greatly strengthen our work and improve our manuscript's presentation. We will include them in the revision.
Thank you for your response
Thank you for the rebuttal and the new experiments that have addressed most of my concerns. I have raised my score to reflect this, albeit I am also not convinced that this work can be considered "embodied" in the full sense. It is worth noting that most of this work's evaluation criteria are designed to measure the performance of LMs in scenarios that do not rely on embodied context-specific physical actions or navigation within virtual worlds. General tasks such as question-answering and dialogue do not seem sufficient to claim embodiment, and while this work is valuable in broadening the general cognitive capabilities of LMs, it may be best to make it clearer that it differentiates itself from traditional embodied AI tasks, where agents operate and interact within specific environments to achieve tasks.
We sincerely appreciate your valuable reviews and are glad to know that our rebuttal and new experiments have addressed most of your concerns. We agree that our evaluation criteria are different from traditional embodied AI studies. This is due to our unique goal of improving the fundamental embodied knowledge of LMs for general language problems, which has not been studied before. We will definitely make the point clearer in the revised version as you suggested!
Thanks for the response! I understand the main contribution of this paper which is to inject world knowledge into language models. I was pointing out that as one of the two main contributions highlighted in the paper (fine-tuning), it was not clear why they would perform better than fine-tuning baselines (as updated above, thank you for the results). Intuitively, and as been widely shown in previous continual learning studies, EWC and other methods reduces catastrophic forgetting, while lack behind fully fine-tuning results. However, it seems that results between GPTJ-FT and FPTJ-E2WM are the opposite. Do you have any intuition on why using E2WM performs better on the in-domain tasks? In terms of embodied and multimodal models, I was not looking for a direct comparison. I was mainly suggesting that as this paper claims the benefits in embodied environment (as also pointed out by other reviews), the evaluation does not seem to be convincing enough. It would be great if the author can provide some comparison to research in embodied agents in the paper revise.
Thank you for your feedback. We would like to reply to your question and comment as follow: **(1) Why EWC outperforms direct finetuning** We want to clarify that our evaluation tasks are **out-of-domain** tasks instead of in-domain ones. Since our goal is to train a general-purpose model, we intentionally designed evaluation tasks that are different from the training tasks (but require similar knowledge) to test its generality. For example, as mentioned in Lines 233-234, plan generation evaluation requires *free-from natural language plans*, while the ground truth for training is *executable plans following the schema* of the specific environments. Similarly, other evaluation tasks such as Housework QA, Object Path Tracking, etc., are very different from the training tasks in nature. In addition, we also introduce diverse evaluation settings (e.g., Vanilla Unseen, Confusing Seen, and Confusing Unseen, as in Figure 3) that differs greatly from the training. As mentioned in the paper (Lines 108, 175-182 and 194-196) and our response, EWC helps preserve the generality of LM capabilities and thus improve the generalization to out-of-domain tasks. On the contrary, previous continual learning studies with EWC typically evaluate finetuned models on in-domain tasks seen during training. **(2) Comparison to embodied agents** As we’ve clarified in the rebuttal and general response, prior works on embodied agents are not comparable to our work. Contrary to other research on embodied agents, we've taken a unique goal and approach. Most of these studies either fine-tune the LLM to create a task-specific model [1][2] or incorporate special modules on top of the LLM [3][4], tailoring it to particular tasks. This specialization contrasts with our objective: we are striving to develop an LLM that serves as a general-purpose model enriched with embodied experiences. We’ve also demonstrated that our models can generalize new acquired knowledge to common general tasks like QA. Notably, other reviewers have acknowledged our evaluation settings (7bDY) and contributions (Reviewers cenq, DfLY and 311c). We will make the point clearer in the revised version! We sincerely thank you for your efforts in reviewing our paper and your constructive suggestions again. We hope we have resolved all the concerns, and we will deeply appreciate it if you could reconsider the score accordingly. We are always willing to address any of your further concerns. [1] Lin et al. On grounded planning for embodied tasks with language models. AAAI 2023. [2] Pashevich et al. Episodic transformer for vision-and-language navigation. ICCV 2021. [3] Shinn et al. Reflexion: an autonomous agent with dynamic memory and self-reflection. 2023. [4] Yao et al. React: Synergizing reasoning and acting in language models. 2022.
Decision
Accept (poster)