Summary
This paper introduces EmbodiedGPT, a new multi-modal foundation model designed for application to embodied (robotic) tasks. EmbodiedGPT combines existing large pretrained (and frozen) vision and language models (namely the ViT-G/14 EVA-CLIP vision model and LLaMA-7B) through the use of an "Embodied-Former" which converts input text tokens and vision features into prefix tokens that can be fed directly to the LLaMA model. To pretrain this Embodied-Former model, this work introduces two new datasets (EgoCOT and EgoVQA), which contain video clips taken from the Ego4D dataset annotated with text captions generated by ChatGPT that, for EgoCOT, describe chain-of-thought-style plans corresponding to the actions taken within the clip, and, for EgoVQ, include questions and answers related to the clip. Using these new datasets and a 3-stage training pipeline, this paper trains the EmbodiedGPT model and, via human evaluations, show that humans generally prefer the captions and embodied plans generated by EmbodiedGPT even when compared to a LLaVA-13B baseline which has many more parameters. Finally, the paper shows that, by attaching an MLP policy head to the output of the Embodied-Former, it is possible to use imitation learning from a small set of demonstrations to train performant physical-manipulation agents in simulation. Experiments show that EmbodiedGPT outperforms ablations of itself and a R3M-based SoTA model for these tasks.
Strengths
This paper studies a very popular, important, and challenging area of work: building foundational models for robotic control and embodied world understanding. The proposed EmbodiedGPT model represents a modest but non-trivial step towards this grand goal. In my view, the major contributions of this are the substantial efforts taken to create the EgoCOT/EgoVQA datasets and execute the complex and computationally expensive 3-stage strategy required for training the EmbodiedGPT model. Generating these datasets and training the model clearly required significant effort and the experimental results appear quite strong.
Weaknesses
While I lean positive for this paper, I will highlight some weaknesses below that temper my excitement.
## Statistical significance of the human results
The only evaluation of EmbodiedGPT against other existing large vision-and-language models comes in a 30-participant human study using 20 tasks (10 captioning, 5 simulated planning, 5 real planning). As this sample size is quite small, and the performance difference sometimes relatively minor, it's challenging to draw meaningful conclusions from the results in Table 1. Moreover, it is hard to know how representative the 20 tasks are of general captioning/planning tasks, how were these chosen?
There are ways to provide a statistical analysis of this data but they are not trivial: you'd need to account for the correlated nature of the data (e.g. that individual participants rated multiple examples and that performance may be correlated with the task type and within each question). While not a perfect treatment, at the very least a paired t-test could be used to compare EmbodiedGPT with the other models.
## Really "foundational"?
While EmbodiedGPT is touted as a "foundation model for embodied AI", the results do not quite support this in my view. In particular, I feel that the EmbodiedGPT misses the critical capacity to handle partial observability and I was surprised not to see any real-world examples of robotic control, more details below.
### Lack of tasks involving partial observability
The only embodied tasks evaluated in this paper involve static robotic arms acting in fully observed (or almost fully observed) environments. There are many embodied tasks (e.g. visual navigation) for which partial observability plays a critical role. From my understanding, the policy component of the EmbodiedGPT model makes decisions from a single visual observation in isolation from all previously seen states and thus would be expected to fail in all such tasks.
### Lack of application to real-world embodied tasks
While it is great to see that EmbodiedGPT outperforms the R3M model in the simulated environments, one of the strengths of the R3M paper was demonstrating that it also functioned well when applied to real robots. While not necessary for acceptance, it's difficult to call EmbodiedGPT a robotic foundation model without at least some real-world results.
## Architectural novelty
This is a very minor point as I consider the training of large models to be a significant contribution in itself, but the proposed EmbodiedGPT architecture is not especially novel: it combines popular pretrained vision/language models with well-known prefix tuning ideas. While you cite Tsimpoukelli et al. (2021), I was a bit surprised to see more discussion of how their work relates to the proposed approach. Moreover, simply searching for "embodied" in the citations of the above paper (i.e. https://scholar.google.com/scholar?hl=en&as_sdt=5%2C48&sciodt=0%2C48&cites=16154696122208258147&scipsc=1&q=embodied) appears to surface several additional citations that should be included in this work.
Questions
I have listed some of my major questions and weaknesses above. If some of these weaknesses can be addressed (especially the point about statistical significance) I expect to increase my score. In the below I include, more minor, line-by-line comments and questions.
Line 60
- "EmbodiedGPT ’s" should be "EmbodiedGPT’s". I suspect this is a problem due to latex macros, try adding {} after your macro inline, i.e. \mymodel{}.
Line 77
- This "200M annotated videos" statement seems like it must be wrong. Later on, you note that there are 2927 hours of video in total, this is approximately 10M seconds of video which would correspond to ~200M frames depending on the framerate, is that what you mean?
Figure 2
- This figure could use some work, for instance, it's very unclear how videos are being input into the model.
Lines 146-147
- What is the value of the "learnable embodied queries"? What makes them "embodied"?
Lines 172-174
- I found this description of the video encoding procedure to be too vague, please add more details.
Lines 181-204
- This entire discussion could be simplified and better presented, the idea here is quite simple and much of the formalism is extraneous. For instance, you introduce a "mapping function (...) M: z -> z'" but this mapping function is just a linear projection and the "M" notation seems to be never used again. You could just say "z is then mapped to z' by via a linear projection."
Line 241
- Extra space after the comma in 9,645.
Lines 225-228
- How is \beta_i "adjustable" if it's defined in this way? It seems to be a fixed function of other fixed values.
- Why not just set [t^{start}_{i}, t^{end}_{i}] to be [(t_{i-1}+t_{i})/2, (t_{i+1} + t_{i})/2]?
Figure 3
- This seems like a surprising qualitative example to choose as the EmbodiedGPT model says many wrong things. E.g. (1) the clock is not actually above the door, (2) the coat hanger is not on the floor, and (3) the coat hangers can be seen towards the top left, not the "near the right side of the doorway".
Figure 4
- Again, this is a surprising example as the video shows it lowering the ring over the cylinder twice whereas EmbodiedGPT seems to only suggest this happens once.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
I would have liked to see some discussion of the limitations I discussed in the _Really "foundational"?_ section of the weakness. Otherwise, limitations are reasonably well addressed.