Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

Summary

The paper introduces a study of pre-trained visual encoders for Embodied AI, and produces, among others, a new pre-trained ViT-L encoder (VC-1) using Masked Auto-Encoding on a dataset combining an expanded collection of egocentric video and ImageNet. VC-1, when adapted for agents solving 17 task types covering dexterous manipulation, locomotion, mobile manipulation, and navigation, performs best in average. The curated set of 17 task types is presented as a new benchmark named CortexBench. The results show that no visual encoder is universally dominant, and that just scaling the pre-training dataset size and diversity does not imply uniform improvements across benchmarks, refuting previous extrapolated results. Further experiments on real hardware are also included, showing the improvement in comparison with the best performing existing Masked Visual Pre-training visual encoder.

Strengths

- Thorough study of pre-trained visual encoders for Embodied AI tasks across different pre-training schemes. - Provides a reference model with high performance (especially after adaptation) across all considered task types. - Thorough experimental section and introduction of the very relevant CortexBench benchmark. - The paper reads really well.

Weaknesses

- In an ideal world, it would have been wonderful to have found a single pre-trained model that could be used without adaptation with high performance across all tasks, but the actual outcome is thoroughly justified in the paper. One open question is whether, with larger and more diverse datasets, a single pre-trained model could eventually be the absolute winner before adaptation, but I agree this is out of scope for this paper.

Questions

- Line 370 states "MAE adaptation does not help TriFinger performance, this is likely due to the mere 300 real robot images available for adaptation.". However, fine-tuning seems to do a good job in this case, contradicting the general consensus derived in Section 6. Is there an intuitive explanation? - Why is prompting/conditioning VC-1 not considered as an adaptation method in the study? It might be a viable not-so-costly alternative to fine-tuning. - Is the advantage of R3M indicating that known architectures might be far from optimal for the goal of having a single visual encoder able to feed all EAI tasks or is the scale of the pre-training datasets considered in the study the most likely reason?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The paper discusses adequately the limitations.

Summary

• This paper presents the largest and most comprehensive empirical study of pre-trained visual representations for Embodied AI based on proposed CORTEXBENCH. • This paper focuses on studying the effect of pre-training data size and diversity for PVRs, further proposing VC-1. • This paper shows that task- or domain-specific adaptation of VC-1 leads to substantial gain by E2E fine-tuning or MAE adaptation.

Strengths

• This paper proposes a comprehensive benchmark CORTEXBENCH for pre-trained visual representation evaluation of embodied intelligence, which is very valuable and meaningful. • This paper studies the effect of pre-training data size and diversity for PVRs, which is novel and insightful.

Weaknesses

• There may be some typos in experiments, such as ‘Trifinger by+7.4 (72.7 → 80.1)’ in the 343 line, and according to the table, it should be Trifinger (71.7 → 80.6). Please carefully check. • Although I think this work has been done well enough. However, there are still many questions that can be explored: based on ViT and the same data, which is better for MIM and time-contrastive video-language alignment. Is ResNet better than ViT when few-shot imitation learning? • This paper only attempts to adapt VC-1, but there are no corresponding experiments for R3M and CLIP. • I strongly recommend including a concise introduction to multimodal-based pre-training visual representation in the related works section, citing relevant sources such as [1-5]. This is crucial as multimodal presentation learning now plays a pivotal role in numerous vision tasks. [1] Instruction-Following Agents with Multimodal Transformer. [2] Language-Driven Representation Learning for Robotics. [3] Open-World Object Manipulation using Pre-trained Vision-Language Models. [4] Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots. [5] LIV: Language-Image Representations and Rewards for Robotic Control. Among them, [3] and [4] explicitly adapt the middle-level output of the pre-trained model through larger policy models, which also utilize the raw image as input.

Questions

• I'm curious how you choose specific tasks under each benchmark. Does this follow some basis? • I'm curious why you did not model the temporal information in policy model. For example, adding temporal attention in ViT, even if utilizing a large amount of video data for pre-training.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

• Although the evaluation of this work is already very comprehensive, I believe that 17 tasks are not sufficient. I found that even for the same benchmark, different tasks exhibit significant differences.

Summary

This paper focuses on the analysis of large-scale pre-trained vision models for interactive tasks. The authors curate CortexBench which includes tasks covering dexterous manipulation, navigation, and also scene-level interaction as the testing benchmark. Further experiments were made on testing existing pre-trained vision models on this task. The authors also control the dataset type, the scale used in pertaining and proposed their VC-1 model pre-trained in an MAE fashion which shows superior performance compared to baselines. Additional analysis were made showing the effectiveness of egocentric images for interaction and also gap between existing pre-trained vision models and the desired "visual cortex" representation.

Strengths

This paper conducts solid experiments on testing existing pre-trained visual representations (PVR) on their own curated interaction benchmark CortexBench. Both the evaluation benchmark and released checkpoints (as promised by the authors) will be extremely beneficial for embodied AI research. The overall analysis is technically sound with proper conclusions made, pointing to future directions for PVR studies.

Weaknesses

One concern of this paper is that except for the conclusion pre-training with egocentric frames is beneficial for interactive EAI tasks, I do not see signature conclusions that could be drawn from the current analysis. After reading this manuscript, I do not feel that the question in the paper title has been properly addressed: - First, there is no task analysis for the curated tasks in CortexBench, for a proper visual cortex, to what extent should we expect with "visual cortex"? are these tasks equally important? are there any relationships between these tasks? - Second, since the current VC-1 model still does not universally dominate either task, what is missing on the training/model side? is it missing for example value condition in VIP or some embodied experience? - Third, as a pre-trained vision representation, what should we expect from VC-1 in addition to its capabilities in CortexBench? Should we expect the model to at least have some sort of objectness as DINO? Can this pre-trained model be used in more general vision settings?

Questions

Please see the weakness section for questions.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have properly addressed the limitations of this paper.

Summary

This paper evaluates different pre-trained visual representations w.r.t. their capability to serve as foundation models for Embodied AI. To do so, they compile a dataset (CortexBench), with 17 simulated EAI tasks. The intuition here is that a single pre-trained visual representation which would perform well across all tasks could be considered as an Artificial Visual Cortex. The paper collects several egocentric video datasets starting with Ego4D, and sequentially adding datasets to create different variations trained with different training datasets. Their proposed artificial visual cortex (VC-1) is trained on all datasets (ViT-L trained with Ego4D + MNI). They also propose adaptations to VC-1 to improve its capability to perform well across different datasets. Finally a proof of concept hardware experiment using VC-1 to learn policies with imitation learning for few shot manipulation tasks.

Strengths

1. Exhaustive training with lots of variations with architectures and increasingly growing dataset. 2. Evaluation conducted on a large scale dataset with a diverse set of EAI tasks. 3. Writing: The paper is easy to read and follow. The figures do a good job of illustrating the dataset and the experiments. Also, the.

Weaknesses

1. No data contribution: CortexBench is a compilation of existing datasets, and it's not clear what new it brings to the table beyond what's already out here. From the introduction, it initially seemed as thought it's newly proposed benchmark in this paper. However, it seems to be a set of already existing datasets. Why is it being claimed/renamed as a new dataset? 2. No new findings: - The first negative result is that specialized pre-trained models are better - that is, in-distribution models perform better than out-of-distribution ones (lines 42-45). This is well known in the machine learning community, and thus, is not a new finding. - Lines 70-85 can be summarized as saying: More data is better, but OOD data is a problem. This too is a well known property of deep learning vision models. - Naively scaling size and diversity: The paper does not study data diversity explicitly, but rather only 3. Data diversity and Dataset size are not controlled: When the new datasets are added to the foundation models, are the number of images held constant? I didn't see this detail in the paper. Thus, it is unclear what helped - dataset size, or diversity? To control data diversity independently, a control would be to reduce number of images per dataset as # datasets is increased to isolate the effect of size and diversity. 4. The adaptive VC-1 does not really work well: of the datasets with prior results in Table 5, VC-1 does better for only 2 task classes. Without adaptation too, it could be trained to do well on one of these task classes alone. Thus, the adaptation took it to performing well from 1 task to 2 task classes. Furthermore, there are no error bars in these results which makes it hard to evaluate the reliability of these findings given the stochasticity of the models. 5. Claim too grandiose: Calling a pre-trained visual recognition model an artificial cortex is a very far fetched claim on several levels of abstraction. Firstly, the cortex does much more than EAI: so testing it only on a small set of functionalities of the cortex means there's a mismatch at a behavioral level between the cortex and the artificial cortex. Secondly, by not comparing with any neural data, there is a mismatch at the mechanistic level as well. A more objective representation of the work done here is that this paper investigates --- Can a single pre-trained representation perform well across a diverse set of EAI tasks?

Questions

Please refer to weaknesses above for questions.

Rating

2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

N/A

Thank you for the rebuttal

I would like to thank the authors for the detailed responses to the reviewers' questions. I reopen some questions that I think were misunderstood or unclear in my original review. - The first unclear question was regarding fine-tuning alternatives: > It is unclear to us how to produce prompts for the tasks in CortexBench, as there is no standard approach that we are aware of. However, we’d be happy to add more comments if the reviewer could provide more details about the prompts they had in mind. I forgot to add the corresponding lines in my original review, but I actually referred to Lines 297-298 in the original manuscript: > We use a broad definition of adaptation [50], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [51] ... Apologies if I am mistaken, but this line of research is no further discussed (or explicitly discarded) for the rest of the study, so it might have been worth it to add on line explaining why this mentioned option is not (or cannot be) considered, despite its potential advantages/interest in other domains. - The second unclear question was about the potential advantages of R3M compared to VC-1 and possible impact in upcoming models. The original manuscript (Lines 288-291) states: > When compared to R3M, VC-1 demonstrates superior performance on average and on 4 of 7 benchmarks (Figure 4). R3M outperforms VC-1 on Adroit, MetaWorld and DMControl benchmarks. It is unclear whether this gap is caused by the different training objective, pre-training dataset, or backbone. In recent years, transformers have been thoroughly used as the standard architecture in a large (and extremely varied) amount of domains, principally due to their advantageous scalability properties. However, it was impressive to see R3M (whose architecture is based on ResNet) perform so well in comparison to VC-1 based on transformers. Without being very concrete in my question (and obviating other important factors like the ones correctly mentioned by the authors in their response), I was meaning to start a discussion on the importance of research of new architectures that could hopefully bring the best of both worlds (inductive biases and scalability) in a principled way. In any case, I am happy with the provided response

Post-rebuttal response

The authors' responses have clarified some of my concerns. I still feel that there are still more analyses to be made to make this benchmark design more meaningful and complete (instead of selecting the most common or popular existing EAI tasks). Therefore, I'm keeping my original rating as borderline accept, however, I will not argue for rejection.