RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation

Summary

This paper introduces RoboMamba, a Multimodal LLM, for robotic reasoning and low-level manipulation using the popular State-Space Model architecture. RoboMamba introduces a multi-stage joint language and vision training pipeline with robot-specific data fine-tuning. RoboMamba demonstrates strong reasoning performance on general and robot evaluation benchmarks while using a small amount of fine-tuned parameters.

Strengths

- Paper is well-written and easy to follow - Well-motivated and comprehensive set of experiments on different general reasoning benchmarks and robot manipulation specific benchmarks - Training dataset and evaluation details are clearly stated

Weaknesses

- There is not much technical novelty in the method. It seems like RoboMamba is combining CLIP with SSMs and fine-tuning on robot-specific data (RoboVQA) which is a generic pipeline that several other MLLMs works have proposed. There are no new insights from the paper. - The parameter-efficiency argument is mainly due to the use of SSM blocks instead of Transformer blocks. I think this is already obvious given the architecture design and is not a major contribution of this work. - Using the term “real-world experiments” is confusing because it implies there is a physical robot executing a trained policy to perform some task which is not the case in this paper. Please provide quantitative results for the real-world reasoning experiments if possible and also provide some failure cases of your method otherwise it seems like the qualitative examples are cherry-picked. - Please provide explanation as to why the RoboMamba training strategy may help learn better representation or capture knowledge that baselines or prior work fail to. - Ablation study in Table 4 seem to suggest that the different fine-tuning strategies doesn’t seem to provide much benefit in terms of the general reasoning benchmarks. The improvement in downstream performance is minimal if you take out certain datasets. What about for the robot manipulation benchmarks? - It would be nice to see experiments showing the scalability of RoboMamba, e.g. more SSM blocks, more data, etc.

Questions

- Why would the baselines perform worse than RoboMamba if they're trained on the same data, unless the main factor is the architectural choice? - Similarly, why would RoboMamba perform better than general VLMs on those benchmarks in Table 1 and how is this comparable if they are not trained on the same data?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors mention that using MLLMs for task planning occasionally results in hallucinations.

Summary

This paper proposes a Mamba-based framework called RoboMamba that utilizes multimodal state space model for robotic reasoning and manipulation. It addresses the limitations of existing MLLMs in complex reasoning and high computational costs. RoboMamba integrates a vision encoder with the Mamba model to align visual data with language embedding, enabling robust visual reasoning and action capabilities. The model uses a simple lightweight policy head for action pose prediction, requiring minimal fine-tuning. Experiments show RoboMamba excels in reasoning and pose prediction tasks, achieving faster inference speeds and state-of-the-art performance in both simulated and real-world environments.

Strengths

1. Using architecture with linear computational complexity like Mamba for improving efficiency fits quite well with the robotic reasoning and manipulation. 2. The design of training strategy is simple and high-efficient, verifying the transferability of the proposed framework. 3. Experimental results on public benchmarks as well as simulated and real-world environments demonstrate the effectiveness and efficiency of the proposed method.

Weaknesses

1. Besides Mamba-like architecture, there exists many network architectures with linear computational complexity, like XCiT[1], xLSTM[2], hence my first concern is can these architectures work for the proposed framework? [1] Ali, Alaaeldin, et al. "Xcit: Cross-covariance image transformers." Advances in neural information processing systems 34 (2021): 20014-20027. [2] Beck, Maximilian, et al. "xLSTM: Extended Long Short-Term Memory." arXiv preprint arXiv:2405.04517 (2024). 2. Following the first question, does Mamba has some unique attributes and advantages specially fit the robotic field? This work lack of related comparison and analysis. 3. The Fig.1 seems a bit of rough, for example, the formats and colors of arrows are inconsistent, some borders have jagged edges, and the overall resolution is not high enough.

Questions

Please refer to the weaknesses of the paper.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the weaknesses of the paper.

Summary

This work introduces RoboMamba to leverage SSM model’s capabilities in non-trivial sequence modeling with linear inference complexity. A simple policy head is employed for finetuning to enable RoboMamba to predict action poses. Evaluation is conducted both in simulation(SAPIEN) and real-world settings and shows consistent improvement compared to previous sota methods such as ManipLLM. Besides, fine-tuning parameters and time are significantly reduced and inference speeds achieve 7 times faster than existing models.

Strengths

1. On both general and robotic evaluation benchmarks, RoboMamba on average all shows strong performance. 2. Compared to previous methods, RoboMamber has fewer fine-tuning parameters and time, faster inference speed.

Weaknesses

1. In Table 1, RoboMamba(2.7B)’s performances on MME, MMB, MM-Vet are still below TinyLLaVA’s results. 2. In Table 2, RoboMamba does not achieve sota performance on some of the both training and test categories. 3. Some minor mistakes like: In Figure 3(b), “Openfalmingo”

Questions

1. Can RoboMamba support more complex robot manipulation tasks? 2. For delicate manipulation objects, RoboMamba seems have inferior performance in Table 2. Could you provide some analysis for this?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

1. As mentioned in Question 1, RoboMamba may not support more complex manipulation tasks. 2. RoboMamba only output 3D poses instead of direct policies.

Summary

This paper proposes RoboMamba, which applies the Mamba state space model architecture for robotic manipulation policy learning. Prior MLLM-based robot policy learning finetunes transformer-based models and suffers from two major problems: reasoning capabilities degrade with visual input; and training computational cost is high for end-to-end policy learning. RoboMamaba leverages the efficient Mamba architecture to improve learning efficiency and demonstrates simple vision encoder enables visual understanding. Similarly, a simple policy head is shown to be sufficient for generating performant robot behavior after RoboMamba had converged on robotic reasoning tasks.

Strengths

This paper’s major contribution is tailoring a state space model for robotic manipulation control. RoboMamba starts with general pretrained vision and language models and fine-tunes different parts of the network for policy learning while maintaining reasoning capabilities of the model. First, it introduces a visual input encoder for Mamba architecture and demonstrates its effectiveness to enable visual understanding within a pretrained Mamba model. Second, to pretrain the model for policy learning, it trains the model on both general VQA and robot VQA datasets. Last, it learns a light-weight policy head for predicting end-effector poses. Experiments show RoboMamba retains performant reasoning capabilities in VQA tasks while being able to generate competent policies.

Weaknesses

The evaluation of robotic manipulation tasks is limited to one benchmark and contains only single-step open-loop action prediction (instead of learning a closed loop reactive policy). It is unclear how much training data is being used and how much variation there is in testing the policy. At the same time, this model has many moving parts and design choices that were not carefully ablated. The presentation of this work can be improved in general for clarity. The authors could better motivate certain design choices as well as evaluation metrics.

Questions

What limits evaluating RoboMamba to predict one-step action? How does it generalize to multi-step or closed loop policy? Is inference spped, i.e. control frequency, the major bottleneck? How does RoboMamba compare with more structured LLM policies such as Code as Policies [1] and VoxPoser [2]? [1] https://code-as-policies.github.io/ [2] https://voxposer.github.io/

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

see weakness

Summary

The paper proposes a method for robot reasoning and manipulation by developing a multimodal large language model (MLLM) based on the Mamba state space model (SSM). The joint model is able to both attack robot reasoning tasks assessed via VQA, and solve robot manipulation tasks by fine-tuning a dedicated head that contains only a small fraction of the model parameters.

Strengths

The paper introduces an MLLM based on the Mamba model built specifically for robotic tasks. The exposition of the method is clear and well motivated. The proposed method takes advantage of the linear complexity of Mamba and, in combination to the reduced number of parameters used for the manipulation policy head, achieves higher efficiency and better performance with respect to state-of-the-art robot MLLM methods. Another strong point is the combination of natural language reasoning abilities and task specific manipulation policies based on the MLLM output. The experimental evaluation is comprehensive, considering both simulated and real-world experiments, where the proposed method achieves competitive results with respect to SOTA methods across multiple benchmarks.

Weaknesses

Although time efficiency is one of the major claimed contributions of the work, the aspect of time efficiency is not well covered in the experimental evaluation. The fact that the proposed method is 7 times faster than SOTA MLLM methods is mentioned a couple of times, however detailed results regarding inference times are not provided. Another aspect not covered in detail, regards the effect the number of parameters in the manipulation head has on the performance and efficiency of the manipulation task. Another question that comes to mind, is whether it is possible to apply the idea of fine-tuning only a small-sized manipulation head to other MLLMs. Does Mamba work better with this constrained policy head and, if yes, why? Finally, another issue regards reproducibility, as details regarding the structure of the projection layers and the MLPs used in the manipulation policy head are not provided. ### Minor comments - L.13: "action pose prediction abilities" is not very clear - L.100 "ManipVQA, enhancing robotic" verb missing - L.192: what is the angle representation considered? - L.235: "on when"

Questions

- Can you please provide results regarding comparison with SOTA methods in terms of time efficiency? - Can the training strategy, including the fine-tuning of only the manipulation policy head, be applied to other MLLMs? - Is the performance in the manipulation tasks affected by the number of parameters in the corresponding head and how?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed in the text.

Post-rebuttal comments

Thanks for the effort of making the response. The rebuttal and the supplemented experiments have basically addressed my concerns. The extended ablation studies in response to comments from other reviewers further verify the effectiveness of the proposed methods. Therefore, I will keep my score.