MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility

Summary

This is a "tech report" that introduces MetaUrban, a simulator that focuses on micromobility, consisting of low-speed moving objects such as pedestrians, bikers, small ground robots, and wheelchairs. The functionality of the simulator seems pretty strong, and it fills the gap of a simulator for applications such as low-speed robots in urban areas.

Strengths

1. The simulator functionality is quite strong, supporting various terrain designs, static obstacles, and moving agents. 2. There are ways to easily retrieve the desired materials for building urban scenarios 3. The authors provide benchmark statistics for RL/IL agents with the simulator, demonstrating its usability 4. It fills the gap of simulating urban low-speed scenarios and is useful for many applications 5. It comes with a dataset for urban scenarios.

Weaknesses

This is a tool that enables future research, clearly there will be feature requests to make it better and easier to use, but I don't see an clear issue with the design.

Questions

1. Will it support easy deployment of agent policies instead of ORCA? 2. What is the runtime performance of the simulator? Is it efficient to run?

Rating

8

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Summary

The paper presents MetaUrban, a simulation platform designed for AI-driven urban micromobility research. It generates interactive urban environments to train and test AI systems like delivery bots and electric wheelchairs. The platform supports diverse layouts, obstacles, and pedestrian interactions through procedural generation, real-world object retrieval, and dynamic agent simulation. MetaUrban-12K, a dataset created from the platform, is used to benchmark Point Navigation and Social Navigation tasks with different AI models. The paper shows how variations in machine design impact performance and highlights the platform's potential to improve the generalizability and safety of AI agents.

Strengths

1. The platform supports varied human and mobile agent behaviors, providing a rich set of interactions that better simulate the dynamics of crowded urban settings for long-horizon tasks. 2. The procedural generation of diverse terrains and object placements creates highly varied environments to help train robust models. 3. The paper explores the impact of different mechanical designs on mobile machines.

Weaknesses

1. The main goal of such a large-scale simulator is to eventually transfer learned models to the real world. However, there is limited experimentation with real robots, and details on how well models trained in the simulator generalize to real-world sensor data are unclear. More real-world testing is necessary to demonstrate effective transferability. 2. The paper does not specify if robots, pedestrians, and other mobility devices can be controlled via standard interfaces like ROS. Without ROS integration, robotics users may need to modify their software pipelines, limiting the platform's usability. This lack of integration would also hinder the deployment of trained models onto real robots, making the simulator less practical for real-world robotics applications. 3. It is unclear how the simulator synchronizes with model updates during benchmarking. If the simulator runs ahead of model updates, this could lead to model decisions being applied to future time steps, potentially reducing the accuracy of training and evaluation. A more detailed explanation of time synchronization between the simulator and model updates is necessary to ensure the integrity of the training process. 4. While pedestrian behavior is simulated, there is little exploration of how AI models handle complex, unpredictable social interactions. Expanding evaluations in this area would improve the relevance of the social navigation task. 5. The paper does not fully explore the performance impact of different sensor configurations (e.g., lidar, cameras). A detailed comparison of sensor types would provide practical guidance for applying models to real-world scenarios. 6. The paper does not mention if the simulator can run in distributed systems or scale across multiple machines. Also can the simulator run in headless mode so that it can be scalably deployed on a cluster. Can the simulator run faster than real time to make model training and evaluation quicker. 7. The paper does not discuss how environmental factors like weather or lighting conditions (day/night) are handled. 8. The platform includes dynamic agents but does not explore multi-agent coordination, competition, or collaboration. More emphasis on multi-agent learning and interactions could make the simulator useful for tasks involving coordination among multiple robots or agents.

Questions

Refer to the weaknesses section.

Rating

8

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Summary

This paper proposed MetaUrban, a simulator for micro-mobility in urban environments. Unlike existing simulators, MetaUrban simulates agents in urban environments while navigating along paths, sidewalks, etc. This is different from existing simulators that either focus on indoor navigation or urban driving. Environments are procedurally generated. The procedure begins by first generating a ground plan that specifies where roads will be, sidewalks, crosswalks, etc. Then terrain and objects are added. To determine what objects to add to a given scene, the authors propose to use real-world urban object distributions and design a pipeline towards this end. Finally, other dynamic agents are added to the scene, such as humans, vehicle, and other robots. Trajectories are created for these agents using the ORCA social forces model and the push and rotate algorithm to resolve deadlocks. The authors then evaluate RL, IL, and Offline RL methods on two tasks, PointNav and SocialNav.

Strengths

The proposed environments have been very carefully designed to represent real-world urban scenes. The paper is well-written and easy to follow. Further, the simulator and choice of tasks is well-positioned and well-motivated. There is an extensive Appendix that provides additional details, experiments, and context.

Weaknesses

The performance of the simulator is a weakness. RGB rendering is, at best, 65 FPS, which is orders of magnitude lower than would be ideal and at least an order of magnitude lower than would be "good". I encourage the authors to see if there are features that can be turned off for the sake of performance if researches want to make that trade-off. For instance, can visual fidelity be reduced (by disabling things like shadows), or can animations be disabled to increase performance? I see the value of including these features for sake of full-fledged evaluation, but these may not be necessary for training and are likely not needed for initial experiments.

Questions

See above.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3