BricksRL: A Platform for Democratizing Robotics and Reinforcement Learning Research and Education with LEGO

Summary

The paper presents a framework for training reinforcement learning policies on LEGO robots in the real world. BricksRL integrates the LEGO robotics hub PyBricks and a reinforcement learning library TorchRL, providing an easy interface to implement and deploy RL algorithms. A robust infrastrure for robot-environment-algorithm communication is implemented for real-world deployment. A camera is integrated in the system beyond LEGO’s standard sensor set, which expands the platform’s capabilities. Experiments on various tasks on three LEGO robots (2wheeler, walker, and robotarm) are demonstrated, validating the system’s capabilities for real-world robotics applications. Ablations are studied to compare sim2real transfer with learning from the real world, and w/ or wo/ camera sensor inputs.

Strengths

Constructing robots with LEGO parts is low-cost and widely available. This paper demonstrates deploying reinforcement learning on LEGO robot, which is very inspiring for low-cost robot research. LEGO parts are modular and reusable, allowing the user to tailor robot designs to specific tasks. This provides a nice platform for cross-embodiment policy research, task-specific robot design, etc. The paper provides an effective solution to deploy reinforcement learning algorithms on low-cost robots, by integrating PyBricks and TorchRL, and implementing a robot-emvironment-algorithm communication via bluetooth connection. The system is proved to be robust by deploying real-world reinforcement learning policies. Detailed experimental results on three LEGO robots are demonstrated, showcasing the system’s capability to train real-world rl policies on low-cost robots.

Weaknesses

Integrating LEGO hub with BricksRL, due to communication overhead, the system frequency is only 11 Hz, which limits the capability of the system to perform certain tasks such as dynamic manipulation tasks. It would be insightful for the authors to provide more discussions and ablations on the robustness of the system. The paper noted that the system has low communication speed, the lack of millimeter-precise constructions, backlash, and noisy sensors. But does the policy learn to be robust to these noises during training? Or are there certain strategies that authors employ to deal with these issues?

Questions

Which simulator is used? Is RoboArm-mixed-v0 trained only using real-world data? Perhaps because sim2real transfer with image is hard? Is the trained policy robust to some slight deviations of robot hardware during explorations, such as shifted parts, sensor drifting, and system latency?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes, the authors addressed the limitations well.

Summary

This paper presents BrickRL which is a system for using reinforcement learning within the context of lego robotics. The paper provides an overview of their setup, how they used TorchRL and PyBricks to interface with the the lego robots. They also provide results that show the feasibility of this system to use off-policy algorithms to train in the real world.

Strengths

Originality: From my undersanding, this is the first paper to combine the lego robotics setup that has been popular in education for a while with RL. They cite other efforts to create low-cost robotics but even those are significantly more expensive than a simple lego setup, which further improves the originalitiy in my mind as a first of its kind very low-cost and flexible setup. Quality: The main claim the authors make in this paper is their method is able to train agents on lego robots that perform well in the real world which they do. It would have been nice to mix both off and on policy algorithms in your results as well as maybe some offline RL just to clearly show the vercitility of your method. That being said, the claim of "agents can learn for lego RL" is clearly answered affermatively (at least for off-policy algorithms) from their results. Significance: Major. This paper on the surface might seem simple as it is just combining lego with RL but in my opinion the significance of low-cost robotics for RL is massive. Researchers with less funding using this system for their work to test different algorithms on low-cost robots that are extremely easy to repair. The results are good enough that this system is clearly working and has the potential to address many accessibility issues with RL in robotics.

Weaknesses

Clarity: The paper has no major errors but can be confusing as it does state things that feel like implementation details in the main paper that should likely go in the appendix and did make it harder to read. Possible improvements: - I think the PyBricksHubClass paragraph feels out of place and a little confusing. This could probably be just a simple "PyBricks provides a class that..." and simply state the benefits. I don't think the fact that it uses BLE is important. - The BaseEnv paragraph also feels out of place. I think that putting it in there and saying that you can use a BaseEnv to create custom environments is redundant. The real benefit here is the second paragraph about enviornment transforms and other models. - The useage of TensorDict feels just like an implementation detail, while it might be critical, these three paragraphs in 3.2 all feel like they could be one or two sentences and then you can move on and save more room for results. - Simlar comment for Client Script and Robot-Environment Communication. It seems just like an implementation detail that might even be completely ok in the appendix. But Communication Speed is a very interesting section that I really like (addresses a possible limitation).

Questions

Some of these questions are going to be in the unanswered limitations as well. - How long do these robots last? Do the motors wear down during training? Are they expensive to replace? - Could you use a wired connection for training? I realize it causes some issues but would it increase training Hz? - How much did the setups you used cost?

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

4

Limitations

The authors don't address many limitations in their paper which I do wish they more clearly did. Scattered around are a few limitations addressed like communication frequency but I think that limitations could be more clearly thought of and addressed.

Summary

This work proposes a flexible and cost-effective platform named BricksRL with LEGO builds aiming to lower the cost of RL research and education. Experiments also demonstrated that LEGO robots can be trained within 120 minutes on normal computers to achieve simple tasks such as moving, walking, and grasping. An offline dataset is also provided for offline RL or imitation learning.

Strengths

- Practical utility has been demonstrated by showing the deployment results of RL methods on three LEGO robots such as SAC, TD3, and DroQ. - This LEGO robot platform is low-cost and flexible with clear instructions and open-source code.

Weaknesses

- BricksRL’s value as an educational platform is clear but the scientific contribution is limited because there's no innovation proposed on algorithms. - A dataset constructed with BricksRL may be of greater value for offline reinforcement learning and imitation learning. - The tasks that BricksRL supports are relatively simple, such as walking, spinning, and reaching.

Questions

- In what way can LEGO robot models be converted to models that can be used in simulation? - Does BricksRL also support urdf model construction or the integration into mainstream simulators like Mujuco or Isaac Sim/Gym?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

As mentioned in Weaknesses, the scientific contribution of this work is limited.

Summary

This paper demonstrates that cheap LEGO robots can be trained in under 120 mins on a laptop to perform simple tasks using sim2real approaches. The paper is motivated by making robot learning research more accessible for educational settings. A software framework called BricksRL is released in open source that integrates PyBricks and TorchRL with Gym environments for controlling modular LEGO-based components, motors and sensors.

Strengths

* The paper is clearly written, with a strong motivation section and a sound implementation of RL algorithms. * The paper is accompanied with instructions on how to build LEGO robots paired with BrickRL for exploring RL/control/robot learning algorithms much more cheaply than with industrial robots. * The finding that simple tasks can be trained for in a couple of hours on a normal laptop is compelling for educational experimentation. * LEGO robots are easily extensible, so one may envision quickly designing higher-degree-of-freedom robots for research

Weaknesses

* The core contributions of this paper are not technically novel - there are no new algorithms introduced, and BricksRL is a standard Gym framework for RL training. * The tasks demonstrated are relatively simple (walking, spinning, reaching). Hence, it is not clear how much value this framework provides beyond educational experimentation in simulation. * The RL algorithms benchmarked may not reflect current state of the art approaches - but this is not the prime focus of this paper. The variance in the plots seems to be extremely high.

Questions

- The paper is currently lacking details on policy architectures used for training. Please add. - Are any of the policies vision-based? As the number of parameters increases, the training time will also increase. This will stress the reliability of LEGO hardware - possibly breakages that may complicate learning on more complex tasks. Some expanded discussion on this would be helpful.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, to some extent. One issue with low-cost robots is easy wear and tear precluding large-scale data collection. The presence of backlash, noisy sensing may also limit experimentation for more precise tasks. Please expand more on limitations in this regard.