Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL

Summary

The authors contribute a mathematical proof showing that transferring a "exploration strategy" in sim2real scenarios as opposed to transfer the learned policy result in an exponential improvement on samples needed to learn the real task.

Strengths

- While not necessarily a fundamentally new idea (the transfer learning and sim2real areas have been long avoiding the naive transfer of policies and biasing exploration has always been one of the chosen strategies to perform transfer), the theoretical results presented confirm and fundament this idea that has been informally explored by the literature. - The authors present a practical, easy to understand, exploration method and show that it overperforms greatly a simple policy transfer method in a somewhat complex sim2real robotic evaluation. - The topic explored is not only adherent to neurips but also very timely in supporting more targeted efforts to apply RL to novel practical domains.

Weaknesses

- I am not sure why a new basic "sim2real" formalization was used to in this paper when it already exists. The problem explored in this paper is completely equivalent to Multi-Fidelity MDPs (described in {1}), in fact, MF-MDPs even have something not incorporated in the description provided that is the cost of sampling in each fidelity. The MF-MDP paper was not even cited so I assume this happened because the authors have not read the paper. I suggest the paper is rewritten to describe the problem as an MF-MDP for standardization (there shouldn't be any impact to the conclusions or contribution fro the paper), but even if not rewritten at the very least this paper has to be cited and incorporated to the related works: {1} - Silva, F. L., et al. Toward multi-fidelity reinforcement learning for symbolic optimization. ALA workshop @ AAMAS, 2023. - Another thing that made me be in the fence regarding my score to this paper is the lack of baselines for the experimental evaluation. There has been many papers exploring a similar idea in the past, so I feel like they should have been incorporated to the experimental evaluation (Cuttler {2} comes to mind immediately, but you also have other newer options).. {2} Cutler, Mark, Thomas J. Walsh, and Jonathan P. How. "Reinforcement learning with multi-fidelity simulators." 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014. - A limitation both in the theoretical and practical side of the paper is that it is assumed that the transfer will happen "one shot", neglecting the fact that either the simulator could be modified to get closer to the real world {3} and the learning process in the simulator repeated, or the approach could already account that the simulator and real world will be different and adapt for that (e.g. {2} and {3}). The present paradigm presented by the authors to "learn from the simulation once and never get back to it" is at least an incomplete picture of all that could be done. {3} - Hanna, Josiah, and Peter Stone. "Grounded action transformation for robot learning in simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.

Questions

- Why is it assumed that the transfer HAS to be made one-shot (transferred completely in the beginning of the learning process in the "real environment")? Wouldn't it be better to have the exploration transfer strategy work as a "loop" where the exploration could be revised according to how the real world differs to the simulator? - Why weren't MF-MDPs used to describe the problem? - it is stated that the conditions for us being able to find a policy that solve the problem in the simulation is that ϵ ≥ 2H^2ϵ_sim. What is the meaning of ϵ here? Finding conditions for a direct policy transfer to work would also be an important contribution, specially if there is some way to test for those conditions with some samples from both the real world and simulator.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Basic research, no potential negative societal impact to explore.

Summary

The authors propose a method where, instead of directly transferring a trained policy from a simulator to the real world, exploratory policies are learned in the simulator and transferred. This approach aims to enable efficient exploration in the real world, particularly in low-rank MDP settings.

Strengths

1. The idea of transferring exploratory policies instead of directly transferring policies trained in simulator is interesting and can potentially help sim-to-real transfer.

Weaknesses

1. The approach relies on several assumptions (e.g., low-rank MDPs, specific access to simulators and oracles) that may not hold in all real-world scenarios. 2. While the method shows promise in specific settings, its generality to a wider range of RL tasks and environments remains to be demonstrated. 3. Limited comparison with other state-of-the-art methods, especially those addressing the sim-to-real gap through domain randomization or adaptation.

Questions

1. How robust is the method to violations of the low-rank MDP assumption? 2. Can the approach be extended to more complex MDP settings? 3. How does the method scale with the complexity of the task and the size of the state/action spaces?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

As in weaknesses.

Summary

The paper shows that for transferring with a large sim-to-real gap, exploration policies have improved capabilities compared to optimally pre-trained policies, which tend to overfit the simulation up to a point where exploration is not sufficient to adapt to the changed circumstances.

Strengths

Overall, the paper is easy to follow and theoretically sound. It includes theoretical deliberations and empirical results with a convincing experiment design. I expect the considered perspective on sim-to-real shift to positively impact research in this area.

Weaknesses

Overall, providing further examples for the theoretical elaborations could help comprehensibility (e.g., Proposition 3). Further minor issues include: - Definition 3.2: PAC is not defined, what is $\delta$ used for? - In Proposition 1: Is the success of the transfer dependent on the horizon of M? - In Proposition 2: What does $\Omega$ refer to? - What is the difference between $\epsilon$ and $\epsilon_{sim}$? - Why is the transition probability $P$ defined to be dependent on the horizon H? Also, I am missing a thorough definition of the Low-Rank MDP, which seems central to the theoretical elaboration. Even though the experimental setup is convincing, I am missing comparisons directly training $\pi$ in the real task as a baseline. Also, to better illustrate and link the aforementioned theoretical elaboration, providing the distance between the considered benchmark tasks would have been interesting. Furthermore, more detailed comparisons (both experimental and theoretical) to advanced pre-training approaches like APT (Liu et al., 2021) or DIAYN (Eysenbach et al., 2018) would further enhance empirical credibility.

Questions

What are the implications of using a low-rank MDP? If the simulation is considered to be free, wouldn't learning $\Pi$ optimal policies improve the transfer (similar to using a pool of exploration policies)? Regarding the transfer described in Alg. 1, are all exploration policies updated, or only the best-performing? If we were to consider even larger shifts, is there a point at which exploration policies no longer provide benefits, and learning from scratch is the most sufficient approach? When considering the transfer from simulation to the real world, pre-training is often motivated by the need to ensure a certain level of safety for execution in the real world. How does this concern affect your approach?

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations have been addressed by the authors but could have been extended.