Summary
The paper proposes a set of "Benchmarks" for Offline Reinforcement Learning. Assuming that the RL practitioner can be equipped with some data about the environment plus an imperfect simulator of the "world", the authors identify four main ways in which the data and simulator can "confound" the RL agent, resulting in suboptimal policies. Guided by those four "challenges" for hybrid offline RL approaches, the authors generate synthetic datasets and prepare custom configurations using the Mujoco simulator. They also provide some analysis of how those four aspects affect some standard learning RL methods.
Strengths
- There is value in the identification of the main challenges that an RL agent might encounter, especially in the identification of the "Confounding bias", which is not as trivially thought of as the other challenges.
- If made freely-available, the datasets/implementations might assist in the development of new hybrid offline+online RL methods.
- The authors have a comprehensive view of all the resources an RL designer might get access to and consider it in their modeling.
Weaknesses
- The main problem to me is that the work proposed is too incremental for a main track paper at ICLR. While there is value in the identification of the main challenges faced by the RL agent in the offline data+simulator scenario, it is a very incremental step from already-existing datasets/works, and procedurally, generating those benchmarks are relatively easy given Mujoco is a simple to execute and easily-available environment. For instance, someone interested in generating a similar benchmark might not even have to run the environment too many times by leveraging already-existing datasets. Starting from (Fu et al, 2020) datasets, one could modify the datasets by simply (i) changing an state variable; (ii) omitting a recorded state variable (iii) adding noise to the recorded action; (iv) adding noise to observation and action, to replicate a similar dataset generation as the one done by the authors, maybe 2 or 3 of the generated configurations only would require running all the experiment again (and even those, represent a small change in the Mujoco environment).
- A lot of work can be done to improve clarity of the work as well. It's not very clear how the datasets replicate the "structure" in Figure 1. The offline data and simulator construction is explained, but how is the agent performance validated? Is there a ground truth environment that is not modified from which performance measures are extracted?
- While the authors' descriptions make sense, there is not enough evidence that the four challenges identified by the authors actually represent what happens in a real environment. I strongly recommend the authors to include a more complex and less-controlled environment than Mujoco, doing the actual work of gathering data and building a simulator for the environment, and showing that those challenges actually represent practical reality. Alternatively, a domain where there are simulators of different fidelities could be used, that way the authors wouldn't need to carry out the work of developing a new simulation.
- I was surprised the authors didn't mention the very related multi-fidelity MDP formulation. The formulation models exactly one of the problems the authors are coping with, imprecise simulators.
Silva, F. L., Yang, J., Landajuela, M., Goncalves, A., Ladd, A., Faissol, D., & Petersen, B. (2023). Toward Multi-Fidelity Reinforcement Learning for Symbolic Optimization. Adaptive and Learning Agents (ALA) Workshop.
- In order to strengthen the contribution of the paper I suggest the authors follow one of the avenues:
1) Develop a new Hybrid algorithm where you can show clear performance improvement across most of the benchmarks
2) Introduce a more complex environment to the benchmarks.