Summary
The paper presents Synthetic Experience Replay, a reinforcement learning algorithm employing a generative model based on diffusion to enrich the training dataset of a learning agent. The method is adapted to offline and online RL, and compared to traditional augmentation strategies (e.g., the addition of random noise) in continuous control tasks, from proprioceptive and visual inputs, while applied to different policy optimization algorithms. The paper also presents detailed analyses on the approach, including results on the quality of the generated data and on using larger neural networks.
Strengths
**Originality**: the general direction of using synthetic data for training machine learning algorithms has been vastly explored in the past, with some applications to reinforcement learning. The approach presented in the paper is, thus, not outstandingly novel. However, one of the main points in the paper is that the main bottleneck in previous approaches leveraging synthetic data in the context of RL has been the quality of the generative model. To the best of my knowledge, this is the first use of diffusion models for synthetic data generation.
**Quality**: the quality of the work is reasonably good. The experiments are well-chosen and the experimental designs are generally good.
**Clarity**: the writing quality is generally very high. The paper is well-written and easy to follow in most parts, with a clear presentation of the experimental results.
**Significance**: I believe demonstrating the effectiveness of powerful generative models in generating synthetic data for reinforcement learning algorithms to use is important for the community, given the simplicity and generality of the approach.
Weaknesses
**Major Concerns**
- My main concern is on the number of seeds employed for the experiments in the paper. I understand the authors might be subject to computational constraints, but I find 4 repetitions only to be generally not enough to fully trust an individual experiment. I would suggest the authors to run more seeds per experiment and, if possible, use the methods proposed in "Deep Reinforcement Learning at the Edge of the Statistical Precipice" (Agarwal at al., 2021) for aggregation across multiple tasks.
- In Section 3.1, the paper says that Tabular VAE and Conditional Tabular GAN have been evaluated using the default hyperparameters proposed in the original paper that applies these generative models to tabular data. However, no dataset from the benchmark employed in the original paper contains robotics interactions similar to the one from D4RL, and this thus begs to question of whether better hyperparameters for such different datasets might exist. I believe the paper would be improved by evaluating those approaches after a careful search for optimal hyperparameters.
**Minor Concerns**
- Some potentially important references are missing from the paper. In particular, I'm thinking of the recent paper on using synthetic experiences (meta-learned) "Should Models Be Accurate?" (Saleh et al., 2022), and of some recent papers applying, albeit with other goals, diffusion models to reinforcement learning such as "Planning with Diffusion for Flexible Behavior Synthesis" (Janner et al., 2022) and "Is Conditional Generative Modeling all you need for Decision-Making? (Ajay et al., 2022).
- The clarity of the paper would benefit from a more explicit description of how the diffusion model is employed for generating the data. One can guess it is naively applied to generating each dimension of each state, action or reward, but being explicit is better than being implicit in this case.
Questions
I ask the author to address the concerns highlighted above in my review.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The computational limitations of the approach are stated in the paper, saying that the approach is computationally better than REDQ. However, the comparison is executed at a replay ratio of 20; it would be desirable to understand what is the actual computational cost of the new data generation and training of the diffusion model even at lower replay ratios.