Author Rebuttal (1/2)
We would first like to thank you for your review! We’re pleased that you found our work well written and noted the importance of our findings regarding the augmented replay ratio. We believe we can address your comments and questions with a few minor clarifications in our revisions, and we hope to discuss these clarifications with you.
## **Core messages of our empirical analysis:**
In our submission, we provide two practical guidelines:
1. **Data augmentation (DA) should focus less on generating reward signal and more on increasing state-action coverage.** In our experiments, most – and sometimes all – of the benefits of a DAF can be explained by an increase in state-action coverage alone, and increasing reward density beyond a small threshold typically provides no further benefit. This guideline contrasts with the core motivation of HER and its follow-up works [1, 2] which focus on generating augmented reward signal to improve data efficiency.
2. **When it is possible to generate a large amount of augmented data, use this data to decrease the replay ratio** (rather than to increase the batch size used for updates.)
We’re pleased that our second guideline was clear, and we hope our response clarifies the first guideline. In our revisions, we will better emphasize these guidelines.
## **Differences between our work and prior works studying visual augmentations:**
This is a great question that we can clarify with a few minor additions to the related work section. There are a few fundamental differences between our work and the related works you’ve provided:
1. Most prior works – including all three you have listed – focused on introducing new types of data augmentation functions (DAFs) or frameworks and demonstrating that they can boost the data efficiency of RL. To the best of our knowledge, our work is the first to systematically investigate which aspects of DA are most responsible for observed improvements in data efficiency.
2. Raileanu et. al [3] and Ko & Ok [4] use augmented data for auxiliary tasks, while we focus on using augmented data for model-free updates. Moreover, these auxiliary tasks aim to make the policy and value function invariant to augmentation, i.e. $\pi( \cdot | s) = \pi( \cdot | \tilde s)$ and $V(s) = V(\tilde s)$, and cannot be used with the dynamics-invariant DAFs we considered. For example, the optimal action generally changes if we translate the agent or goal, so it is undesirable to have $\pi( \cdot | s) = \pi( \cdot | \tilde s)$.
3. As you noted, all three works you listed consider visual augmentations that only transform the agent’s state and generate augmented data with the same semantic meaning as the original data. As such, they study DA for representation learning. In contrast, the dynamics-invariant DAFs we consider generate augmented data with a new semantic meaning and thus study DA to improve exploration.
## **Novelty of our framework:**
The core novelty of our framework is that it permits fine-grained control over many hyperparameters relevant to DA, allowing us to systematically study different aspects of DA.
Crucially, our framework guarantees that the agent uses the same ratio of augmented to observed data (or update ratio) in each update by sampling observed and augmented data from separate replay buffers. Existing DA frameworks [*e.g.*, 5,6] and applications of DA [*e.g.*, 7, 8] sample data from a shared replay buffer containing both augmented and observed data, so the update ratio can change across updates. Moreover, with a shared replay buffer, the update ratio and the augmentation ratio (the number of augmented samples generated per observed sample) are entangled; increasing the augmentation ratio increases the update ratio.
Given that DA is so widely studied and that most prior work focuses on improving over baselines rather than understanding why certain methods perform well, we hope the analysis enabled by our framework is the first of many which move us towards the development of further practical DA guidelines.
We hope our response clarifies our work's novelties with respect to the existing DA literature. If you have further questions, we are more than happy to discuss them!
[1] Adrychowicz et. al. “Hindsight Experience Replay.” NeurIPS 2017
[2] Li et. al. “Generalized hindsight for reinforcement learning.” NeurIPS 2020
[3] Raileanu et al. “Automatic data augmentation for generalization in deep reinforcement learning.” arXiv:2006.12862, 2020.
[4] Ko, Byungchan, and Jungseul Ok. "Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning." NeurIPS, 2022.
[5] Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020.
[6] Adrychowicz et. al. “Hindsight Experience Replay.” NeurIPS 2017.
[7] Fawzi et. al. “Discovering faster matrix multiplication algorithms with reinforcement learning.” Nature 2022.
[8] Abdolhosseini et. al. “On Learning Symmetric Locomotion.” ACM SIGGRAPH 2019.