Response from authors
> The discussion and analysis of chosen data augmentation techniques in Section 3.2 lacks sufficient depth
We thank the reviewer for recognizing the innovation in our approach, its computational efficiency, and the value of our JS divergence analysis. However, we respectfully disagree with the reviewer’s concern regarding the lack of sufficient depth in the data augmentation. While we greatly value the contributions of methods like DrAC and SVEA, these methods involve algorithmic changes specific to online RL, **which differ from our objective of providing a simple, algorithm-agnostic solution for offline RL with visual observations.** Our focus is on achieving non-algorithmic changes to offline RL using simple, effective visual augmentations, as demonstrated by RAD [1], with four specific techniques detailed in Supplementary Section B.0.3. These augmentations, combined with diffusion-based upsampling, are central to our goal of enhancing data diversity and generalization without altering the underlying RL algorithm. **To address this further, we expanded the “Related Work” section to clarify these distinctions in more detail and included the missing papers from the three suggested by the reviewer.**
_[1] Laskin et al. Reinforcement Learning with Augmented Data. NeurIPS 2020_
> The proposed Generalization Performance metric $G_{perf} = \frac{T_{test} - B_{test}}{B_{train} - B_{test}}$
needs better justification.
Regarding the generalization metric, its design is grounded in established principles commonly used in reinforcement learning (RL) generalization studies. **See the original Procgen paper [1] Section 2.2. They use the following $R_{norm} = \frac{R - R_{min}}{R_{max} - R_{min}}$.** The idea being to normalize w.r.t. what is possible given the setup. Here $Rmin$ is the lowest possible score, i.e., the test performance of the baseline (with poor generalization), and $Rmax$ is the highest possible score, the train performance on the baseline training data, which is the least diverse and easiest to overfit. **We added this discussion in the paper (section 4.3.1) as we agree it appears plucked from thin air in the current draft, so thank you for flagging.**
_[1] Cobbe et al. Leveraging Procedural Generation to Benchmark Reinforcement Learning. ICML 2020_
> The experimental results reveal a critical misalignment with the paper's claimed contribution to "Zero-Shot Visual Generalization.
Thank you for highlighting this critical aspect of our work on zero-shot generalization. We respectfully disagree with the reviewer’s assertion that the paper does not demonstrate improved zero-shot generalization, as we show this in Procgen (see aggregate performance added to Table 3). Additionally, we present the FDD approach (Table 2), where we observe improvement in the generalization gap for the DMC environments. That said, we understand that the improved performance in the original environment in Table 1 (not necessarily a bad thing!) could lead to confusion. **We are happy to rephrase the title if you have a recommendation.** One proposal could be “Synthetic Data Enables Training Robust Agents from Offline Data,” as our agents perform well across a wide range of settings. **We also updated Tables 2 and 3 to include $Test/Train$ and $Train-Test$ results for both environments, aligning with the metrics suggested by the reviewer.** Please let us know if this makes more sense now.
> insufficient exploration of diffusion model design choices
Thank you for highlighting the need for deeper exploration of diffusion model design choices. **Our diffusion model builds directly on SynthER’s design, which allows us to make consistent comparisons with the V-D4RL benchmark and ensures the reproducibility of our results.** SynthER’s supplementary material (Section B) already provides extensive ablations on model architecture and denoiser parameters, and our findings closely align with these results. As such, duplicating those ablations in our work would have added redundancy without offering additional insights. Instead, we focused on demonstrating the effectiveness of our combined augmentation and diffusion-based upsampling method in the context of offline RL. **To provide clarity, we have added references to SynthER’s ablations in the updated Supplementary Section C.3,1. Additionally, we conducted and included a table on latent space size ablations in the updated supplementary material, addressing a gap not explored in SynthER’s ablation work (Section C and Section F).** While this analysis was initially omitted for brevity, we now provide it to offer further insights into the relationship between latent space dimensionality and performance, complementing SynthER's findings.