Summary
The paper studies the use of diffusion models in offline multi-task reinforcement learning for planning and synthetic data generation. Both approaches use prompting to encode task-specific conditions for the generative model along with a transformer backbone. In the multitask setting, the approach outperforms prior SOTA baselines on MT50. In the synthetic data setting, the approach improves downstream TD3+BC performance.
Strengths
- Clear and well-written presentation of the method
- A thorough set of baselines and strong evaluation on the challenging MT50 benchmark for MTDIFF-p
- Elegant method of removing the requirement of one-hot task encoding by transition prompting
- Strong results showing positive multi-task transfer in both planning and synthetic data generation, the algorithm neatly extends single-task equivalents in both areas
Weaknesses
- Only custom settings with large amounts of data are considered, it would be useful to understand what the minimum data needed to be effective is. Additionally, it would be useful to evaluate on pre-existing offline datasets for a more representative comparison.
- In Figure 6, it would be useful to compare MTDIFF-s-single with the existing single task baseline, Synthetic Experience Replay. [1]
- More thorough analysis of data quality would be valuable rather than just downstream performance, e.g. error of the transitions. Furthermore, downstream RL performance is only computed with one RL algorithm.
- The two parts of the paper - multitask planning and synthetic data generation - are not necessarily connected. For example, the synthetic data (MTDIFF-s) is used for an entirely different RL algorithm, TD3+BC, and does not contribute to the performance of the planning algorithm (MTDIFF-p).
Minor:
- Line 242: Typo ‘fine-grind’
- Line 248: the author’s definition of ‘near-optimal’ is the same as D4RL ‘full-replay’ and ‘sub-optimal’ is the same as D4RL ‘medium-replay’. These descriptions may be more clear to offline RL practitioners
- Figure 4: choosing one sampled trajectory for each approach is likely cherry-picking, I would suggest rendering N samples
- Figure 5: missing standard deviation
- Figure 6: missing the baseline without synthetic data
- Line 15: Unclear what ‘high-quality’ and ‘low-quality’ mean as it seems MTDIFF-s models the original distribution
[1] Synthetic Experience Replay. Cong Lu, Philip J. Ball, Yee Whye Teh, Jack Parker-Holder.
Questions
- How would the prompting perform if the tasks were not readily discernible from the initial transitions? E.g. multi-task settings with sparse task-specific reward
- What is the minimum data required to make the algorithm work?
- What is the speed of planning and sampling transitions with the transformer model? How does this compare to related methods?
- How does MTDIFF-s-single compare to Synthetic Experience Replay? I.e. what is the benefit of modeling full trajectories instead of transitions?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
Discussion of limitations hidden in Appendix, should be moved to the main paper.