OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning

Summary

This paper discusses the safe dataset mismatch (SDM) problem, highlighting how low-reward or unsafe samples in datasets can harm offline safe RL. Conditional distribution shaping (OASIS) is proposed to mitigate this problem by generating high-reward and safe samples via diffusion models and promoting general offline safe RL algorithms with the generated data. This paper evaluates OASIS through extensive experiments across various safe RL tasks and different types of datasets with varying data distributions.

Strengths

- The paper is well motivated. The influence of imbalance and biased data on offline safe RL is an important but underexplored problem. Solving the proposed problem through diffusion-based data generation is intuitive, reasonable and novel. - The empirical evaluation and ablation studies are comprehensive, carefully demonstrating the significance of the SDM problem and the effectiveness of the proposed approach. - Theoretical analysis provides certain guarantees for the proposed approach. - The paper is well written and well organized.

Weaknesses

- Some technical and experimental details are a bit confusing. See Questions 1, 2, 3, 4. - Assumptions 1 and 2 seem kind of idealized to directly bound the distribution and policy discrepancy, since $\epsilon_{score}$ and $\epsilon_{inv}$ cannot be directly calculated or estimated. Admittedly, analyzing the distribution and policy discrepancy based on diffusion-generated data may be difficult and beyond the scope of this work. Maybe more discussion and explanation could help justify these assumptions. - Directly excluding mismatched data from training datasets could be an another intuitive approach for SDM problem. So it would be better to discuss the performance if tempting and conservative data were directly excluded from the full datasets, utilizing the remaining datasets (i.e., Full dataset - Hybrid dataset) for training. A comparison between these two approaches (i.e., adding matched data vs. reducing mismatched data) would be valuable.

Questions

1. What is the size of the generated dataset $D_g$? I would like to confirm whether you directly train the policy on $D_g$ or a mixture of $D_g$ and the original datasets? 2. How do you calculate the reward condition $\hat R$ based on the given cost $\hat C$? 3. Why are the thresholds in Figure 5 different for different types of datasets? Did you use the same threshold when constructing different types of datasets? This is a concern because one tempting dataset constructed under the thresholds 20 may be considered tempting for one threshold 20 but not for another threshold 60. 4. In Figure 6, how did you obtain the performance of baseline methods (e.g., CDT) under a specific $\alpha$? Did you use the data of size $\alpha$ generated by OASIS for these baselines? It would be helpful to provide more details about the data efficiency experiments.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation are discussed in the paper.

Summary

Offline Safe Reinforcement Learning (RL) is used to learn policies satisfying cost constraints from a given dataset. This proves to be a challenge when the dataset is biased in a certain way. This paper introduces a method to use the offline training dataset to capture the environment using a diffusion model that can generate data conditioned on our cost constraints and performance objectives. Experimental results show the approach generates data that better fit the constraints and outperform alternatives like dataset reweighting.

Strengths

- The paper is well organized and clearly shows the strengths of diffusion models to generate offline data with the given cost/performance objectives. - Theoretical results show that the policies learned from the generated data are cost constrained given some reasonable assumptions on the model. - Extensive comparisons are made to different SoTA baselines in offline RL with and without data generation.

Weaknesses

- Data generation and training can be a slow process due to the use of diffusion models which are known to be computationally heavy. - Selecting hyperparameters such as number of denoising steps might vary results significantly. For example, Table 2 has greatly varying policy costs for different values of $K$ (albeit similar performances). - While results are mostly consistent, it is hard to say when the proposed method prefers to act more conservatively with lower cost (or more riskily i.e., higher reward). This is reflected in the reward and costs in Table 1 (e.g., CarRun). A study on a hyperparameter change (apart from cost/return targets) to control this balance would be helpful.

Questions

1. How are the initial states decided to generate the data from the diffusion model? Are they the same as the initial states of the training dataset or randomly sampled from the training dataset trajectories? 2. How long is inference time for the OASIS model i.e., data generation time? Is that included in the training time (L999, App C.3)? 3. How do we decide the “target” cost given the cost threshold to get the best performance? Is the study done in Fig. 7 required for each setting or is there a good heuristic to set these targets? How expensive is this study (Fig. 7) over target values (i.e., multiple inference runs)? On a related note, how were these targets set for Table 1 on OASIS and the compared baselines? 4. Is it right to say the primary reason for the success of the proposed method over the diffusion baselines (like FISOR), a more realistic conditional generative model that yields optimal generated data satisfying our cost constraints? 5. Are the learned labeling models (inverse dynamics, reward, and cost) only used for labeling the data generated from the diffusion model? 6. In Fig. 6, are all models using data generated by the same diffusion model? OASIS handles the diffusion model training and data generation with BCQ-Lag as the actual Offline RL policy learning. This makes me a little confused. How are the curves related in Fig. 6? 7. Why is hybrid an interesting dataset setting vs. full? 8. Typos: - L224 (Sec 4.4) Theoretical - L973 (App C.1) min instead of max

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Diffusion models being computationally heavy (see Weakness)

Summary

This paper proposes OASIS, which uses a conditional diffusion model to reshape the dataset distribution and achieve effective offline safe RL learning. Theoretical analysis gives the error upper bound of distribution reshaping and constraint violation upper bound. A large number of experiments show that the proposed algorithm has significantly improved compared to offline safe RL baselines.

Strengths

- The logic is clear and the paper is easy to understand. - The theoretical analysis is sufficient. - A large number of experiments prove the effectiveness of the proposed algorithm.

Weaknesses

***W1:*** Some baselines are not introduced in related work, such as CVAE, FISOR, etc. This may cause difficulties in understanding. ***W2:*** Typos, such as line 173 "reweighing" -> "reweighting"; wrong citations, such as line 290 "COptiDICE [17]" -> "COptiDICE [16]"

Questions

***Q1:*** Compared with CVAE, OASIS shows that the conditional diffusion model has a better ability to generate according to the condition information, as shown in Figure 7(c), but this example is a bit simple. Can the author show more comparisons of the two similar to Figure 7(c)? For example, add the OASIS generation results to Figure 8(c)? ***Q2:*** When showing the effectiveness of the newly generated dataset, in addition to showing the distribution of the generated state like Figure 7(c), it is also necessary to verify whether the annotations of the inverse dynamics model and reward & cost models are accurate. Can the author supplement the accuracy of these three models? ***Q3:*** By comparing the results of OASIS and CDT, I came to a conclusion: in offline safe RL, both methods conditioned on cost and reward, generating data is more effective than generating policy. I wonder if the author agrees with this conclusion? This conclusion seems to be uncommon in the field of offline RL, or it may be because I am not familiar with distribution shaping methods.

Rating

6

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

See the weaknesses and questions sections.

Response to Authors

Thank you for your response, which has addressed most of my concerns. However, I still have a few additional questions: --- Supplement to ***Q1***: As I mentioned earlier, the example of Car-Circle seems somewhat simplistic. If time permits, could you please add the generation results of OASIS to Figure 8c? Supplement to ***Q3***: Yes, I agree that the conclusion you provided is more accurate and reasonable. Based on the current discussions, I will at least maintain my score.