Offline reinforcement learning (RL) aims to learn optimal policies from offline datasets, where the parameterization of policies is crucial but often overlooked. Recently, Diffsuion-QL significantly boosts the performance of offline RL by representing a policy with a diffusion model, whose success relies on a parametrized Markov Chain with hundreds of steps for sampling. However, Diffusion-QL suffers from two critical limitations. 1) It is computationally inefficient to forward and backward through the whole Markov chain during training. 2) It is incompatible with maximum likelihood-based RL algorithms (e.g., policy gradient methods) as the likelihood of diffusion models is intractable. Therefore, we propose efficient diffusion policy (EDP) to overcome these two challenges. EDP approximately constructs actions from corrupted ones at training to avoid running the sampling chain. We conduct extensive experiments on the D4RL benchmark. The results show that EDP can reduce the diffusion policy training time from 5 days to 5 hours on gym-locomotion tasks. Moreover, we show that EDP is compatible with various offline RL algorithms (TD3, CRR, and IQL) and achieves new state-of-the-art on D4RL by large margins over previous methods. Our code is available at https://github.com/sail-sg/edp.
Paper
Similar papers
Peer review
Summary
This paper focuses on the improvement of computation efficiency of Diffusion-QL by adopting the property of marginal distribution in the diffusion model and the variance control scheme proposed by DPM-Solver. Besides, this paper extends the scope of compatibility with other offline RL methods, from value-based to policy gradient methods. This is achieved by directly approximating the clean examples from corrupted examples at arbitrary diffusion time steps.
Strengths
- This paper is well-written, and the proposed method is easy to understand. - The evaluation of value at risk seems interesting. - The technique seems sound, and the results seem strong.
Weaknesses
- The discussion of limitation seems to be neglected. - More clear discussion is needed to highlight the contribution of this paper and show the novelty. - The proposed algorithm seems to significantly increase the inference computation complexity by introducing 10000 iterations for each action. - More experiments are needed to support the contributions sufficiently. - Some typos are needed to fix through further proofreading.
Questions
See my detailed comments below. ==Major concerns== - In lines 233-235 of Section 4.5: according to Equation (9), the action approximation is the mean value of action Gaussian distribution, so why adopting $\hat{a}^0$ can not reduce high variance? - In Section 4.4, the improvement of sample efficiency (from $ K=1000$ to $K\approx 15$) comes from DPM-Solver rather than the action approximation. Firstly, if the author adopts action approximation, whether the $a_K$ comes from normal Gaussian distribution? If so, the noise prediction function $\epsilon_theta (a^K,K;s)$ should predict the action based on the state and the Gaussian noise $a^K$. Intuitively, generating proper actions based on states seems more difficult, i.e., that classical policy function. - As for the log-likelihood, why can we not obtain the exact log-likelihood of $\pi_{\theta}(a|s)$ through $log~p(a^K)+\sum_{k=1}^{K}log~p_{\theta}(a^{k-1}| a^{k}, s)$? - Does the action approximation improve the computation efficiency of diffusion policy? - I hope the authors explain how they use the action approximation during training. - During the sampling stage, the DPM-Solver speeds up generation rather than action approximation. If action approximation just works on Equation (10), I think it is very similar to the “Target Policy Smoothing Regularization” technique in Section 5.3 of the TD3 paper. I want to know why the authors use the predicted $\hat{a}^0$ rather than $a^0+\epsilon$. ==Minor concerns== - How does the author avoid out-of-distribution actions' effects on the Q function? - DPM-Solver directly makes me understand the sample efficiency of EDP, but I can not understand why the training efficiency can also be improved? So I hope more description of the reason. - In line 318-320 of Section 5.1: during training the diffusion model, we sample a step $t\sim [1, K]$, then train the diffusion model according to Equation (5). If the $K$ is large, we can still train the diffusion model. So I hope the authors explain this claim more clearly. ==Typos== - Line 188: Eqn. 7 -> Eqn. (7). - Line 176: Eqn. 1 -> Eqn. (1). - Line 201: Eqn. 11 -> Eqn. (11). - Line 292: Eqn. 4 -> Eqn. (4). - Line 518 of Appendix: Eqn. 5 -> Eqn. (5). - Line 519 of Appendix: Eqn. 9 -> Eqn. (9). Similar typos can also be found in many places in Appendix. - Line 565 of Appendix: Tab. ??. - Line 575 of Appendix: Tab. ??.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors do not discuss the limitations and discuss broader impacts.
Summary
The authors propose a method to efficiently train diffusion based policies in the offline-RL setting. The authors suggest three main tricks to enable this: 1) Removing the need to backpropagate through the diffusion sampling chain to update the policy by using what the authors call action approximation; 2) Replacement of intractable policy log likelihoods in reinforcement learning objectives with the ELBO, which allows using diffusion policies generally with many popular offline RL algorithms; 3) Using a fast DPM-Solver for action sampling during policy execution. Altogether, these tricks allow for relatively fast training of expressive multimodal policies which achieve state of the art results across the D4RL benchmark suite.
Strengths
* The paper is generally well written and easy to follow. * The use of action approximation facilitates training diffusion policy models with much larger diffusion noising time K, with the authors using K=1000. The RL community has recently taken great interest in the prospect of using the expressiveness of diffusion models for policies, and this simple trick seems to make diffusion policy training practical without any noticeable drawbacks. * The authors also demonstrate that we can use diffusion policies with other common offline RL algorithms such as TD3+BC, CRR and IQL by simply replacing the log likelihood with the diffusion ELBO. This is a valuable contribution to the field, as it could open the door to use diffusion policies to work with many general RL algorithms, a strict improvement over simple gaussian policies. * The authors show strong empirical performance across the D4RL benchmark suite. EDP policies are especially strong compared to the FF counterparts on the more challenging antmaze and kitchen suite, which involve high multimodality due to undirected demonstrations.
Weaknesses
* The main proposal to efficiently train diffusion policies hinges on action approximation, which uses a much higher variance estimate of $\hat{a}^0$. There are no ablations to show how this affects the final diffusion policy as compared to one trained without action approximation. There is a comparison against Diffusion-QL in the paper, but with a very large difference in diffusion timesteps K, so the effect of action approximation itself is difficult to gauge. I have listed some questions related to this point below, which if clarified would be helpful. * The use of DPM-Solver is highlighted as a major contribution is speeding up the sampling process compared to DDPM sampling in Diffusion-QL. It is a well known fact that faster diffusion sampling methods than DDPM exist which can be used for sampling, and Diffusion-QL could have used this as well. While it is useful that the authors show the time savings for using DPM-Solver over DDPM sampling, this is not a novel contribution. * There isn’t discussion about important hyperparameters related to the method. A very important hyperparameter that could be discussed more the diffusion timesteps K (I have noted this as a question below as well). Are EDPs more brittle to train than their FF counterparts in algorithms like IQL with regards to changing hyperparameters?
Questions
* Generally, diffusion models generate higher quality samples with larger diffusion timesteps K, and the authors note in Tab. (1) that training EDP with K=1000 matched or beat Diffusion-QL which was trained with much smaller K~5 to 100. The authors cite this ability to train diffusion models with higher K without having to backprop through the chain as one of the main advantages of EDP. I am curious how much of an impact K actually has in these offline RL benchmark tasks, given that Diffusion-QL has similar scores. I am interested to see an ablation study (at least in some environments) where EDP is trained with different values of K starting at K=1, to see if generally increasing K actually significantly improves policies in these domains. * Energy-based Action Selection (EAS) seems necessary to come close to the performance of DQL using the proposed EDP policies. The appendix shows that other ways to sample actions from the diffusion model result in much worse performance. DQL does not to my knowledge require EAS to produce good action samples from its diffusion policy. Is this due to the effect of action approximation resulting in training a worse diffusion model? * As an extension to the above point, have the authors considered using sampling guidance techniques like Classifer-Free Guidance (CFG) to reduce variance of the final policy? * The authors allow for training diffusion policies with popular offline RL algorithms which require policy log likelihoods by replacing it with the ELBO. Could this be done with lower bounds of simpler deep generative models, such as the ELBO of a Variational Autoencoder as well? I am not aware of prior work that has done this. It could be that diffusion policies are overkill for the expressiveness needed to learn high value policies in the tasks under consideration, in which case these same tricks could be applied to train simpler VAE policies for example.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
The authors have not addressed technical limitations of their work. They have briefly addressed societal impact of better reinforcement learning methods in the appendix.
Summary
The paper proposes to learn a policy for several offline RL tasks in the D4RL benchmark by parameterizing a policy with a diffusion model. The authors claim that their approach is computationally efficient and more compatible with several other RL approaches like maximum likelihood based approaches when compared to Diffusion-QL which is the primary baseline in their experimental studies.
Strengths
1) The paper is well written and easy to read 2) The authors give a clear and concise preface to the method and results in the introduction part of the paper which sets the flow for the rest of the paper. 3) The authors compare their approach with the Diffusion-QL method which introduced the idea of formulating policies as a diffusion model. They explain the issues with the Diffusion-QL approach and try to overcome those shortcomings through EDP 4) Their approach is compatible with max-likelihood based approaches such as IQL (which is one of the SoTA in Offline RL) 5) The run extensive experiments and ablations on the D4RL benchmark and the results reported are consistently better than Diffusion-QL
Weaknesses
1) Line 206: The authors claim that both policy approximations are empirically similar when calculating the objective though there is no evidence provided for the same. Would be good to know their intuition on that as well. 2) There is no discussion around why EDP + IQL works significantly better in some tasks like kitchen, adroit whereas it under-performs in the locomotion tasks (where EDP + TD3) performs better. Would be good to know why one works better than another in these cases. Is there any investigation done in that direction? 3) Barely no discussion around limitations of the approach. 4) Would the approach be robust to data from multi-modalities? There is a mention of why that would be an issue with gaussian policies but there is no details on that in the latter parts of the paper. 5) Would the method work on more complex offline-RL task like CARLA? (why was that excluded from the investigation? I understand that diffusion-QL does not test their method on this task but was it ever tried out?)
Questions
Please refer to some of the questions mentioned in the weakness section; Minor comments: 1) Is the speed only reason for using the DPM-Solver? Are there other alternatives that were considered 2) Please rephrase line 274 (Do not like phrasing the evaluation protocol of another peer-reviewed paper to be labelled as cheating)
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
I do not see any discussion around limitations of the approach or other assumptions made.
Summary
The focus of this paper is to enhance the diffusion policies introduced in Diffusion-QL for offline reinforcement learning. The authors address the challenges of training and sampling efficiency by incorporating action approximation and employing an advanced ODE solver for diffusion policies. They conducted extensive experiments to demonstrate the effectiveness of their proposed approach, known as Efficient Diffusion Policy (EDP).
Strengths
1. This paper is well structured and easy to follow. 2. The conducted experiments are of enough quantity and quality.
Weaknesses
> I think some claims need to be more careful or accurate. “… reduce the diffusion policy training time from 5 days …” “However, it takes tens to hundreds more time to train a diffusion policy than a diagonal Gaussian one” I tried Diffusion-QL before and I don’t think it takes about 5 days for training in their default setting (K=5). In my case, it is only about 20 hours, so I am not sure what setting is inferred here to be 5 days. Moreover, “In comparison, our training scheme only passes through the network once an iteration, no matter how big K is.” This statement is in correct. According to Algorithm 1 and the description in the paper, “To improve the efficiency of policy evaluation, we propose to replace the DDPM sampling in Eqn. (4) with DPM-Solver”, the policy evaluation part still needs the action samples with the full reverse process, which takes a number of steps even with DPM-Solver. > The novelty is kind of limited. In short words, EDP tried two things: 1. replace the real action samples generated by the full reverse process in policy improvement step with one-time real action prediction. 2. replace the original DDPM solver with DPM-solver. > The Energy-based Action Selection (EAS) is not a new thing and has been studied by other works, such as “Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling (ICLR 2023)”. > Following diffusion model conventions, I use $x$ to represent $a$ here. This paper uses $\hat{x_{0, t}}$ to replace $x_0$ in policy improvement step. Note using $\hat{x_{0, t}}$ to replace $x_0$ is not a common thing in diffusion models, since the $\hat{x_{0, t}}$ has different meaning compared to $x_0$. $\hat{x_{0, t}}$ usually represents the mean of all $\hat{x_0}$ that could goes back to the true $x_0$ distribution from time step $t$. Hence, $\hat{x_{0, t}}$ could be with much noise and averaged when $t$ is large. Not sure why here the $\hat{x_{0, t}}$ could replace the $x_0$ in Eq (7). If so, why this procedure could not be applied onto Eq (6) as well?
Questions
See weakness.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes
Summary
This paper proposed EDP to address the existing limitation of diffusion policy in offline RL. EDP relies on an action approximation to construct actions from corrupted ones, thus avoiding running the Markov chain for action sampling at training. shows that EDP achieves 25+ speedup over Diffusion-QL at training time on the gym-locomotion tasks in D4RL. Extensive experiments by training EDP with various offline RL algorithms, including TD3, CRR, and IQL justify the superiority of the proposed method.
Strengths
* The proposed method EDP focuses on improving the Diffusion-QL baseline from two aspects including computational cost and limited applicability to different RL algorithms, which is the major contribution of this paper. It provides a new way to apply a complicated generative model in RL. * Extensive evaluation of datasets of different domains is performed. EDP can outperform several widely-used algorithms by replacing the originally-used Gaussian policy with the refined diffusion policy. And both the training and sampling speed of EDP are way faster than that of Diffusion-QL. Besides, it seems EDP is easier to tune compared with Diffusion-QL as it uses the same hyperparameters for tasks belonging to the same domain and could achieve satisfactory performance. * The unreasonableness of evaluation protocol OMS is pointed out and RAT is adopted to justify the performance of the proposed method and baselines.
Weaknesses
* The compared baselines in Table 2 and Figure 1 are a little bit weak, which cannot demonstrate the superiority of EDP. As the authors claimed that the proposed method outperforms pre-sota methods, more powerful model-free baselines should be included such as EDAC[1], RORL[2] on Locomotion tasks, X-QL[3], InAC[4], BPPO[5] on other domains. Otherwise, I believed the superiority of the proposed method is overclaimed to some extent. [1] An, Gaon, et al. "Uncertainty-based offline reinforcement learning with diversified q-ensemble." *Advances in neural information processing systems* 34 (2021): 7436-7447. [2] Yang, Rui, et al. "Rorl: Robust offline reinforcement learning via conservative smoothing." *Advances in Neural Information Processing Systems* 35 (2022): 23851-23866. [3] Garg, Divyansh, et al. "Extreme Q-Learning: MaxEnt RL without Entropy." *The Eleventh International Conference on Learning Representations*. 2022. [4] Xiao, Chenjun, et al. "The In-Sample Softmax for Offline Reinforcement Learning." *The Eleventh International Conference on Learning Representations*. 2022. [5] Zhuang, Zifeng, et al. "Behavior Proximal Policy Optimization." *The Eleventh International Conference on Learning Representations*. 2022.
Questions
* It seems there exist mismatch of results between pre-sota methods from Figure 1 and Table 2? * Line 227 points out that Tab 1 reflects the benefits of increasing K. However, it seems Tab 1 doesn't include this information? * The GPU memory needed by Diffusion-QL is quite large. I'm curious whether EDP has the same problem?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
N/A
Reviewering Input Needed
Hello Reviewers, The authors have made efforts to address your comments on their work via the rebuttal. Part of the NeurIPS review process is participating meaningfully in the rebuttal phase to help ensure quality. Please read and respond to the author's comments today, latest tomorrow, to give everyone time to respond and reach proper conclusions. Thank you all again for your assistance in making NeurIPS a great conference for our community.
Reviewer Response Needed
Hello Reviewer, The authors have made efforts to address your comments on their work via the rebuttal. Part of the NeurIPS review process is participating meaningfully in the rebuttal phase to help ensure quality. Please read and respond to the author's comments today, latest tomorrow, to give everyone time to respond and reach proper conclusions. Thank you for your assistance in making NeurIPS a great conference for our community. -- Your AC
I appreciate the authors' responses that clarified my questions. After reviewing their explanations, this paper possesses the necessary qualities for acceptance.
Thank you so much for your positive feedback and for acknowledging the merit of our work. We truly appreciate the time and effort you have dedicated to comprehensively reviewing our manuscript. For all the typos raised by you, we will definitely fix all of them the future version. We would be extremely grateful if you could kindly update your rating.
Thank you for the detailed response to my questions. I will maintain my score recommending acceptance of the paper.
Thank you for your feedback
We appreciate your detailed review and suggestions. We will incorporate the comments and involve new results into the final revision. Thank you again!
I'm disappointed by the author's feedback. - The Diffusion-QL paper repeatedly states its use of K=5 in experiments, e.g., “We found N = 5 performs well on D4RL (Fu et al., 2020) datasets, which is also a small enough value for cost-effective training and deployment. “, and “In the following D4RL tasks, we set a moderate value, N = 5, to balance the performance and computational cost.”. Additionally, the official Twitter-associated Diffusion-QL repository doesn't label itself as "official." You specifically mentioned Twitter-associated repo so you should know they provided another official repo. It's misleading to overstate improvements by selecting hyperparameters for baseline methods. Claims in this paper, especially on efficiency, should be factual. - The response does not make sense to me. In Algorithm 1, there is one section saying, “Sample next actions with DPM-Solver”, the actions samples here need a full reverse process. The statement “our training scheme only passes through the network once an iteration, no matter how big K is” is incorrect. - The authors admit their proposed methods are simple. Actually, there is nothing specific new in the design of diffusion policy. The authors mainly proposed to use the $\hat{a_{0,t}}$ to replace the actual action samples. The likelihood-based extension is not real likelihood estimation. - You know other works have studied EAS, but I didn’t see any references in Section 4.5. - The intuitive explanation does not convince me. You suggest that policy improvement does not need precise action samples. The observation is intriguing but seems counterfactual. In light of these points, my concerns remain. While I recognize the empirical performance, it's vital for claims to be accurate. I'm eager to hear further from the authors.
Official Comment by Authors [1/2]
Dear Reviewer TjdE: Thank you very much for your effort at reviewing our paper and valuable comments. We now address your concerns as below. > R2-Q1. You specifically mentioned Twitter-associated repo so you should know they provided another official repo. It's misleading to overstate improvements by selecting hyperparameters for baseline methods. We’d like to clarify that we originally intended to paste an url pointing to the code snippet here. However, as external links are strictly prohibited, we break down the url into indicators like organization, repo and file. When we started this project, the only available codebase was under the Twitter organization (the other one was not yet publicly available). We apologize that we did not notice the newly released one has different default parameters. In addition, We’d like to clarify that in our experiments, we always emphasize **the configuration of our reimplemented baseline methods** and our improvement over the baseline. Moreover, we’d like to emphasize that action approximation brings the following intriguing benefits, which should not be neglected. Enabling training with large diffusion steps (1000 steps in EDP versus 5 steps in Diffusion-QL). Large diffusion steps give significant performance boost on kitchen, adroit and antmaze. Large diffusion steps allow us to use one set of hyperparameters for environments from the same domain. We will add further clarifications about the default configurations of Diffusion-QL and the main benefits of using action approximation in our revisions to avoid confusions. > R2-Q2. Incorrect statement about “our training scheme only passes through the network once an iteration, no matter how big K is”. In terms of this statement, we respectfully disagree. We refer the reviewer to section 4.4, line 217 to 227, where a detailed explanation on the efficiency is provided. A typical actor-critic RL algorithm alternates between policy evaluation and policy improvement, both of which affect the training efficiency. Specifically, *policy evaluation* is used to estimate the state or state-action values for the current policy, *i.e.*, Eqn (6). *Policy improvement* instead focusing on improving the current policy based on the value estimation, as shown in Eqn (7). In section 4.4, we specifically mention that EDP can only reduce the time steps to 15 for policy evaluation (line 218-229). But for policy improvement, EDP does not need backpropagation through the whole sampling chain and only needs to forward and backward the network once per iteration. Please note that our claim is for policy improvement, as explained in line 220-226. > R2-Q3. “Actually, there is nothing specific new in the design of diffusion policy.” “The likelihood-based extension is not real likelihood estimation.” We’d like to emphasize that our training strategy enables training diffusion policies with much larger steps, *e.g.*, 1000, which bring clear performance gains on adroit and kitchen environments. Moreover, we are not aiming at estimating the likelihood. Our target is making diffusion policies compatible with likelihood-based RL methods, so that we can approximately maximize the policy likelihood when a diffusion policy is trained with such algorithms. As a result, EDP works nicely with IQL and CRR, which further boosts the performance on antmaze, adroit and kitchen. We argue that the importance of our contributions should not be disregarded. > R2-Q4. “You know other works have studied EAS, but I didn’t see any references in Section 4.5.” We thank the reviewer for bringing up this discussion again. We apologize for the confusion caused. Nevertheless, we respectfully disagree with the reviewer, and would like to further clarify on this point. The Energy-based Action Selection (EAS) involves two distinct steps: 1) sampling $K$ actions from $\pi_\theta(a\mid s)$ and 2) selecting the action with the highest $e^{Q(s, a)}$. In fact, the exact form of EAS has been less discussed in the literature and people use it for different purposes. In our case, we use it for reducing the variance of a diffusion policy. Mathematically, such a sampling scheme can be treated as sampling from a non-parametric distribution $p’(a\mid s)\propto e^{Q(s, a)}\pi_\theta(a\mid s)$. Here, we refer to MPO **only to provide intuitive explanations to why EAS improves the final performance**, because $p’(a\mid s)$ can be considered as a one-step improved policy given **current** policy $\pi_\theta(a\mid s)$ and an **estimated** Q values $Q(s, a)$. We **do not suggest that MPO uses EAS for policy improvement**. Specifically, MPO performs policy improvement by minimizing the difference between the current learning policy $\pi_\theta(a\mid s)$ and the non-parametric one-step improved policy $e^{Q(s, a)}\pi(a\mid s)$. Such a scheme is completely different from the EAS. We will further clarify this and improve the presentation in our revisions.
Official Comment by Authors [2/2]
> R2-Q5. “The intuitive explanation does not convince me. You suggest that policy improvement does not need precise action samples. The observation is intriguing but seems counterfactual.” We’d like to emphasize that our intuitive explanations in A5 of the rebuttal are actually two-fold. We first explain why action approximation can not be used for policy evaluation, then explain why it can be used for policy improvement. We did not claim “policy improvement does not need precise action samples”, our point is: compared to policy evaluation, “Though \hat{x}_{0,t} is not the precise action, it can still provide valuable guidance for policy optimization.” There is some empirical evidence for this from the field of learning from demonstration[R3-R5]. These papers show that when some imperfect demonstration data are added for RL optimization, the performance can be greatly boosted. Moreover, we also draw a connection between our method and classifier guidance, which we believe is able to better understand why action approximation works. In the end, we do admit that this intuitive explanation is based on our understanding of RL algorithms, learning from demonstration, offline RL and diffusion models. We do not have theoretical verification for it, but only empirical support that action approximation does not hurt performance much, and it can enable training diffusion models with much larger diffusion steps. [R3] Gao, Yang, et al. "Reinforcement learning from imperfect demonstrations." arXiv preprint arXiv:1802.05313 (2018). [R4] Jing, Mingxuan, et al. "Reinforcement learning from imperfect demonstrations under soft expert guidance." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020. [R5] Kang, Bingyi, Zequn Jie, and Jiashi Feng. "Policy optimization with demonstrations." International conference on machine learning. PMLR, 2018. If you have further questions, please do not hesitate to let us know. Authors of Paper 6867
Thank you for your continued discussion. While many of my initial concerns have been addressed, I believe the paper could benefit from further clarifications to ensure the accuracy of the claims presented. I've accordingly adjusted my score. - I noticed no issues with the Github repositories. However, I'd like to highlight that Diffusion-QL explicitly mentioned using K=5 for all their experiments. When your paper reports a 5-day running time of Diffusion-QL based on K=100 and then contrasts it for efficiency, it may come off as an unfair comparison. It would be beneficial to add clarity to this point. - For enhanced clarity, it might be useful to specify "in policy improvement" within the relevant sentences. - If you're considering EAS as one of the significant contributions of your work, it's essential to provide a comparison with other works and what something new you have contributed. - It's worth noting that the use of action approximation and its effectiveness is an interesting observation in your paper.
Dear Reviewer TjdE, Thank you very much for your quick reply and constructive comments! We will definitely enhance the clarity of the paper accordingly following your suggestions. Thank you again! Authors of Submission6867
Thank you for the answers to my questions/concerns. I believe most of my questions were answered comprehensively. I hope you make the necessary changes as you have mentioned in the above rebuttal. I will maintain my score to acceptance with these changes being included.
Thank you for your feedback
We appreciate your detailed comments and suggestions. We will polish our paper further and incorporate new changes into the final revision. Thank you again!
Decision
Accept (poster)