Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans
Diffusion-based planning has shown promising results in long-horizon, sparse-reward tasks by training trajectory diffusion models and conditioning the sampled trajectories using auxiliary guidance functions. However, due to their nature as generative models, diffusion models are not guaranteed to generate feasible plans, resulting in failed execution and precluding planners from being useful in safety-critical applications. In this work, we propose a novel approach to refine unreliable plans generated by diffusion models by providing refining guidance to error-prone plans. To this end, we suggest a new metric named restoration gap for evaluating the quality of individual plans generated by the diffusion model. A restoration gap is estimated by a gap predictor which produces restoration gap guidance to refine a diffusion planner. We additionally present an attribution map regularizer to prevent adversarial refining guidance that could be generated from the sub-optimal gap predictor, which enables further refinement of infeasible plans. We demonstrate the effectiveness of our approach on three different benchmarks in offline control settings that require long-horizon planning. We also illustrate that our approach presents explainability by presenting the attribution maps of the gap predictor and highlighting error-prone transitions, allowing for a deeper understanding of the generated plans.
Paper
Similar papers
Peer review
Summary
In this work, authors propose a new metric called “restoration gap” that is able to measure the reliability of plans emerging from diffusion-based planners. They then suggest using this metric along with guidance, “restoration guidance”, to encourage the diffusion-based planner to generate more reliable plans. In addition, they attempt to improve the performance of restoration guidance by using an attribution map regularizer. Finally, they evaluate their method on three different offline control benchmarks.
Strengths
Overall Strengths: 1. The problem of detecting and/or improving unreliable plans generated by diffusion-based planners is both interesting and important and I appreciate the authors for their efforts towards this direction. 2. Paper is well-written and well-presented with good quality visualizations, plots and tables. 3. Experiments are covering a good-enough domain of tasks and compare against a large number of benchmarks.
Weaknesses
Overall Weaknesses: 1. A critical weakness, in my view, is the lack of substantial improvement demonstrated by the proposed solution compared to the Diffuser benchmark. Across various experiments, RGG exhibits only marginal enhancements (if any) over Diffuser, and similarly, RGG+ shows only minor improvements (if any) over RGG. This observation raises two possibilities: either the solution itself lacks significant impact or the current experimental setup fails to adequately capture the method's importance. While I appreciate the authors' overall research trajectory, it is my belief that this particular submission requires further development and refinement. 2. Additionally, there are some additional limitations in the work. Firstly, to establish the significance of their approach, it would be valuable to investigate how it compares to rejection-based methods, especially in scenarios where the problem context remains fixed. For instance, in the context of the Maze2D experiments, the primary objective is to find solutions that navigate around obstacles. One potential approach could involve incorporating collision-detection code and subsequently discarding samples that do not satisfy this condition. 3. Furthermore, to demonstrate the significance of the proposed metric, it would be valuable to show how guidance with restoration gap compares with guidance with realism score and rarity score (or other readily avaible metrics).
Questions
Overall, the paper is well-written and easy to follow. However, there are some parts that need improvement. For me, these include: - In equation 4, the addition of trajectory optimization is vague. Maybe, a reference to section 3.2 of Janner et al. 2022 where trajectory optimality in diffusion models is introduces could be beneficial. - In equation 5, g is defined but is not explained what it is supposed to be and how it is supposed to be used. For someone new to the field, this is problematic. - In figure 2, the realism gap and rarity score are shown alongside restoration gap but they are not explained until section 5.1. I recommend moving this explanation to the caption of figure 2 or to section 3. - In section 3, it is explained why restoration gap is an intuitive measure of plan reliability. It would be better if you could clarify what reliability really means. Are reliable plans ones that are within distribution? Moreover, a discussion on how much noise is required is also essential as too much noise may push the recovered samples too far from the original and too little will not provide any improvements. - Attribution map is a good option for showing the explainability of diffusion-planner with the trained gap predictor. However, its significance as a regularizer is not supported by the experimental results. Therefore, I would remove section 4.2 as it doesn’t provided any added value. - Overall, the enhancements offered by the new method (RGG and RGG+) in comparison to Diffuser do not appear to be substantial, if any. As a result, the overall significance of the work is diminished in my perspective. - Following section 5.1, it would be valuable to show how guidance with restoration gap compares with guidance with realism score and rarity score. This could potentially support the fact that the newly proposed metric is more suitable for the task than other available metrics. - Throughtout the paper, it is claimed that the proposed approach for improving unreliable plans emerging from diffusion-based planners is novel but this claim is not supported. Firstly, the provided solution which is to use guidance based on some form of reliability-measuring metric, is not necessarily novel and it seems to be the immediate and instinctive response to addressing the issue. Moreover, the newly proposed metric which is based on the addition of noise for sample recovery and purification has been explored before as well (see SDEdit by Meng et al. and Diffusion Models for Adversarial Purification by Nie et al.). Therefore, the authors’ claim in the approach being novel has to either be removed or clarified.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
1 poor
Limitations
In addition to the recommendations on the specific technical parts (7 bullet points in “Questions”), I highly recommend developing the idea of purifying diffuser-generated plans further using various techniques and comparing what provides the best performance. Moreover, designing an experimental setup that captures the significance of reliability of plans is highly beneficial to support the significance and impact of the work.
Summary
This paper investigates the potential of diffusion models in addressing long-horizon sparse reward tasks. While diffusion models are effective generative models, they are not inherently guaranteed to generate feasible plans. To overcome this limitation, the study proposes an approach to refine unreliable paths generated by diffusion models and provides guidance to enhance error-prone plans. The evaluation of individual plans generated by diffusion models is done using a novel metric called the "restoration gap" which measures the restorability of a given path. The "restoration gap" metric effectively identifies plans with a low error probability, which is considered an indicator of plan feasibility. This metric is estimated using a function approximator called the "gap predictor." With the help of the gap predictor, a Restoration Gap Guidance (RGG) is defined, which refines diffusion planners and improves the low-quality plans generated by diffusion model. Moreover, the gap predictor can lead the agent in an undesirable direction due to estimation errors during the denoising process. Therefore, RGG is augmented with a regularizer to prevent plans from heading in the wrong direction, and this modified RGG is referred to as RGG+. The proposed approach is evaluated on three different benchmarks. The experimental results demonstrate that the approach enhances planning performance compared to the benchmarks. Overall, the paper presents a novel method for refining unreliable plans generated by diffusion models, improving the planning performance in long-horizon tasks. The approach is demonstrated to be effective in various offline control settings and provides explainability through the use of attribution maps.
Strengths
Novel Approach: The paper proposes a novel approach to refining unreliable plans generated by diffusion models. It introduces the concept of restoration gap as a metric to evaluate the quality of individual plans generated by the diffusion model. Theoretical Justification: The restoration gap metric is theoretically justified, and the paper provides an analysis of its properties. It demonstrates that the restoration gap can effectively detect artifacts (infeasible plans) with bounded error probabilities. Improvement in Planning Performance: The proposed approach, called Restoration Gap Guidance (RGG), is shown to enhance the planning performance of the diffusion model. By utilizing the restoration gap metric and a gap predictor, the RGG process guides the reduction of the restoration gap and improves the quality of the generated plans. Explainability: The paper highlights the explainability aspect of the approach. It presents the attribution maps of the gap predictor, which provide insights into error-prone transitions and enable a deeper understanding of the generated plans. Though, I believe this aspect needs to be evaluated further to prove its usefulness in different settings. Empirical Evaluation: The proposed approach is evaluated on three different benchmarks in offline control settings, including Maze2D, Locomotion, and Block Stacking tasks. The experiments demonstrate the effectiveness of the approach in improving planning performance compared to other baseline methods. Regularization Technique: To mitigate the risk of adversarial artifacts, the paper introduces an attribution map regularization method. This regularization prevents the gap predictor from pushing plans in the wrong direction, resulting in improved refining guidance.
Weaknesses
This could be a weakness or a limitation (or a misunderstanding). If it’s limitation please elaborate on it in the Limitations section. Basically, the proposed approach appears to assume that the diffusion model is well-trained and makes almost no mistakes. In the restoration gap part, the effectiveness of the "restoration gap" metric relies on the assumption that when a new trajectory is provided to the diffusion model to generate new plans, this trajectory data is assumed to have a similar distribution with the dataset which is used to train the diffusion model. In such cases, the restoration gap, based on L2 distance, might be applicable. However, if a new trajectory data which doesn't have a similar distribution is given to the diffusion model, the diffusion model might generate significantly different trajectories compared to the original trajectory. In this case, the restoration gap based on the L2 distance could potentially fail to provide accurate guidance. Can you comment on that? Not weaknesses but just some small remarks are provided below: I’d consider the third point in the contribution list as a result, and thus a verification of the proposed approach, not a contribution. Line 226-227: I wouldn’t say RGG improves the planning performance of Diffuser in 5 out of 6 tasks confidently. 2-3 out 6 seems like a better judgment of these results. Line 229: I think the claim “... our approach of refining Diffuser with RGG either matches or surpasses most of the offline RL baselines. Additionally, it significantly enhances the performance of Diffuser …” is too strong. Please be more precise and fair here. It would also help if you color-code the best- and second-best-achieving scores (surely while considering Diffuser, RGG and RGG+ as one algorithm, e.g., if RGG+ has the best score, do not consider RGG as the second-best if it has the highest score amongst others) in Table 1. That helps the reader put in perspective the performance of diffusion-based models to other RL approaches. Line 235: repetition of a reference - “... and MOPO (Yu et al., 2020)”
Questions
What can you say about the trade-off in terms of additional complexity and thus probably increased learning time, induced by the proposed extension, vs. the performance improvement? I assume the lengths of the original trajectories and those generated by diffusion models are the same, right? If yes, are there additional steps to ensure this, or are they by default equal? Maze environments might sometimes be solved using different trajectories. For instance, in Figure 2 (last row and last column maze), the generated trajectory takes the shortest path. However, it is possible for the diffusion model to take the longest path if the maze environment has been previously solved using longer trajectory paths. If these observations are stored in the dataset and used to train the diffusion model, it may generate longer trajectories but the original trajectory might be the shortest one. Is this a possible situation? If yes, how does this affect the restoration gap metric and would such a case cause it to inaccurately assess the quality of the generated paths? What could be the reason(s) for the relatively smaller improvement on the Multi2D environment compared to a better improvement for the Maze2D scenario (Table 2)? How is the performance score computed for Maze2D, Multi2D environments? E.g., the other score you shared for CQL seems to have been scored between 0 and 100 in the original paper (Fu et al., 2020). Your scores are above 100 though, how is this possible?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors briefly mentioned the limitations of the proposed approach in the appendix section. I believe the main text should include those limitation discussions. In addition, if any of the questions / comments above relates to a limitation, please include them as well.
Summary
The paper proposes a novel metric named restoration gap for evaluating trajectory quality generated by diffusion models. It further proposed restoration gap guidance to refine the diffusion generation. The restoration gap predictor also provides explainability of the model. Experiment results indicate improved performance of the proposed model.
Strengths
The contribution of the paper is solid: The restoration gap gives an intuitive metric for measuring trajectory quality: plans with low quality are hard to restore after perturbation; Following this intuition, the author proposed to train a gap predictor and uses its gradient to guide the generation process. Experiment results give strong support for the claim of the paper. The presentation of the paper is clear. The paper is well-structured with intuitive visualizations.
Weaknesses
To apply the restoration gap guidance, a hyperparameter is introduced to weight the signal from the gap predictor; and then to mitigate the issue from the gap prediction error, an attribution map regularization is added with another hyperparameter weight. It is not clear how sensitive the performance of the model is to these hyperparameters and how hard it is to tune them.
Questions
Can author give some results / explainations on how sensitive the performance of the model is to the additional hyperparameters introduced (beta, lambda in eq. 22) and how hard it is to tune them?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
While the restoration gap is intuitive in the given examples (2D map with walls), it is unclear how the gap works in more complex environments or environments with fewer hard constraints.
Summary
This paper introduces a novel approach to refining unreliable plans generated by diffusion models in long-horizon, sparse-reward tasks. While diffusion-based planning has shown promise, the generative nature of diffusion models can result in infeasible plans, limiting their usefulness in safety-critical applications. The proposed approach utilizes a metric called restoration gap to evaluate plan quality and provides refining guidance to error-prone plans. Additionally, an attribution map regularizer is presented to prevent adversarial refining guidance. The effectiveness of the approach is demonstrated on three benchmarks, and the explainability of the generated plans is highlighted through the attribution maps.
Strengths
1. Detailed and comprehensive experiments: The article provides a thorough evaluation of the proposed approach on three different benchmarks in offline control settings that require long-horizon planning. This comprehensive experimentation strengthens the validity of the findings. 2. Novel idea and solution: The article introduces a new metric, restoration gap, and a corresponding refining guidance approach to address the limitations of diffusion-based planning. This novel approach contributes to the field and expands the understanding of plan refinement in such tasks. 3. Addressing the issue of error-prone plans: The article successfully tackles the challenge of refining unreliable plans generated by diffusion models. By providing refining guidance based on the restoration gap metric, the proposed approach improves the feasibility and quality of the plans.
Weaknesses
1） Accuracy of restoration gap metric: This paper uses the difference between the generated trajectory and the original input trajectory to evaluate plan quality. However, this approach may be inaccurate for small-sample cases where the distribution of the expert dataset leads to rare but valid trajectories being considered abnormal. Further exploration is needed to address this limitation. 2） Lack of experimentation on balancing coefficients: The article does not conduct an ablation study to examine the balance between the opportunity cost function J's guide and the opportunity restore guide. It would be beneficial to investigate how to strike a balance between these coefficients to achieve optimal plan refinement.
Questions
1）How does the proposed approach address the potential issue of rare but valid trajectories being considered abnormal due to the distribution of the expert dataset? Have any measures been taken to mitigate this limitation? 2）Could you provide more insight into how the coefficients for the opportunity cost function J's guide and the opportunity restore guide are balanced in the proposed approach? Has any experimentation been conducted to determine the optimal balance between these coefficients?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
the article should address the issues related to the accuracy of the restoration gap metric in few-shot case and conduct experiments to explore the balance between coefficients in the opportunity cost function and the opportunity restore guide.
Summary
The paper proposes to refine the predictions of a diffusion-based planner by identifying unfeasible plans prior to execution. The newly introduced "restoration gap" measures the reconstruction error for a given sample under some noisy perturbation, and it is argued that this gap is a reasonable proxy for feasibility (i.e. if the restoration gap is high for a sample (plan), then that plan is probably not feasible). The authors show that the proposed metric can identify "artifacts" with bounded error probabilities. The proposed approach trains a separate model known as the gap predictor, which simply predicts the restoration gap for a given sample. Then this gap predictor, which is differentiable, can be used to guide the diffusion sampling process away from unfeasible plans. Additionally, since errors in the gap predictor could lead to model exploitation, the authors propose to use attribution maps to perform regularization. Experiments on various offline control tasks (Maze2D, Mujoco locomotion, Block stacking) show that the proposed approaches (with/without regularization) outperform several strong offline RL baselines. The authors also show how the attribution maps can be used to identify specific parts of samples which contribute the most to the restoration gap, which is useful for identifying problematic transitions for instance.
Strengths
The proposed work seems novel and the experimental results show that the proposed work is state-of-the-art. From my readthrough, I didn't identify any glaring technical flaws and the approach itself seems sound. Section 5.3 is very solid and highlights a pretty interesting application of the proposed approach that I haven't seen before.
Weaknesses
I disagree a bit on the high-level motivation for this work. I don't think there is anything about the restoration gap that specifically targets "unfeasible" plans (by "unfeasible" I mean plans that we cannot execute due to certain constraints, which is different from plans that are just bad under some reward function). For instance, I would expect that highly OOD plans have very high restoration gaps even if they are technically feasible to execute, because our diffusion model will struggle to reconstruct plans that are not in the training distribution. My suspicion is that this approach is equivalent to performing some kind of low temperature sampling to ensure predicted samples are high confidence. If that is the case, then I think the paper needs some rewriting because the approach really is not related to feasibility I think. Would be curious to hear if the authors have a counterpoint to this, but otherwise I think this should be addressed. Since I do think this is fairly major concern with the writing, my score is a bit lower, but I still think the core technical contribution of this paper is reasonable so I'm happy to bump my score if the paper is revised, or it turns out that I am incorrect in my suspicions. The experimental results aren't terribly impressive, in most settings the proposed approach is barely better than Diffuser.
Questions
Is it possible to adapt the metrics compared to in 5.1 to downstream control? I think it would be helpful if the authors could show that the proposed restoration gap metric is actually better than prior metrics on relevant control benchmarks. I'm not super familiar with these other metrics so I don't know how feasible that is. I would suggest including more formal definitions for "faithful" and "infeasible" plans. I have no idea what a faithful plan is. I think I know what an infeasible plan is, but given the aforementioned confusing around the motivation for the paper, it's possible I have a different definition in mind. Also for some reason I am unable to open the supplementary materials zip file, so nothing in there affected my review. === I have read the rebuttal and bumped my score up. I am in favor of acceptance.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Yes.
I appreciate the comprehensive rebuttal from the authors. On the topic of experimental results, I think the empirical results are more impressive with the added context. As for the high-level motivation, I think we're on the same page about the fact that the approach targets unfeasible trajectories using "OOD"-ness, where the paper argues (and empirically supports) that when trained on expert datasets, unfeasible trajectories are likely to be OOD for our model. I would still suggest that the authors be careful when doing revisions to distinguish being feasible and being OOD since they are definitely different, although I agree that this is a useful heuristic and would be of interest to the research community. Also, thanks for the Dropbox link. Not sure what the issue was, but much appreciated. With all that being said, I'll bump my score up and recommend acceptance.
We are pleased to hear that our rebuttal effectively addressed your concerns, and we are sincerely grateful for the increased score and your recommendation for acceptance. We will make sure to clarify what unfeasible trajectories are and discuss how they relate to OOD plans in the final version of our paper. If you have any remaining suggestions or concerns, please let us know before the discussion period ends. Once again, we extend our gratitude for your invaluable suggestions and comments.
I appreciate the authors' thorough response during the rebuttal process and applying the suggested changes. I agree that the experimental results with the additional context are now more noteworthy. Regarding novelty, I appreciate the discussion provided by the authors. While I understand that the application of an existing technique to a new domain may yield novel insights, I must clarify that *in my view*, novelty also encompasses the introduction of genuinely new concepts that significantly contribute to the advancement of a particular problem domain. I would suggest to tone down the claims of novelty in the paper to ensure a clear and accurate representation of the work's contributions. I acknowledge that the notion of novelty can vary between reviewers, and I respect that other reviewers have expressed their recognition of the novel aspects of the approach. However, I still feel it is important to emphasize that my assessment of novelty is based on the understanding that introducing new techniques or concepts is a fundamental aspect of what constitutes novelty. I still have some confusion regarding the main motivation of the paper which is to target infeasible plans using restoration gap. The diffusion model will failt to reconstruct plans that are out of distribution, however infeasible plans and OOD plans are not necessarily the same. Therefore, I agree with other reviewers and I strongly suggest distinguishing between being feasible and being OOD in the paper and why the current set-up targets both. After thoughtful contemplation of the feedback provided by both the authors and my fellow reviewers, I am leaning towards adjusting my initial score for the paper. While I maintain the belief that the concerns raised in my original review are valid, the thorough discussion regarding infeasible versus OOD plans, along with the additional context provided for the experimental results, has led me to view this work more positively.
We are pleased to hear that you acknowledge that our experimental results are more significant with the added context. Regarding the aspect of novelty, we appreciate your perspective, and we understand the distinction you've made between applying existing techniques to new domains and introducing entirely novel concepts. While we agree that our approach is not a genuinely new concept, we believe that the contributions of the proposed metric with theoretical justification and consequent application on refining diffusion planners are not subtle and hold substantial merit. To address your viewpoint, we will elaborate on related concepts (Meng et al., 2021; Nie et al., 2022) and highlight the distinguishing features of our approach in the Related Work section. In regard to the main motivation of our work, we would like to clarify what infeasible plans are once again and elaborate on how the restoration gap identifies those plans. We’ll also discuss how the restoration gap is related to OOD-ness. Infeasible plans are those that are incompatible with system constraints and cannot be executed. The high-level motivation behind identifying such plans is that the diffusion model would fail to reconstruct OOD plans. It motivates the restoration gap to perturb the initial plan and then reconstruct it. As we discussed with Reviewer iw4y, the restoration gap implicitly addresses infeasible plans through the lens of OOD-ness. By utilizing the diffusion planner’s temporal compositionality property, which enables the stitching of in-distribution subsequences, we argue that most infeasible plans are likely to be OOD, while feasible plans would generally remain within the distribution. As such, the restoration gap becomes a tool for identifying infeasible plans through their OOD characteristics. In response to your feedback, we will make sure to elaborate on the relationship between the restoration gap and OOD-ness. We genuinely appreciate the time and effort you've dedicated to thoroughly review our manuscript and offer your valuable insights. We sincerely hope that this response, coupled with our earlier rebuttal, has effectively addressed your concerns. If you have any further suggestions or concerns, please feel free to share them with us before the discussion period concludes. Once again, we extend our gratitude for your thoughtful consideration and feedback. **References** - Meng et al., Sdedit: Guided image synthesis and editing with stochastic differential equations. In *ICLR*, 2021. - Nie et al., Diffusion models for adversarial purification. In *ICML*, 2022.
I appreciate the clarifications by the authors, I will surely consider these points in my final evaluation.
Thank you for acknowledging the novelty and contributions including performance improvement and explainability of our work. In the final version, we’ll incorporate the clarifications we mentioned in our rebuttal into the paper. Furthermore, we will tone down our claim regarding the planning performance and integrate the content from the Limitations section in the Appendix into the main body of the paper. If you have any additional suggestions or concerns, please feel free to share them with us before the discussion period ends. Once again, we extend our gratitude for your invaluable suggestions and comments.
Thanks for the author's patient response
Thanks for the author's patient response. I believe the proposed solution and theoretical analysis are both effective and correct.
We are glad to hear that our rebuttal was helpful and that you find both the proposed solution and theoretical analysis effective and correct. In response to your feedback, we will provide insights into the accuracy of the restoration gap through theoretical analysis in the revised paper. Furthermore, we will elaborate on the sensitivity of the planning performance in relation to the choice of scaling coefficients, along with an explanation of our tuning process. If you have any remaining suggestions or concerns, please let us know before the discussion period ends. Thank you once again for your invaluable suggestions and comments.
Decision
Accept (poster)