Diffusion models have risen as a promising approach to data-driven planning, and have demonstrated impressive robotic control, reinforcement learning, and video planning performance. Given an effective planner, an important question to consider is replanning -- when given plans should be regenerated due to both action execution error and external environment changes. Direct plan execution, without replanning, is problematic as errors from individual actions rapidly accumulate and environments are partially observable and stochastic. Simultaneously, replanning at each timestep incurs a substantial computational cost, and may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal. In this paper, we explore how we may effectively replan with diffusion models. We propose a principled approach to determine when to replan, based on the diffusion model's estimated likelihood of existing generated plans. We further present an approach to replan existing trajectories to ensure that new plans follow the same goal state as the original trajectory, which may efficiently bootstrap off previously generated plans. We illustrate how a combination of our proposed additions significantly improves the performance of diffusion planners leading to 38\% gains over past diffusion planning approaches on Maze2D, and further enables the handling of stochastic and long-horizon robotic control tasks. Videos can be found on the anonymous website: \url{https://vis-www.cs.umass.edu/replandiffuser/}.
Paper
Similar papers
Peer review
Summary
The authors address the online replanning problem within diffusion-based models by introducing their method, RDM, which uses the likelihood of trajectories to decides when and how to replan.
Strengths
- The topic of when and how to replan is an important one, particularly within the diffusion model community where it has not, to the best of my knowledge, been addressed before. - The methodology section appears to be sound, and the idea of choosing to replan and what type of replanning should happen based on the trajectory likelihoods is compelling. - The paper is clear, well-written and easy to follow.
Weaknesses
- The experimental results are lacking in terms of comparison with other simple baselines that are able to replan. While the authors partially address this in the ablation study of Section 4.4, it is unclear why these are also not used in the main experiments. Given the aim of the paper is to provide a good replanning strategy, I do not believe this should be an ablation only, but rather that most results should include “no replanning baselines” (i.e., the ones already provided), and “replanning baselines” as proposed in the ablation. This would allow a reader to take conclusions regarding the actual improvement from performing no replanning to a basic strategy to the proposed method. - The absence of replanning in the baselines could justify the significant drop of performance in the results in Section 4.1, 4.2 and 4.3. A comparison with replanning baselines (both fully and, for example, with a randomly selected number of steps) strengthens the case that the “when” and “how” to replan are key to the success of the results. - The paper is also lacking an ablation with a variable number of replanning steps. In Section 4.4 the authors compare DD which replans “using two fixed intervals”, but how does the performance change for a higher replanning frequency? **Minor comments**: - Typo in abstract, line 1, should be “have risen as a promising”
Questions
- How does this method compare to a fixed, fully replanning strategy at different intervals in the main scenarios compared in the experiments? - How are the thresholds in Algorithm 1 tuned, and how does the method perform for different $l_1$/$l_2$ values?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors address some limitations of their method in Section 6 of the paper, and there are no concerns regarding potential negative societal impact for this work.
Summary
The submission describes a method for deciding when and how to "replan" when using diffusion models for inferring plans, as in Decision Diffuser [1]. The task is essentially imitation learning (IL): given a training dataset of plans and features, the task is to predict a new plan given novel features. As in [1], the IL method consists of training a diffusion model (DM) to predict state sequences given contexts, and the features happen to include the reward of the plan. A high-reward plan can therefore be inferred by conditioning the model on a high reward value. The replanning aspect of this work is useful because running the diffusion model may be expensive, making it difficult to use diffusion-model-based planning to do receding-horizon control. Replanning only when necessary saves computation compared to planning every cycle.
Strengths
Originality: The ideas of trying to infer when and how to replan when planning with diffusion models seem novel. I'm not aware of other work on inferring when to replan in general, so that general idea may be novel as well. A nice feature of applying probabilistic methods to planning is that it becomes quite natural to ask such questions. Quality: There are some promising aspects to the experiments. I appreciated that the experiments were targeted towards answering specific questions that were clearly articulated in lines 182-185. The robotic control (RLBench) results were also promising in that the success rate was significantly higher than that of a baseline without replanning. Significance: The results show that simply replanning based on some simple heuristics is a viable technique for boosting the performance of diffusion-based planning methods. This is a potentially useful and interesting observation for anyone interested in applying diffusion models to solving planning problems in practice.
Weaknesses
Quality: I believe the weakest part of the submission is that the experiments are focused mainly on "apples-to-oranges" comparisons of RDM to model-free RL and decision diffusion models without replanning. None of the baselines have the ability to respond to new information at test time, as far as I can tell. In the case of the model-free RL algorithms, this is a bit unclear—theoretically, the learned policies could condition on "real-time" information, but it's unclear whether this is the case. Still, even if the RL policies do condition on novel information, comparing a policy learned offline to a planner that can be evaluated online, seems a bit unfair. It would make more sense to me if the experiments were focused more on comparisons between RDM and other replanning methods. Since the goal of intelligent replanning is to reduce computation with respect to replanning every cycle, I think it would also be fair to evaluate the net effect of different replanning strategies on the tradeoff between solution time and solution quality. For example, if we replan at fixed intervals, but evaluate that for a range of intervals, what trade-off do we see between solution time and quality compared to RDM? If we replan every cycle, is the solution quality much higher than RDM? Although the ablation study in section 4.4 does analyze the effects of different replanning strategies, this analysis feels unsatisfying for a few reasons. First, it is unclear how the threshold / interval for replanning were chosen for DD and SDM, which simply replan based on time or state distance thresholds. Ideally, these thresholds would be chosen to maximize some metric evaluated on a validation set, but I could find no details about this. I also note that the baseline for figure 7 is not zero, which gives a very misleading impression about the relative performance gap between (e.g.) SDM (replan triggered by state distance) and RDM, which is actually pretty small. There are also no error bars on this plot, so it is hard to tell whether this result is significant. This experiment was also only run on a single task (hopper-medium-expert)—the results would be much more interesting if these simple baselines (DD, SDM) were run on the entire suite of tasks. If it turns out that SDM performs almost as well as RDM on most tasks, then that would make RDM significantly less appealing. Clarity: The problem statement is unclear. On my initial read-through of the paper, I was under the impression that RDM (the proposed method) was an imitation learning (IL) method—until I reached the experiments, which compared RDM to RL methods. I then backtracked to Section 2.1 (problem setting), which states: "We consider a trajectory optimization problem similar to Diffuser[17]... the objective of the trajectory optimization problem is to find a sequence of actions … that maximizes J… which is the expected sum of rewards over all timesteps." After reading the reference for Diffuser[17], this paragraph convinced me that my original belief was wrong, and that RDM is in fact an (online) RL method that uses a diffusion model as a planner—just like Diffuser[17]. I then at some point encountered a Decision Diffuser[1] reference in the paper and decided to revisit that reference to refresh my memory. I was very confused because the paper was not what I remembered it being, and in fact Decision Diffuser[1] is an IL method. I eventually realized that Diffuser[17] and Decision Diffuser[1] are different methods: the first is an RL method, and the second is an IL method. However, they both share common authors, use similar methods, and have similar names. I then re-read lines 89-91, ("we follow the formulation of planning in Decision Diffuser[1]…") which convinced me that my original impression was correct: RDM is an IL method, not an RL method. Despite the unfortunately similar names, I believe the real root of my confusion is that the paper lacks a crisp problem statement—the reader should not have to guess as to whether the problem addressed is IL, or offline/online off-policy/on-policy RL. It's unfortunate that Section 2.1 (Problem Setting) introduces the method in RL terms and explicitly states that the problem is similar to that of Diffuser[17], which is an RL method. Instead, that paragraph should probably state something like "Our work addresses a problem similar to that described in Decision Diffuser[1]: given a dataset of training state trajectories and rewards, our task is to predict a state trajectory similar to what was observed in the training dataset, while conditioning on a query reward value. This is essentially an Imitation Learning problem where we observe and condition on rewards, but it is also similar to RL, in the sense that the goal is to produce a trajectory with high reward." The comparison to RL methods also confuses things—the submission should be clearer about why RDM is compared to RL methods. It would also be beneficial to call Diffuser by a different name (e.g., Planning Diffuser) to disambiguate it from Decision Diffuser. The exposition of the method could also be significantly improved. The explanation of when to replan in section 3.1 is a bit too verbose and doesn't convey the basic idea well, which is simply to evaluate the likelihood of a plan where states/actions at previous timesteps are replaced with those observed during plan execution. A simple figure would be very helpful here. Significance: One factor that may limit the significance of this work is that the topic is relatively niche—it attempts to solve a particular problem (performance of replanning) with applying a particular planning method (Decision Diffuser), which itself is still relatively immature. Solving niche problems is ok, but the potential for significant impact would be greater if there were some interesting take-away for a more general audience.
Questions
How were the intervals / state distance thresholds set for the DD and SDM baselines in figure 7? Have you tried evaluating the trade-off between solution quality and planning time for different replanning strategies (e.g., for different possible threshold values in DD and SDM)?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
1 poor
Contribution
2 fair
Limitations
The paper adequately addresses the limitations of this work.
Summary
The manuscript introduces a technique for enhance the motion planner rooted in diffusion models, encompassing decisions concerning the timing of replanning and plan trajectories upon the existing path. The strategy for timing replanning utilizes the inherent estimated likelihood of the trajectory in diffusion models as a criterion for replanning. In terms of forming new trajectories, the authors either completely reconstruct the plan or modify it based on future contexts determined by specific guidelines. The concept is clear-cut, and the outcomes appear to be quite promising.
Strengths
Replanning is important for robots to execute trajectories robustly, deciding when to replan and replanning algorithm are the core challenges of replanning problem. Tackling these problems can help a lot with diffusion based robot motion planning methods. Moreover, the overall writing quality is good, and the core idea is presented well.
Weaknesses
It's not surprise that the performance gets better by adding replanning strategy into diffusion model. For long horizon planning problem in Section 4.1, the authors compared performance of RDM and baselines, however, none of the baselines contains replanning strategy, making the effectiveness of replanning timing decision method proposed by the author hard to evaluate. This also happens to Robotic Control tasks in Section 4.3. As for stochastic environment in Section 4.2, the author demonstrates that RDM outperforms baseline models in environment with stochastic transition models. How different levels of stochasticity will affect the performance is later proved in Section 4.4. However, only one environment is tested, and the performance of RDM and other baselines are relatively close, which is inadequate to evaluate the effectiveness of replanning strategy in RDM. In Section 4.4, the author compares the performance for models using different replanning strategy, and comparison of different fixed interval/different threshold of state distance deviation is missing, which makes the performance of DD/SDM not convincing enough. There are also some points which are not demonstrated very well: - The detailed implementation of how the replanned trajectory is generated is missing. - The theoretical proof of the effectiveness of likelihood function for partially-executed trajectory is missing, and the threshold of likelihood to choose how to replan is selected empirically which may significantly impact the performance.
Questions
- For long horizon planning tasks and robotic control tasks, is using likelihood better than using state distance deviation as replanning criteria? - Comparing baselines with different replanning timing decision methods and RDM, how much efficiency improvement will RDM achieve? - For stochastic environment, the advantage of methods implementing replanning strategy comparing to non-replan methods should be increased when stochasticity arises, but the performance gets closer shown on Figure 6. Why?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The author addressed the limitations well.
Summary
This paper introduces Replanning with Diffusion Models (RDM), which utilizes an internally estimated likelihood of the current plan to determine when to perform replanning. The authors propose various strategies for replanning in different scenarios.
Strengths
1. The introduction of Replanning with Diffusion Models (RDM) and the use of an internally estimated likelihood of the current plan for replanning strategies is novel. 2. The experimental results demonstrate impressive performance in robot control and stochastic environments, surpassing baselines like IQL.
Weaknesses
1. The validation of the proposed method is not fully comprehensive. In RDM, a key aspect is the estimation of likelihood, but the authors did not discuss the accuracy of the estimation or its impact on final performance. 2. The paper contains some language issues and typos that should be carefully reviewed. For example, in line 178, it should refer to Figure 7 instead of Table 7. Additionally, in line 37, "we propose a principled approach to xxx" has a grammatical error. 3. In the robot control experiments using the RLBench domain, comparing IQL or DT as baselines might not be sufficient. It would be beneficial to compare against other agents known for performing well in the RLBench domain.
Questions
1. I am concerned about the computational resource requirements introduced by the Diffusion Models. For achieving similar performance levels, could you provide insights into the training time and resources needed for IQL, DT, and RDM? 2. While I understand the advantages of replanning, which reduce unnecessary exploration, in RL learning, accurate judgments cannot be made without visiting certain states. How can you ensure that the states not visited after replanning are indeed redundant for learning? 3. The direct introduction of noise into the D4RL dataset (Sec. 4.2) seems unreasonable. Why not collect data directly from random environments to validate this point?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
N/A
Summary
This paper studies how to effectively replan with diffusion models. The authors propose an adaptive online replanning strategy using diffusion models. This strategy employs the estimated likelihood of a plan's success to determine when replanning is needed, avoiding frequent, computationally expensive replanning. It also ensures new plans align with the original trajectory's goal, leveraging previously generated plans efficiently. This method led to a 38% performance improvement over previous diffusion planning approaches on Maze2D and enabled the handling of stochastic and long-horizon robotic control tasks. Main Contributions: 1. A method to determine when to replan with diffusion 2. A method to generate new plans while utilizing the existing plan
Strengths
- **Movitation**: Online re-planning is a natural and important complement to many planning methods, thus the motivation of this work is important. - **Methodology**: This paper proposes two methods: one is used to determine when to replan while the other one is for how to replan. The two methods worked together to conduct effective online replanning with diffusion models. As far as I know, the proposed methods are novel. - **Performance Improvement**: The proposed method significantly improves the performance of diffusion planners, with a reported 38% gain over past diffusion planning approaches on Maze2D. And the authors also showcased that the proposed method enable the handling of stochastic and long-horizon robotic control tasks.
Weaknesses
- It is still unclear to me why replaning at every time step does not work. In the figure 7, the Dcesion Diffuser replans at fixed intervals and it performs much worse than the proposed method. However, the authors did not mention how large is the interval is. If the replaning frequency is very low than it should apprently works worse than the proposed method. And in the abstract, the author mentioned that "replanning at each timestep may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal". I am not convinced by this argument without any exmaples or further explanations. - It is also unclear to me why "Replan on State Distance (SDM)" works so badly according to Figure 7. In Section 3.1, the authors mentioned that "However, in many cases, even if the actual state the agent is in matches with the state in the plan, new information about the environment may make the current plan infeasible." This does not make too much sense to me. I appreciate it if the authors could provide further details about this point. - Computation budget is an important factor to consider when comparing different methods. (If we have an unlimited computation budget, I feel it is a good idea to replan at each timestep.) However, the numbers of replanning used in each baseline are not present in the paper. If the numbers of replanning vary a lot among different methods, then I would not think it is a fair comparison. I wish the authors could provide the numbers of replanning used in each method. - Although the proposed method aims to avoid frequent, computationally expensive replanning, it's not clear how computationally efficient the new method is, and whether the computational cost of determining when to replan offsets the benefits gained from less frequent replanning.
Questions
- It would be great to provide the number of "replan from scratch" and the number of "replan with future context" used in each experiment, since this would help the audience further understand how the replanning helps the agent. - Why does "replan with future context" work better than "replan from previous context"? Could you give any intuitive explanations? - See the weakness section for other questions.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
- This method only applies to diffusion models as it adds noises at some diffusion steps. It is unclear how to utilize this method in other planning methods. - See the weakness section for other limitations.
Thank you and we are looking forward to your post-rebuttal feedback!
Dear AC and all reviewers: Thanks again for all the insightful comments and advice, which helped us improve the paper's quality and clarity. The discussion phase has been on for several days and we have not heard any post-rebuttal responses yet. We would love to convince you of the merits of the paper. Please do not hesitate to let us know if there are any additional experiments or clarification that we can offer to make the paper better. We appreciate your comments and advice. Best, Author
Response to Rebuttal
Thank you for the detailed rebuttal. **On the comparison**: I thank the authors for the extra results, as they contextualize the performance improvements of RDM. I believe include this in the paper will make the case for the method stronger. **On the replan interval**: the failing to plan with a high replanning frequency is an interesting observation. These results show that a simple fixed time replanning strategy significantly underperforms RDM, which again makes the case for the method stronger. **On the replan threshold for RDM and SDM**: these results are expected, but also showcase the interesting monotonic behavior of the returns based on the threshold. The differences between these returns and the ones obtained by simply reducing the replanning interval also showcases the non-trivial nature of the method. The authors have addressed all the concerns I had regarding the paper, so I am updating the score accordingly.
Dear reviewer XopQ, Thank you for your comments on our response. We are happy that your concerns have been addressed. We still have a few days left in the discussion period. If you have any further questions or if there is anything else we can provide to show the merits of our work, please don't hesitate to let us know. Thank you! Best, Authors
Dear reviewer G16t, Thank you again for your comments and suggestions on our paper. We hope that our responses and new results have addressed your questions and concerns. We still have a few days left in the discussion period. If you have any further questions, please don't hesitate to let us know and we'll be happy to address them. Thank you! Best, Authors
Dear reviewer kWoD, Thank you again for your comments and suggestions on our paper. We hope that our responses and new results have addressed your questions and concerns. We still have a few days left in the discussion period. If you have any further questions, please don't hesitate to let us know and we'll be happy to address them. Thank you! Best, Authors
Dear reviewer rt81, Thank you again for your comments and suggestions on our paper. We hope that our responses and new results have addressed your questions and concerns. We still have a few days left in the discussion period. If you have any further questions, please don't hesitate to let us know and we'll be happy to address them. Thank you! Best, Authors
Dear reviewer bzBf, Thank you again for your comments and suggestions on our paper. We hope that our responses and new results have addressed your questions and concerns. We still have a few days left in the discussion period. If you have any further questions, please don't hesitate to let us know and we'll be happy to address them. Thank you! Best, Authors
Rebuttal response
Thank you for adding the results comparing the total diffusion steps and task performance across different methods. This does alleviate a significant concern of mine.
Hi, Thank you for your help reviewing this paper and for participating in the discussion with the authors. It seems that the authors' response resolves resolves at least some of the issues that you raised in your initial review. Please be sure to update your review and rating accordingly. Best,\ AC
Dear reviewer bzBf, As the rebuttal period is ending today, we wonder if our response answers your questions and addresses your concerns. If yes, would you kindly consider raising the score? Thanks again for your very constructive and insightful feedback! Best, Authors
Looking forward to your post-rebuttal feedback!
Dear AC: Thanks again for handling our submission efficiently! Till now, we are still waiting for the feedback from Reviewer G16t, kWoD and rt81. We fully understand the reviewers might be too busy to respond timely, but we do hope for more discussions and convince them of the merit of our work! Could you please help us reach out to the reviewers for feedback? We appreciate your kind support! Best, Authors
Thanks for the rebuttal!
Thank the authors for the rebuttal, and some of my concerns have been addressed. See the follow-up questions below. - **About replanning at every time step:** What happens if the interval is very small? Can you provide an intuitive example? - **Computation cost of determining when to replan:** This is the 4th point I raised in my initial review, I wish the authors could provide some details about it.
Thanks for your response
Dear Reviewer: Thank you for the insightful questions. **Q1. About replanning at every time step.** We visualized the agent's trajectories under different replanning intervals in Figure XIII in the rebuttal PDF. We can see that if the replanning interval is very small (in Figure XIII \(c\), the agent replans at every time step), the agent appears "staggering" in the environment. This is because the plans generated by different environmental steps may be inconsistent. This observation confirms that replanning at every time step would prevent consistent progress towards the goal, which results in a worse performance. **Q2. Computation cost of determining when to replan.** The computation cost of RDM depends on the number of diffusion steps for planning and replanning. The number of diffusion steps for planning and replanning is about $2 \times 10^4$ while the number for determining when to replan is about $3 \times 10^3$ in stochastic environments. And for long-horizon planning, the numbers are about $2 \times 10^3$ for planning and replanning and $2 \times 10^2$ for determining when to replan. Kindly let us know if our responses have addressed your questions and concerns, and if you have further questions. Thank you! Best, Authors
Thank you for the response. I don't have further questions for now. I have revised my rating accordingly.
Follow-up on score updates!
Dear reviewer bzBf, As the rebuttal period is ending soon, we wonder if our response answers your questions and addresses your concerns. If yes, would you kindly consider raising the score? Thanks again for your very constructive and insightful feedback! Best, Authors
Rebuttle Response
Thank you for taking the time to provide a thorough explanation. It's evident that the RDM replanning method holds an advantage over more basic replanning techniques, especially in longer sequence tasks. However, its performance in environments with high levels of stochasticity seems to be an area of potential improvement. While optimizing the timing of replanning via likelihood function to enhance the planning success rate is a valuable insight, it doesn't drastically alter my initial impression. Nonetheless, I truly appreciate your efforts in clarifying the methodology.
Decision
Accept (poster)