DiffuserLite: Towards Real-time Diffusion Planning

Diffusion planning has been recognized as an effective decision-making paradigm in various domains. The capability of generating high-quality long-horizon trajectories makes it a promising research direction. However, existing diffusion planning methods suffer from low decision-making frequencies due to the expensive iterative sampling cost. To alleviate this, we introduce DiffuserLite, a super fast and lightweight diffusion planning framework, which employs a planning refinement process (PRP) to generate coarse-to-fine-grained trajectories, significantly reducing the modeling of redundant information and leading to notable increases in decision-making frequency. Our experimental results demonstrate that DiffuserLite achieves a decision-making frequency of 122.2Hz (112.7x faster than predominant frameworks) and reaches state-of-the-art performance on D4RL, Robomimic, and FinRL benchmarks. In addition, DiffuserLite can also serve as a flexible plugin to increase the decision-making frequency of other diffusion planning algorithms, providing a structural design reference for future works. More details and visualizations are available at https://diffuserlite.github.io/.

Paper

References (55)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer r5j29/10 · confidence 4/52024-07-08

Summary

The paper introduces a lightweight framework employing a Plan Refinement Process (PRP) for generating trajectories from coarse to fine-grained levels. This approach reduces redundant information modeling, significantly enhancing planning efficiency. DiffuserLite achieves an impressive decision-making frequency of 122.2Hz, making it 112.7 times faster than existing frameworks and suitable for real-time applications. Additionally, it can be integrated as a flexible plugin to accelerate other diffusion planning algorithms. Experiments demonstrate state-of-the-art performance on multiple benchmarks, highlighting DiffuserLite's robustness, efficiency, and practical applicability across various domains.

Strengths

- DiffuserLite achieves an impressive decision-making frequency of 122.2Hz, making it 112.7 times faster than existing frameworks, which is crucial for real-time applications. - The PRP generates trajectories from coarse to fine-grained levels, reducing redundant information modeling and significantly enhancing planning efficiency. - DiffuserLite can be integrated as a flexible plugin to accelerate other diffusion planning algorithms, demonstrating its adaptability and potential for widespread use across various domains.

Weaknesses

The paper is well-written and easy-to-follow, I do not see obvious weakness.

Questions

- The proposed DiffuserLite is a classifier-based conditional diffusion model. How does it provide gradients? - There may exist some physical constraints for the generated trajectory, which are not differential and can not provide gradients. How does DiffuserLite work in that case?

Rating

9

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Reviewer 7Mwp6/10 · confidence 3/52024-07-12

Summary

The paper introduces a method to accelerate diffusion model-based planning called the Plan Refinement Process (PRP). This method divides the planning of the entire trajectory into several temporal spans, focusing on the most recent spans in each planning stage, thereby discarding redundant distant information. The paper also employs techniques such as critic design and rectified flow to ensure both the frequency and accuracy of planning. Experimental results show that the proposed DiffuserLite, which incorporates PRP, achieves a high decision-making frequency, which is significantly faster than predominant frameworks. This significantly enhances the real-time applicability of diffusion planning in various benchmarks. Additionally, DiffuserLite's flexible design allows it to serve as a plugin to improve other diffusion planning algorithm.

Strengths

One of the advantages of the paper is the combination of a coarse-to-fine architecture with diffusion model based planning. This method avoids redundant information in long-term planning and significantly improves planning efficiency. The authors also integrate classifier-free guidance (CFG), introduce a new critic, and use techniques such as rectified flow. As a result, the proposed DiffuserLite possesses the capability for fast and successful planning. Another advantage of the paper is the ability to flexibly apply the proposed method to other diffusion planning frameworks. This approach significantly increases the speed of planning inference with only a slight sacrifice in performance. The comprehensive comparisons in the paper demonstrate the potential of the proposed PRP in diffusion planning applications.

Weaknesses

One of the primary innovations of the paper is the introduction of the Plan Refinement Process (PRP). However, this hierarchical approach is not entirely original, as similar concepts have been introduced in prior works such as HDMI and HD-DA, mentioned in the 6. Related Works section. Additionally, many of the other improvements, such as the use of rectified flow, rely on existing methods. As a result, the paper's novelty is somewhat limited.

Questions

In the Abstract, the website provided by the authors does not contain updated details and results. Please remember to update it. In line 126, the authors mention that agents often struggle to reach distant states proposed by the planning process. Could you provide a deeper explanation for this? Otherwise, what is the significance of long-term planning? In lines 176-178, why does this critic design make it difficult to distinguish better trajectories in short-term planning? What is the "uniform critic" mentioned in line 186? Could you further explain it? What are the specific metrics used in Tables 2, 3, and 4?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Area Chair Dh7m2024-08-13

Reminder to reviewer

Reviewer 7Mwp, As the author-reviewer discussion phase comes to a close, have the authors satisfactorily addressed your concerns? Please engage in the discussion if you still have any unsolved concerns. Best, AC

Reviewer XA4d5/10 · confidence 3/52024-07-13

Summary

This paper introduces DiffuserLite, a lightweight framework that utilizes progressive refinement planning to reduce redundant information generation and achieves real-time diffusion planning. Key contributions are: - Introduced the plan refinement process (PRP) for coarse-to-fine-grained trajectory generation, reducing the modeling of redundant information. - Introduced DiffuserLite, a lightweight diffusion planning framework, which significantly increases decision-making frequency by employing PRP. Results: - Achieved a decision-making frequency of 122.2Hz, 112.7x faster than existing frameworks and SOTA performance on multiple benchmarks, e.g. D4RL, Robomimic, and FinRL

Strengths

Originality: This paper introduced a novel approach in diffusion planning: PRP, and leads to various benefits, resulting in SOTA in various benchmarks. Quality: Results are impressive. The experiment design is rigorous with comprehensive evaluation across multiple benchmarks. There are detailed information about how the experiments are set up and carried out. Clarity: The paper is well written. The figures are in good quality and helps illustrate key points. Significance: The improvements and the flexibility of the method could potentially benefits real-time decision-making applications.

Weaknesses

Details on Implementation and design choices: Discussion in the main paper about design choices and hyperparameter selection could benefit this paper. More ablation study would be helpful to understand how this method can be generalized. Many of the details are in appendix, it requires people to read though the appendix to fully understand the paper. Some of the tables can benefit from clearer presentation. e.g. table 6. While it does save some space, but it's really hard to read. Complexity and generalization: While the paper shows very impressive results on the benchmarks in the paper, the multi-level refinement process introduce quite some complexity. Additional theoretical analysis or more experimental results would help. It could be useful to provide data-driven recommendations on how to make design choices (e.g. for different levels) to make it more useful. Further discussion on the scalability of this method to more complex, real-world scenarios would be valuable.

Questions

1. The paper discussed L = 2,3,4, what would the behavior be if L >=5? 2. If the paper can share any any specific limitations or trade-offs the authors find during the development of this method, that could be helpful to discuss. It will guide people on how to use the method and strengthen the paper.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed limitations in the paper.

Reviewer xeNN6/10 · confidence 4/52024-07-13

Summary

TLDR; Unlike traditional methods that generate the entire trajectory at once, this process gradually refines the plan at each stage, reducing computational costs and improving real-time performance. DiffuserLite is a lightweight diffusion planning framework designed to increase decision-making frequency in real-time applications. Traditional diffusion planning methods are limited by the high computational costs of modeling long-horizon trajectory distributions, leading to low-frequency decision-making. To overcome these limitations, DiffuserLite introduces a Progressive Refinement Process (PRP) that reduces redundant information and progressively generates fine-grained trajectories. As a result, DiffuserLite demonstrates superior performance compared to existing methods across various benchmarks such as D4RL, Robomimic, and FinRL. Additionally, DiffuserLite is designed as a flexible plugin that can be easily integrated with other diffusion planning algorithms, making it a valuable contribution to future research and practical applications.

Strengths

A simple idea with a order of magnitude speed improvement. - The PRP process reduces time cost by starting with coarse goals and refines them into detailed goals. - It reduces unnecessary redundant planning and ensures faster inference. - With the speed improvement, DiffserLite shows reasonable performance. Flexibility (RQ3) - In the form of a plugin, it can be applied to other planning methods such as AlignDiff.

Weaknesses

- The terms or concepts such as "reflow" lack adequate explanation. Although they are mentioned in the appendix, it is necessary to provide an explanation or a reference to this section in the main text. Even a basic explanation within the main part would be beneficial. - The abbreviation “PRP” is used to abbreviate multiple terms. On page 2, it refers to "plan refinement process," while on page 4, it stands for "progressive refined planning." Although these terms appear to be used interchangeably, it would be helpful to clarify this in the text. - It would be beneficial to have a theoretical justification for the experimental results. The concept is straightforward, but it would be helpful to explain why certain configurations of levels are better in the ablation study comparing the levels of DiffuserLite. Currently, the analysis seems too intuitive and empirical. - DiffuserLite adopts Conditional Filtered Guidance (CFG) instead of Conditional Guidance (CG) to prevent speed degradation. However, the adaptation of CFG requires adjusting the target condition at each level in the multi-level structure. It would be beneficial to compare the performance and speed of DiffuserLite using CG in addition to CFG. - Equation 10 in the paper shows that the optimal value function is added as a term in the Critic to solve zero Critic values issue in sparse reward settings for fine-grained planning. However, using this optimal value requires an offline methodology, which limits the approach. Methods studied in Online RL may replace optimal value. For example, it would be possible to consider studies on exploration strategies such as intrinsic motivation or curiosity-based reward, or value-based method enhancement studies such as double Q-learning or dueling DQN.

Questions

- What is the default model for Lite w/o PRP that you mentioned in section 5.5? You mentioned a one-level model, but the default design in Table 6 has three temporal horizons [5,5,9]. Is the entire planning horizon used as the temporal horizon? It would be helpful to clearly specify the actual temporal horizon used in the study. - In Figure 1 and Table 2, the scores of DD and Diffuser for the antmaze diverse dataset are recorded as 0. However, in the ICLR 2024 paper "Reasoning with Latent Diffusion in Offline Reinforcement Learning," those models show higher scores for antmaze-diverse-v2. This discrepancy suggests that either a different dataset version was used or the hyperparameter settings need to be adjusted for a fair comparison. - Trade off between depth of searching and removing redundant information seems good idea. But is there a optimal point for this trade off? Or does it considered by task? - I think critic design is one the most important part in this paper, but there are lack of explanation. Were there any literature to solve this problem by modifying Critic parts? Were there any other research that used the sum of discounted reward and discounted optimal value to solve the problems caused by sparse rewards? - According to Equation 10, Critic has the optimal value term. However, this approach is limited to offline methodology. Are there any alternatives or methods to overcome this limitation? - The paper explains that CFG is adopted instead of CG to prevent speed degradation. Given that CFG requires adjusting the target condition at each level in the multi-level structure, are there any methods to mitigate this limitation? Additionally, could you provide experimental results comparing the performance and speed of DiffuserLite using CG? - Table 2 shows that DiffuserLite-R1, which uses rectified flow, achieves the best performance across most metrics. However, the paper lacks a detailed analysis of these results. Could you explain the specific mechanisms through which rectified flow contributes to performance improvement?"

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

- It is insufficient to prove the way to choose the number of planning levels and the temporal horizons. It would have been beneficial to compare the number of planning levels or temporal horizon with respect to the learning / inference frequency, analyzing the trade-offs when these factors are changed. Additionally, comparing whether it is better to choose the temporal horizon in a bottom-up or top-down manner according to the levels would have provided a more comprehensive analysis. - DiffuserLite may also have some societal impacts, such as expediting the deployment of robotic products, and it could potentially be utilized for military purposes." If they provide more examples that highlight the practical aspects of DiffuserLite, readers may more easily understand the potential of the paper. - For some baseline(DD and Diffuser), the results shown in Table 2 are too low to make the experimental results appear unfair. Refer to results from Table 1 of the existing study (arXiv:2309.06599) on diffusion-based models performing one-shot planning. For the same benchmarks (antmaze-diverse, kitchen-partial, kitchen-mixed), they showed better baseline performance. It is expected that even with the step-by-step planning method of this study, proper hyper-parameter tuning could achieve at least similar performance.

Reviewer xeNN2024-08-10

I have carefully reviewed your detailed responses to my comments. Your commitment to addressing the points raised is appreciated. Given the promised revisions and the forthcoming code release, I believe the simple yet effective idea of PRP could indeed be valuable in reducing the notorious planning time of diffusion models for planning. While I find the empirical results compelling, I would have been even more supportive if there were stronger theoretical justifications for the approach. Nonetheless, the practical benefits of the method are clear. Considering these factors, I am revising my overall assessment from 5 to 6. Here are a few remaining questions. 1. Regarding CFG and CG: In your response, you mentioned that CG was not used for several reasons, including decision frequency. Is there a difference in performance (score) between using CG and using CFG without considering rectified flow? In Section 4, you stated that you used CFG for decision frequency. However, in the conclusion, you mentioned the limitations of DiffuserLite with CFG being the main cause of these limitations. Could using CG address this issue? It would be helpful to explain why it was necessary to improve decision frequency by opting for CFG instead of CG. As it currently reads, your writing suggests, "We use CFG because CG is not good, but CFG also has limitations!" If CG can address this issue, it would be better to explain why decision frequency is important despite those limitations. If not, it would be clearer to explain that the issue is not about the limitations between CG and CFG. 2. Additional question about the plugin: Have you tried using this method as a plugin for other backbones besides AlignDiff? It would be beneficial to include any tests that show whether it can be applied to more major methods like Diffuser or HDMI, rather than AlignDiff.

Authorsrebuttal2024-08-13

Thank you very much for your thoughtful feedback and for increasing the score. I sincerely apologize for the delayed response (it took some time to train and test the models). I hope the following reply addresses your questions: **A1.** The normalized scores for the CG/CFG guided versions are listed below. We observe a slight performance drop when using CG. This drop was also noted in our earlier experiments, which was one of the initial reasons for choosing CFG. I believe the first work to propose using CFG was Decision Diffuser (DD), motivated by CFG's superior performance in image generation tasks compared to CG. As for why CFG performs better in decision-making tasks, there's still a lack of systematic research. Since this is not the focus of DiffuserLite, we didn't discuss it in detail. However, I have observed that CG-generated trajectories are more prone to OOD issues as guidance strength $w$ increases, which might be one reason for the performance drop. |env|CG|CFG| |---|---|---| |halfcheetah-me|87.9±1.6|88.5±0.4| |kitchen-p|66.2±4.1|74.4±0.6| |antmaze-ld|46.1±6.2|68.0±2.8| **A2.** Can CG address the issue of adjusting the target return in CFG? I don't think it can fully address this. In practice, CG requires tuning one hyperparameter (guidance strength), while CFG needs two (guidance strength and target return). Although it seems CG has one less parameter to tune, the guidance strength has a more significant impact on performance. The target return is normalized to [0, 1] based on the max/min values in the dataset, so during inference, choosing a relatively large value like [0.7-1.1] usually works well and is easy to tune. On the contrary, we must carefully tune the guidance strength for both CG and CFG. **A3.** Why did we choose AlignDiff as the backbone? This is because Diffuser/DD/HDMI are all reward-maximizing algorithms (And DiffuserLite can be seen as DD+PRP). We felt that these algorithms with PRP would still be solving the same problem in the same domain, which wouldn't effectively demonstrate the "flexible plugin" capability. Therefore, we chose AlignDiff, a preference-matching algorithm, to see if DiffuserLite could be applied to a completely different task domain - and the answer is yes. Can DiffuserLite be used with Diffuser and HDMI? Absolutely. To integrate with Diffuser, the CG version mentioned in A1 & A2 can be viewed as Diffuser+PRP (if we ignore the inverse dynamic model). To integrate with HDMI, HDMI's heuristic method for automatically selecting high-quality sub-goals could also be used to provide different levels of training data for DiffuserLite, allowing PRP to generate more meaningful and high-quality subgoals. However, implementing HDMI+PRP might have to wait until they open-source their code. Thank you again for your insightful comments and patience! If you have any further questions during the discussion phase, please don't hesitate to contact us. We'll do our best to respond and further improve the paper.

Reviewer XA4d2024-08-12

Thank you for your comments and the detailed results. They helped solve many of my questions. "the core algorithm comprises approximately 100 lines of code" -> Just to clarify, by complexity, I mean more on the complexity required when making design choices and generalize to other tasks, base models, etc. Therefore, complexity may not be measured by the lines of code, e.g. even if the solution is just one line of code, but there are 100 parameters to tune to ensure good results, I still think the complexity are there. Thank you for the detailed results for design choice recommendations. It could be super helpful if they can be included in a certain way in the final version.

Authorsrebuttal2024-08-13

Thank you for your positive feedback on our work and your appreciation of our rebuttal! I now better understand what you meant by "complexity." I'm sorry to say that introducing PRP does indeed require the selection of additional design choices. However, in practice, as you can see from the experimental results, selecting these design choices is not particularly challenging, and the performance is not very sensitive to these choices. I believe that this introduced "complexity" won't be a significant issue. Thank you once again! If you have any other questions during the discussion period, please don't hesitate to contact us. We'll do our best to respond and further improve the paper!

Authorsrebuttal2024-08-13

Dear Reviewer 7Mwp, --- As the deadline for the discussion phase is approaching, we are particularly eager to receive your feedback, as we believe some of your comments may stem from misunderstandings. Your engagement in discussion and feedback is truly important to us. We also greatly hope that the detailed explanations, additional experiments, and code releases in the rebuttal can fully address your concerns. --- Warm regards, Authors of submission 9426.

Reviewer 7Mwp2024-08-13

Reply

Thanks a lot for the efforts of the authors. I have carefully gone through the replies and added results. I really appreciate the authors' efforts. Most concerns are addressed. I would like increase my rating.

Authorsrebuttal2024-08-13

We are very pleased to be able to address your concerns. Your positive recognition means great to us, and we truly appreciate it. Your feedback is significant, and the explanations provided in the rebuttal will be incorporated into the revision to help readers better understand our work. If you have any further questions during the upcoming discussion period, please feel free to contact us, and we will do our best to resolve. Thank you for your time and efforts once again!

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC