Summary
TLDR; Unlike traditional methods that generate the entire trajectory at once, this process gradually refines the plan at each stage, reducing computational costs and improving real-time performance.
DiffuserLite is a lightweight diffusion planning framework designed to increase decision-making frequency in real-time applications. Traditional diffusion planning methods are limited by the high computational costs of modeling long-horizon trajectory distributions, leading to low-frequency decision-making. To overcome these limitations, DiffuserLite introduces a Progressive Refinement Process (PRP) that reduces redundant information and progressively generates fine-grained trajectories. As a result, DiffuserLite demonstrates superior performance compared to existing methods across various benchmarks such as D4RL, Robomimic, and FinRL. Additionally, DiffuserLite is designed as a flexible plugin that can be easily integrated with other diffusion planning algorithms, making it a valuable contribution to future research and practical applications.
Strengths
A simple idea with a order of magnitude speed improvement.
- The PRP process reduces time cost by starting with coarse goals and refines them into detailed goals.
- It reduces unnecessary redundant planning and ensures faster inference.
- With the speed improvement, DiffserLite shows reasonable performance.
Flexibility (RQ3)
- In the form of a plugin, it can be applied to other planning methods such as AlignDiff.
Weaknesses
- The terms or concepts such as "reflow" lack adequate explanation. Although they are mentioned in the appendix, it is necessary to provide an explanation or a reference to this section in the main text. Even a basic explanation within the main part would be beneficial.
- The abbreviation “PRP” is used to abbreviate multiple terms. On page 2, it refers to "plan refinement process," while on page 4, it stands for "progressive refined planning." Although these terms appear to be used interchangeably, it would be helpful to clarify this in the text.
- It would be beneficial to have a theoretical justification for the experimental results. The concept is straightforward, but it would be helpful to explain why certain configurations of levels are better in the ablation study comparing the levels of DiffuserLite. Currently, the analysis seems too intuitive and empirical.
- DiffuserLite adopts Conditional Filtered Guidance (CFG) instead of Conditional Guidance (CG) to prevent speed degradation. However, the adaptation of CFG requires adjusting the target condition at each level in the multi-level structure. It would be beneficial to compare the performance and speed of DiffuserLite using CG in addition to CFG.
- Equation 10 in the paper shows that the optimal value function is added as a term in the Critic to solve zero Critic values issue in sparse reward settings for fine-grained planning. However, using this optimal value requires an offline methodology, which limits the approach. Methods studied in Online RL may replace optimal value. For example, it would be possible to consider studies on exploration strategies such as intrinsic motivation or curiosity-based reward, or value-based method enhancement studies such as double Q-learning or dueling DQN.
Questions
- What is the default model for Lite w/o PRP that you mentioned in section 5.5? You mentioned a one-level model, but the default design in Table 6 has three temporal horizons [5,5,9]. Is the entire planning horizon used as the temporal horizon? It would be helpful to clearly specify the actual temporal horizon used in the study.
- In Figure 1 and Table 2, the scores of DD and Diffuser for the antmaze diverse dataset are recorded as 0. However, in the ICLR 2024 paper "Reasoning with Latent Diffusion in Offline Reinforcement Learning," those models show higher scores for antmaze-diverse-v2. This discrepancy suggests that either a different dataset version was used or the hyperparameter settings need to be adjusted for a fair comparison.
- Trade off between depth of searching and removing redundant information seems good idea. But is there a optimal point for this trade off? Or does it considered by task?
- I think critic design is one the most important part in this paper, but there are lack of explanation. Were there any literature to solve this problem by modifying Critic parts? Were there any other research that used the sum of discounted reward and discounted optimal value to solve the problems caused by sparse rewards?
- According to Equation 10, Critic has the optimal value term. However, this approach is limited to offline methodology. Are there any alternatives or methods to overcome this limitation?
- The paper explains that CFG is adopted instead of CG to prevent speed degradation. Given that CFG requires adjusting the target condition at each level in the multi-level structure, are there any methods to mitigate this limitation? Additionally, could you provide experimental results comparing the performance and speed of DiffuserLite using CG?
- Table 2 shows that DiffuserLite-R1, which uses rectified flow, achieves the best performance across most metrics. However, the paper lacks a detailed analysis of these results. Could you explain the specific mechanisms through which rectified flow contributes to performance improvement?"
Limitations
- It is insufficient to prove the way to choose the number of planning levels and the temporal horizons. It would have been beneficial to compare the number of planning levels or temporal horizon with respect to the learning / inference frequency, analyzing the trade-offs when these factors are changed. Additionally, comparing whether it is better to choose the temporal horizon in a bottom-up or top-down manner according to the levels would have provided a more comprehensive analysis.
- DiffuserLite may also have some societal impacts, such as expediting the deployment of robotic products, and it could potentially be utilized for military purposes." If they provide more examples that highlight the practical aspects of DiffuserLite, readers may more easily understand the potential of the paper.
- For some baseline(DD and Diffuser), the results shown in Table 2 are too low to make the experimental results appear unfair. Refer to results from Table 1 of the existing study (arXiv:2309.06599) on diffusion-based models performing one-shot planning. For the same benchmarks (antmaze-diverse, kitchen-partial, kitchen-mixed), they showed better baseline performance. It is expected that even with the step-by-step planning method of this study, proper hyper-parameter tuning could achieve at least similar performance.