Diffused Task-Agnostic Milestone Planner

Summary

This paper proposed a novel method to solve long-horizon, sparse-reward tasks and multi-task problems, which outperforms offline RL methods on many benchmarks.

Strengths

- Provide an elegant and general idea to solve sparse-reward problem -- which is hard for RL-based methods. - The paper is well-written and easy to be understood. - Both theoretical proof and empirical details are provided, which make the claims in the paper persuasive. - Good results.

Weaknesses

- How much do the sampling methods matter? In the paper, the number of sampled points are fixed. If the authors can provide more ablation results with varying number of sampling points or use other sampling methods, the results will be more instructive.

Questions

Mentioned in weakness.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

No obvious societal impact.

Summary

This paper extends diffusion-based latent milestone planners to long-term planning, vision control, and multi-task settings. Specifically, an encoder is trained to project observations into the latent space. The authors employ goal-conditioned imitation learning to train both the encoder and the prior goal-conditioned policy. After that, a diffusion probabilistic model is trained to model the milestone trajectories. In order to generate the shortest path leading to the goal state more efficiently, the authors take into account the temporal distance between consecutive milestones in addition to the initial state and goal state. At the low level, the prior goal-conditioned policy is used to sample atomic actions, guiding the agent toward the given milestone. Experiments conducted on selected tasks from the Deep RL for Real Life (D4RL) and CALVIN benchmarks demonstrate superior performance over the baseline models.

Strengths

This paper is well-organized. They proposed shortest-path guidance method is effective.

Weaknesses

- I think the current experiments are not enough. Could the authors also provide results on the D4RL MuJoCo locomotion tasks? It would strengthen the paper if better performance could also be achieved there. - I'm wondering why the PointMaze tasks were not selected for comparison with the results from the Diffuser[1]. - Can the authors ablate on the choice of image encoder in the image-based planning to verify that joint train a goal-conditioned actor and critic is necessary? [1]: Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the International Conference on Machine Learning, Baltimore, US, Jul 2022.

Questions

- Why are there only three different goals in the multi-goal setting? Isn't the initial state and goal state assigned manually and the goal-conditioned plans are sampled with inpainting?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper introduces a hierarchical architecture named DTAMP to solve sequential decision-making problems. Specifically, the high-level part of the architecture is realized by a diffusion model to decompose the long-term goal into several short-term milestones. Then, the low-level part makes basic decisions at fixed intervals until the milestone is reached, and the policy is trained by goal-conditioned imitation learning. The results show that the proposed approach can achieve state-of-the-art performance on D4RL and CALVIN benchmarks.

Strengths

1. The method combines the diffusion model with hierarchical reinforcement learning, which can achieve state-of-the-art performance on multiple benchmarks. 2. The intermediate latent goals, i.e., milestones obtained by the proposed approach, can be represented not only by traditional discrete labels but also by images, as shown in Figure 4.

Weaknesses

1. The biggest concern would be that the training process is not clearly explained. The authors do not state whether they trained the diffusion model and imitation learning separately or synchronously. If they're trained separately, how to obtain the labeled data of milestones? The authors should supplement enough information. 2. In Table I, the target interval value denotes the target temporal distance between milestones, which is set to be relatively smaller than the maximum value. Then, there is no restriction on the target interval value when the ratio is 1.0, which should be the same as in the unconditioned experiment. The authors should make a further clarification on the increased timesteps when the ratio is 1.0. 3. The structural relationship of Figure 1. is quite confusing. According to my understanding, the diffuser (as shown in the right subfigure) should be performed first, and then the actor and critic (as shown in the left subfigure) should be performed to obtain the trajectory. It is recommended to arrange the subfigures from left to right and establish the relationship between the two sub-figures. 4. There are some minor mistakes in this paper, such as typos in Line 122, 176, and 'inference time' in Figure 3.

Questions

1. The authors state that the trained actor is limited in inferring a suitable action for a distant goal if the offline data does not contain a trajectory connecting the current state and the goal state at the end of Section 3.1. Then, when the amount of data decreases, will the performance of the proposed approach decrease significantly? 2. In this paper, the number of planned intervals K is a fixed value. Will different values of K have a significant impact on performance, and is there a situation where the milestone cannot be reached after K intervals?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

See above.

Summary

This paper uses sequence modeling method in applications like long-term planning, vision-based control, and multi-task decision-making. They formulate a novel method which uses diffusion-based generative sequence model to plan a series of milestones in a latent space and to have an agent to follow the milestones to get the task done. Their method can learn control-relevant, low-dimensional latent representations of milestones that makes it possible to efficiently perform long-term planning and vision-based control. They train an encoder to extract milestone representations from observations, and an actor to predict actions to follow the milestones, using a goal-conditioned imitation learning fashion. Their method uses a denosing diffusion model to generate an array of milestones conditioned on a target state. The minimum temporal distance diffusion guidance method is used in this paper to make the proposed method plan the shortest path to reach the target state.

Strengths

1) The encoder trained using the method proposed, encodes control-relevant features into unique latent representations, which lets efficient planning even if the observations are high-dimensional. 2) The diffusion-based planner enables flexible planning for a variety of tasks and takes advantage on multi-task decision-making problems. 3) Using the trained actor as a feedback controller makes an agent robust against the environment stochasticity, and makes it possible to plan milestones only once at the beginning of an episode which largely reduces inference time compared to the existing sequence modelling methods. DTAMP has lower inference time compared to other baselines. 4) DTAMP outperforms the baseline methods on the every environment.

Weaknesses

1) Language of the paper could be improved. Some of the words are repetitive and there are quite some typos. for eg: "letting the the diffusion " and many other - at other places. 2) DTAMP: the ablation study on how the design of diffusion model was done is missing.

Questions

1) Did the authors re-implement all the baselines mentioned in Table2 for the comparison? 2) why and how did authors take the decisions to built the diffusion model how they did?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1) The proposed method might not work on the goal-reaching tasks and authors have not used it for the task not having goal reaching objective. 2) Their method cannot solve a new task not included in dataset. Authors can also elaborate on societal impact of their work.

Summary

The paper studies the problem of latent-space planning for sequential decision-making tasks using generative diffusion models. Similarly to works such as Diffuser, they train an endpoint-conditionned generative model to generate sub-goals along a path starting at a given state and reaching a certain goal. Differently to Diffuser, they perform the generation in a latent space pre-trained with imitation learning. The method achieves good results on D4RL environments as well as on a pixel-based planning task.

Strengths

* The paper is well-written and, for someone who is not expert in planning or diffusion models, can be very easy to understand. * The proposed approach makes sense in the current context, i.e. planning to visit subgoals in long-horizon tasks. I appreciate the approach and specifically the contribution related to this line of work, as it reduces advances in the field of planning and/or control to advances in the field of generative models (i.e. better diffusion models -> better goal trajectories -> better performance to some extent). * Experimental results show that the proposed method has competitive performance with baselines and latest SOTA, both on state-based and pixel-based domains.

Weaknesses

* I am not sure how exactly the approach differs from Diffuser (Janner et al, 2022), at least in the second phase when generating subgoals? From what I understand, the difference is in the first phase, where a latent space with desirable features for control is learned. * Based on Figure 4 of the Decision Diffuser paper, the method achieves high returns on the D4RL locomotion task, while results from your Table 2 paper seem to indicate otherwise. Can you explain this discrepancy in results? * I am curious how the method performs against recent generative-model-based model-free algorithms such as the contrastive value estimation line of work (Contrastive Learning As a Reinforcement Learning Algorithm by Eysenbach et al. 2023 and Contrastive Value Learning: Implicit Models for Simple Offline RL by Mazoure et al. 2023)?

Questions

* Can you elaborate on the distinction of your DTAMP method vs the Diffuser algorithm? * If possible, can you show a comparison against the method from “Contrastive Learning As a Reinforcement Learning Algorithm”? * Can you elaborate on the discrepancy in results between Decision Diffuser paper and your Table 2? * How was the noise schedule chosen for various experiments? If you perform state-based denoising, even in a latent space, I would imagine its structure to be somewhat different to the latent space of a pixel-based tasks.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Main limitation is that the empirical contribution and evaluation is focused on goal-conditionned tasks, which reduces to maximizing the 0-1 sparse reward function, as opposed to the general “sum-of-trajectory-rewards” objective.

Summary

The paper proposes uses a diffusion-based generative model to plan a sequence of milestones in a latent space and have the agent follow this latent plan to accomplish a given task. The authors show results on AntMaze, and more importantly on the CALVIN Benchmark to show the effectiveness of their method for image-based planning.

Strengths

- The direction of using diffusion-model based generative models for planning is definitely quite interesting. I also agree with the authors that generating the plan in a learned latent space is a promising direction. - I also thought that the use of classifier-free diffusion guidance to encourage the trajectory follow the shortest path was pretty neat. - I appreciate the authors presenting experiments on a partially observed environment like CALVIN! Showing that their method can come up with a reasonable latent plan for partially observed environment is promising. - I also liked that the authors tried visualising the learned latent space. (Fig 2. in Supp). The visualisation shows how the proposed approach, can plan successfully in the learned latent space to sequentially accomplish multiple tasks in the environment.

Weaknesses

- In L175, the authors mention that their approaches can only do the diffusion step at the beginning once, compared to existing methods which have to predict future trajectories at every step. Can the authors describe how can the policy recover from failure to reach a `milestone' in their case? If the agent's current state and the desired state is never closer than the given threshold, then according to the Alg. 1, the agent will never recover from this. - Can the authors comment on the performance of their approach will work when both the environment, and the state space is more complex and partially observed? Something that comes to mind is ImageNav in indoor environments [1] [2]. These environment seem more challenging since trajectories are often longer, and predicting intermediate milestones is much harder than CALVIN like benchmarks. - For Robustness against environment stochasticity experiments in the supplementary, isn’t the comparison to DD, and Diffuser a bit unfair? The experiments compare to DD and Diffusion by only allowing them to only predict the whole trajectory at once. But inference time related issues aside, DD and Diffuser model which predict the future trajectory at each timestep should be more robust to stochasticity in the environment than the methods which predict the whole trajectory at the beginning.

Questions

- For an exhaustive one-to-one comparisons with existing methods, it’d be good to show experiments for HalfCheetah, Hopper, and Walker 2D as done in both Janner et al, and Ajay et al. Additionally, these aforementioned papers also include block stacking experiments which aren’t included in this manuscript. - I only partially understood Fig 2 in Supplementary. How did the authors generate corresponding images for certain milestones (X) in the planned trajectory? Are these images corresponding to the state the agent reached when it was "closest" to the milestone?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Thank you for your responses!

Thank you for responding to my concerns. I am satisfied by the answers and I am happy to keep my rating.

Response

Thank the authors for the additional experiments! I'm satisfied with the results, and I'm happy to increase my score.