DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation

Summary

Offline RL utilizing Q-functions can be further enhanced if it effectively handles OOD data generated by diffusion. The paper suggests improving IQL to train a value function using OOD but constrained actions, by instead using a skill prior learned by diffusion. Furthermore, the paper suggests adaptive re-evaluation, which re-plans the trajectory if the future value function becomes worse than the current value function. DIAR outperforms prior approaches in long-horizon, sparse-reward environments.

Strengths

S1. DIAR consistantly outperforms prior works * Table 1 shows that DIAR outperforms prior offline RL & diffusion-based offline RL works in 7 out of 9 tasks. S2. Simplicity of proposed Adaptive Re-evaluation * Adaptive Re-evaluation simply compares the future value function and current value function while decision making, preventing the policy from heading to worse states. * With the assumption of goal-conditioned RL, the method is valid.

Weaknesses

W1. Lack of novelty compared to LDCQ * It seems that DIAR is an incremental improvement of LDCQ, which changed the base offline algorithm from BCQ to IQL. Specifically, DIAR uses the same procedure of LDCQ for getting the latent priors, using $\beta$-VAE for latent representation and getting the latent priors via diffusion. * Please correct me if my understanding is incorrect in Q1. W2. Restricted to goal-conditioned tasks * Due to the sparse-reward assumption for the AR (adaptive re-evaluation), the application is limited to goal-conditioned tasks. * Since DIAR is limited to goal-conditioned tasks, comparison with offline goal-conditioned RL algorithms will be insightful. Quick way to do improve this point will be adding the result of HIQL [1], which already have experimental results for AntMaze and Kitchen environments. If authors have enough time and resources, comparison with other offline GCRL methods such as GoFar [2], SMORe [3] will be informative. * Additionally, it will be exciting if there is a way to generalize AR process to dense reward tasks, making the whole algorithm generally applicable. W3. Effectiveness of re-evaluation * According to Table 2, the performance improves in 5 tasks, and decreases in 4 tasks. * Analyzing the failure cases of re-evaluation will be informative to further understand this behavior. [1] Seohong Park, et al. HIQL: Offline Goal-Conditioned RL with Latent States as Actions, NeurIPS 2023 [2] Yecheng Jason Ma, et al. Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression, NeurIPS 2022 [3] Harshit Sikchi, et al. SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning, ICLR 2024

Questions

Q1. Lack of novelty compared to LDCQ * DAIR seems to be IQL version of LDCQ + Adaptive re-evaluation. Is this understanding correct? * Would you mind to highlight the differences between LDCQ and DIAR? Q2. Comparison with goal-conditioned RL method * Can you compare DAIR with offline goal-conditioned RL methods (e.g. HIQL, GoFar, SMORe)? * Comparison with goal-conditioned RL will be informative for those willing to apply DIAR for goal-conditioned tasks. Q3. Why adaptive re-evaluation sometimes degrades the performance? * The performance decreases in 4 out of 9 tasks when AR is applied. * Examples for the failure cases of adaptive re-evaluation (e.g. Maze2D environments) and analysis for those will be informative to further understand the results. Q4. Effectiveness of AR * Can you apply AR for other methods (e.g. LDCQ, IQL) and see the improvements? One can apply AR for Q function to increase, if there is no value function in the method. Please do not hesitate to note the challenges if you have for applying AR to other methods. * If you have any, can you share the idea of generalizing AR process to dense reward tasks? It will be exciting if there is a way to generalize AR process, making the whole algorithm generally applicable. Q5. Loose bound of AR * While deriving the formula of AR, it seems that the tight bound for $V(s_{t+H})$ is $\gamma^{-H} V(s_t)$. * Can you share your thoughts on using a tighter bound $V(s_{t+H}) \geq \gamma^{-H} V(s_t)$ instead of $V(s_{t+H}) \geq V(s_t)$?

Rating

3

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Summary

The paper presents a novel offline reinforcement learning (RL) framework that leverages diffusion models to address challenges such as out-of-distribution samples and long-horizon decision-making. By introducing an Adaptive Revaluation mechanism, the DIAR framework dynamically adjusts decision lengths based on current and future state values, enhancing long-term decision accuracy. The Q-value overestimation is mitigated through the generation of diverse latent trajectories. Empirical results on tasks like Maze2D, AntMaze, and Kitchen demonstrate that DIAR consistently outperforms state-of-the-art algorithms, underscoring its potential for real-world applications in robotics and autonomous systems.

Strengths

1. The introduction of the Adaptive Revaluation mechanism is novel. It dynamically modifies decision lengths based on comparative state values, enhancing decision-making flexibility. 2. This approach is systematically evaluated, with performance improvements validated by extensive empirical results, as shown by the statistically significant outperformance of DIAR over comparable models. 3. The structure of the paper is well-organized and logical, breaking down the methodology, implementation, and adaptive mechanisms in detail.

Weaknesses

1. The experiments are primarily conducted on three well-known offline RL environments (Maze2D, AntMaze, and Kitchen) of total 9 datasets. While these provide a foundation, they are limited in diversity, which could restrict understanding of DIAR’s generalizability. Including more datasets makes the environmental results more persuasive. 2. Adaptive revaluation is proposed as a mechanism to improve decision flexibility, yet its theoretical grounding is somewhat limited. For instance, the model could benefit from a more rigorous analysis of how adaptive revaluation specifically affects trajectory optimization, particularly in long-horizon scenarios where trajectory value predictions might become noisy or overly optimistic. 3. The mixed training of Q and V, as well as the use of weighted training techniques, has not been evaluated through an ablation study, making it unclear what their contributions are. 4. There is no sensitivity analysis on the hyper-parameters, such as $\tau$ and $\beta$.

Questions

1. The article states that the mixed training of Q and V reduces overestimation issues by utilizing latent states generated by the diffusion model. However, I question the effectiveness of this approach in mitigating overestimation, as the latent states from the diffusion model are also not part of the original data. Additionally, there is no pessimistic approach incorporated into the value function during this process. How can this be justified?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Summary

This paper presents a diffusion-guided offline reinforcement learning method, DIAR, designed to address challenges posed by out-of-distribution samples and long-horizon planning. Specifically, DIAR first trains a VAE to extract trajectory representations, which are then used as generation targets for training a corresponding diffusion model. DIAR subsequently leverages the representations generated by the diffusion model to support the learning of the value function and policy network. Finally, the authors validate the effectiveness of DIAR through experiments on sparse reward tasks.

Strengths

1. Offline RL problem in sparse reward tasks is significant, and previous methods have struggled to effectively address it. 2. Compared to the baselines, DIAR demonstrates superior overall performance on sparse reward tasks.

Weaknesses

1. The writing in this paper could be improved, as the logic is not always coherent, and some sentences are difficult to understand. For example, “However, offline RL relies on the given dataset, so the learned policy may be inefficient or misdirected if the data is poor quality or biased.” in lines 44-46. 2. The novelty of the method is limited, as both learning a latent diffusion model [1] and implicit Q-learning [2] are existing approaches. 3. The use of the diffusion model lacks clear motivation. The authors need to further explain why it is necessary to input the latent representation $z$ during the IQL stage. 4. The adaptive revaluation approach is not entirely reasonable, as expert policies in many tasks do not satisfy $V(s_{t+1}) > V(s_t)$. For example, this is often the case in finite-horizon tasks with positive reward functions. Consequently, the theoretical analysis in the paper relies on overly strong assumptions that do not generalize well to other tasks. [1] Reasoning with Latent Diffusion in Offline Reinforcement Learning. ICLR, 2024. [2] Offline Reinforcement Learning with Implicit Q-Learning. ICLR, 2022

Questions

See weaknesses.

Rating

3

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Summary

The paper proposes an offline RL algorithm utilizing latent diffusion skill models for temporal abstraction, and Q learning with these skills. During policy rollouts, the learnt value function and temporally abstract world model are used to evaluate whether the currently used skill is optimal. If not, a new skill latent is selected. The method is demonstrated on D4RL tasks.

Strengths

1. The method has good performance on D4RL compared to baselines.

Weaknesses

1. The paper is messily written in my opinion, and it is difficult to parse what the major contribution of the paper is. This seems like primarily an engineering paper, but this is not clearly communicated. 2. The method is a combination of existing offline RL algorithms (primarily LDCQ and IQL), but there is no proper reason given for this particular configuration of components. The only novel addition seems to be the use of the value function for deciding when to stop executing a skill, but this is a simple iterative improvement. 3. Novelty is not strictly necessary, but the additions made here are not well justified at all with no coherent story surrounding it. 4. This is not a direct criticism of the paper, but D4RL has been quite over-optimized in the offline RL community now, small engineering improvements to boost the score in this benchmark does not give any signal to the true value of the method. 5. More general writing criticism, a lot of the paper repeats itself and feels like padding more than informative content. For example, section 4.3 “Theoretical Analysis of DIAR” is very elementary and adds no value.

Questions

1. What is the primary contribution of the paper? Do the authors pitch the paper as a novel offline RL algorithm? 2. Since the authors only evaluate on D4RL, why is there no evaluation of the locomotion tasks (half-cheetah, walker2d, hopper)?

Rating

3

Confidence

4

Soundness

2

Presentation

1

Contribution

1

General Response

Dear Reviewers and Meta Reviewer, Thank you for taking the time to review our submission, "DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation." We deeply appreciate your thoughtful feedback and constructive suggestions, which have greatly contributed to improving the quality of our work. Below, we provide a detailed response to each of your comments. - We have provided a more detailed explanation of our concept, DIAR, and clarified the distinctions between our approach and existing methods. - This study focuses on addressing the challenges of long-horizon tasks with sparse rewards, and our approach was specifically designed to tackle these issues. Through experiments, we demonstrated that our method performs well in environments such as Maze2D, AntMaze, and Kitchen. - We tested Adaptive Revaluation (AR) under scenarios that required certain assumptions, and we are considering extending it to a broader range of applications in the future. - We revised complex sentences in the paper, improving their clarity to make them easier for readers to understand. Should you have any further questions or suggestions, please put your comments on OpenReview. We will address all the raised concerns according to the reviewing policy.