Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning

Summary

This paper introduces Collaborative World Models (CoWorld), a novel approach to offline reinforcement learning (RL) with visual inputs. CoWorld conceptualizes offline RL as an online-to-offline transfer learning problem, utilizing an auxiliary online simulator to address the challenges of overfitting in representation learning and value function overestimation. The method involves training separate source and target world models and actor-critic agents, aligning their latent spaces, and using a three-stage learning procedure to improve performance on the target task.

Strengths

- The idea is novel: CoWorld presents an interesting perspective on offline RL by framing it as an online-to-offline transfer learning problem. - The problem is important: The method aims to tackle both overfitting in representation learning and the trade-off between value function overestimation and over-conservatism. - Ablation studies and Analysis: The paper includes comprehensive ablation studies that validate the contribution of each component in the CoWorld framework.

Weaknesses

1. Dependence on auxiliary online environments: CoWorld heavily relies on the availability of an online environment in a sufficiently similar domain, which may limit its applicability in real-world scenarios. 2. Unclear rationale for separate world models: The decision to maintain separate source and target world models, rather than a single jointly trained model, is not adequately explained or supported. 3. Marginal performance improvement: The performance gains of CoWorld over simpler baselines (e.g., fine-tuning) are sometimes marginal and highly sensitive to hyperparameters. 4. Computational complexity: The alternation between online and offline agents until convergence may be computationally expensive and potentially unstable. 5. Strong assumptions: The requirement of a high-quality simulator for a similar source domain contradicts the motivation for offline RL, which assumes online data collection is costly or dangerous. 6. Insufficient clarity in the presentation: The necessity and function of individual components—such as world model learning, state alignment, and reward alignment—are not clearly delineated, leading to potential confusion.

Questions

1. How dependent is CoWorld's performance on the quality and similarity of the online simulator to the target task? 2. Have you considered training on multiple source domains to improve robustness and generalization? 3. How does the computational complexity of CoWorld compare to baseline methods in terms of training time and resource requirements? 4. Can you provide more insight into the process of selecting an appropriate source domain, and how this might be done without leaking test-time information? 5. How would CoWorld perform if the source and target domains have significantly different observation spaces?

Rating

5

Confidence

3

Soundness

3

Presentation

1

Contribution

2

Limitations

Yes, the authors have discussed the limitations in the submission.

Summary

This paper studies model-based offline reinforcement learning with visual observations. This work focuses on the overfitting problem in state representation and the overestimation bias for the value function and introduces the environment model from related tasks to present an offline-online-offline solution. The proposed solution contains state alignment, reward alignment, and value alignment. The empirical studies conduct the experiments with transfer across heterogeneous tasks.

Strengths

1. The proposed setting is very interesting to me. The author provides a new perspective on reinforcement learning cross-task transfer from the angle of transfer learning, which is challenging. 2. The proposed solution is reasonable. When online interaction in the current task is unavailable, leveraging online interaction from related tasks is similar to traditional transfer learning. Although the offline-online-offline process appears somewhat complex, it indeed provides an idea that may promote related research.

Weaknesses

1. The provided results have significant standard deviations. I am not sure if the authors taking 10 episodes over 3 seeds provided a sufficient evaluation. This issue can be referred to: [Deep reinforcement learning at the edge of the statistical precipice. NeurIPS’21]. 2. In fact, I highly appreciate the contributions made by the authors, if the statistical results of the experiment are reliable (see Weakness 1). This work successfully achieved transfer in multiple environments, which is rare for me, and makes a significant contribution. But where does the success of the transfer come from? The success of state alignment is relatively easy to understand, similar to traditional visual domain adaptation. But what does the success of reward alignment come from? From information like "objects should be picked" in the reward signal? This seems to impose relatively strict limitations on the relevance of the task, which is lacking in discussion in the current version. 3. Similarly, what are the conditions for mitigating value overestimation? Figure 2 is well visualized, but it seems to implicitly assume that the source critic and the ideal target critic perform similarly on the broad support. If based on such an assumption, cross-environment transfer does not seem so difficult. I still affirm the contribution of this work, but can the authors provide a formal analysis of this and determine the applicable conditions for the proposed approach?

Questions

1. How to ensure that source critic is beneficial to the target task, and is it possible to have a negative transfer when facing conflicting tasks? 2. See the weakness 3. --- After rebuttal, my concerns have been addressed well. I decide to raise my score to 6.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have provided a discussion about the limitations.

Summary

This paper studies CoWorld, a model-based RL method that learns the policy through an offline-online-offline framework. Specifically, CoWorld assumes that, in addition to the offline dataset, there is another (simulated) environment for online interaction. Through the offline dataset and interactive environment, CoWorld constructed two world models to train the separate policies. Next, this paper introduces three components to enhance the policy performance, including offline-to-online state alignment, online-to-offline reward alignment, and min-max value constraint. The proposed CoWorld and several baselines are evaluated on Meta-World, RoboDesk, and DMC benchmarks. From the results presented, this paper suggests that CoWorld outperforms existing RL methods by large margins.

Strengths

- This paper proposes an interesting method for training an RL policy by co-learning from both the offline dataset and the interactive environment. - This work presents a reasonable number of ablation studies and experiments to verify the proposed method. These experiments cover various directions, including environmental settings, component contributions, and hyperparameter sensitivity. - This paper is generally well-written and easy to follow. The depicted figures and tables are well-articulated.

Weaknesses

- While I agree the idea of ​​CoWorld is interesting, I feel that it may not be easy to find suitable practical applications. After all, a low-cost, high-efficiency interactive environment is itself a strong assumption. It would be more convincing if the author could describe practical situations where the proposed method is applicable. - Related to the first point, CoWorld learns the separate world models from both the offline dataset and the online environment. I'm curious about how significant the difference between these two sources can be. The setting used in this paper is only a slight difference (4 vs. 5 in the action space, or scale in reward space). But when the two resources are significantly different, can CoWorld still work? For example, (1) one environment with dense reward and another one with sparse reward (not just scale), or (2) the observation and action spaces are obviously different from the two sources. - From Table 2, I am concerned about the claim that the proposed CoWorld significantly outperforms other baselines because (1) in several cases, the CoWorld's performance overlaps with other methods when considering its standard deviations, and (2) the results are obtained from only three random seeds. I encourage the authors to conduct more experimental runs and/or provide statistical analysis to eliminate this concern. - A minor point: while this paper is generally well-written, I found some typos that should be easily fixed: 1. The notation T represents both the target environment and the maximum number of time steps. (line 14 in Algorithm 1) 2. Target/Offline (**S**) $\rightarrow$ should be T? (Table 4, page 13) 3. ... on DMC **Meidum**-Expert Dataset $\rightarrow$ Medium (line 481, page 14) 4. ... as illustrated in **Table 1** $\rightarrow$ should be Table 6? (line 516, page 17) Based on the observed strengths and weaknesses, I will set my initial recommendation as "Borderline accept". I will be happy to raise my rating if the authors appropriately address my concerns or point out my misunderstandings.

Questions

Please consider addressing my concerns and questions in Weaknesses; I will list some key ones: 1. In what practical scenarios is the proposed CoWorld method applicable? Can you provide more examples and details? 2. How does CoWorld operate when the two world model architectures have greater differences? Can it still effectively transfer information between these two resources? 3. How to prove that CoWorld is really significantly better than other methods? Can you provide the results of more experimental runs or statistical analysis?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

A major limitation is that the proposed CoWorld method is applicable to only a limited number of practical application scenarios. Other limitations are adequately discussed in the Limitation Section.

Summary

This work proposes a model-based algorithm for offline visual reinforcement learning using domain transfer learning. The proposed method demonstrates superior performance over existing approaches in several baselines.

Strengths

S1. The paper presents the necessity of proposed modules in the algorithms through ablations and validation in multiple setups. S2. The proposed method outperforms baselines in different benchmarks.

Weaknesses

W1. The cost of training time is higher in comparison to baselines.

Questions

Q1. How does the performance of DV2 Finetune, DV2 Finetune+EWC, and CoWorld look when the x-axis is training wall-clock time instead of the number of training iterations?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have addressed the limitations.