How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimization

Summary

This paper investigates the problem of model shift in model-based Reinforcement Learning (RL). The author proposes a two-stage model learning method called USB-PO (Unified model Shift and model Bias Policy Optimization). The first stage is the same as previous work, updating based on the previous model using model replay buffer. The second stage is based on the previous model and the replay buffer, aiming to simultaneously reduce model shift and model bias after the model first stage update. The paper also provides a theoretical analysis for the proposed method, and the effectiveness of the method is demonstrated in experiments conducted on MuJoCo.

Strengths

1. As far as I know, the method proposed in this paper is novel. 2. The authors provide theoretical analysis and motivation for their method. 3. The ablation study in the paper is rather comprehensive.

Weaknesses

1. What is the basis for lines 133-135? I don't think such a conclusion can be derived from Definition 1 and Theorem 1. 2. What is the relationship between Theorem 2 and 3? Both seem to be proving different bounds for the same term, what is the significance of this? Judging from the appendix, there appears to be a typo in Theorem 3. 3. In the experiment section, the performance curve of the MBRL method CMLO drops much more than what is shown in the original paper. I would like to know the reason for this. 4. The baselines STEVE and SLBO are too outdated, and they can no longer be considered as state-of-the-art (SOTA) baselines for MBRL.

Questions

1. The two-stage model learning will inevitably lead to more time consumption. Can you specify how much the time consumption has increased? 2. Please compare with some recent MBRL methods instead of STEVE and SLBO, such as PDML [1] and ALM [2]. Reference: [1]. Wang et. al. "Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy," in ICML 2023, https://arxiv.org/abs/2207.12141. [2]. Ghugare et. al. "Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective," in ICLR 2023, https://arxiv.org/abs/2209.08466.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. The time efficiency needs to be reported. 2. More baseline comparisons are required. 3. It would be desirable to compare with more benchmarks, such as DMC and Metaworld, not just MuJoCo.

Summary

This paper proposes a new MBRL algorithm which adaptively adjusts the model update by considering the model shift (distance towards previous step update) and the model bias (towards true env model) simultaneously, which avoids model overfitting and shows better empirical performance compared to only considering either of them.

Strengths

- This paper clearly discusses the drawbacks of previous MBRL approaches, either considering model bias only, which gives possible excessive model updates, or considering model shift only, which might be subject to overfitting to the previous model. A natural and neat idea is proposed by unifying the two aspects together to further fine-tune the model, which supported by strong empirical results. - The experiments section is thorough. The ablation study is interesting, which shows the performance of USB-PO is better than removing any of the model bias or shift term. Section 5.2 is also nice, which empirically verifies the assumptions in the method development, by removing the consideration of $\Delta$, as well as the visualization on the optimization objective value to see how the update happens. These experiments facilitate the understanding the method.

Weaknesses

- The biggest weakness of the paper is that it is very poorly written, the idea is simple but the way the paper organizes and writes make it very difficult to convey the main message. For example, Section 4.1 is too general and it is hard to get what it is trying to say. Algorithm 1 does not show any important message, for example, Line 115 says the optimization objective is MLE loss and the phase 2 is for finetuning, which is too high-level and it makes readers confused. - This might be correlated with point 1, the algorithm in Algorithm is not super clear to me. It seems Equation 3 is used to learning $M_2$, but the Algorithm 2 Line 2 says we get $M_2$ from $M_1$ update, this is confusing. Is Line 3 only for finetuning $M_2$ given the previously learned $M_1$, $M_2$? These are the main points of the paper and it is important to make things clear and precise.

Questions

- For Line 195, could the $M_1$ ensembles be also used in the bias term estimate? - The experiment in Figure 6, is it possible to try more lr of the MBPO to showcase the advantage? - Another ablation study is to varying the number of ensembles for the models, and comment on the computational complexity.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

N/A

Summary

The paper considers model based RL (MBRL), focusing on the question of fine-tuning the model while learning the policy (the policy optimization). While prior works treat these two aspects somewhat separately with tuning thresholds, the authors derive a single cost function that incorporates both. The paper presents a lengthy derivation of the cost function and shows through inequalities and assumptions that it is a well grounded approach, and then numerical experiments validate the approach and compare to MBRL methods as well as model free (MFRL) methods.

Strengths

The paper is very clearly written, with good motivation and placement of the results in the state of the art. The idea of combining the model refinement and policy learning is perhaps obvious; what is not obvious is how this can be carried out. The paper does this theoretically, and then carries this through to a useful algorithm. The derivations are clear and the assumptions needed to bring this to a numerical method are laid out. Theorem 2 is nice because it bounds the expected return difference between and 2 models and their respective policies. This should be useful in other learning contexts. The meta-algorithm (Algorithm 1) puts the overall ideas across in a way that doesn’t require the reader to understand the subtleties of the proofs.

Weaknesses

The reviewer doesn't see any significant weaknesses in the paper. Some clarifying questions are listed below.

Questions

Perhaps it is worth saying more about the assumption after eqn (9), regarding policy shift being small wrt the model bias? When would this not be true? Deriving eqn (10) relies on a Gaussian assumption. Could you say why this is generally valid, or when it might not be so valid? Similarly in eqn (11) the Wasserstein distance is easily calculated for the Gaussian case. So here the Gaussian assumption is carrying through? After Algorithm 2 statement, “following [18]”: What is the elite set of B models? And you choose one of these with uniform probability? How large is B typically? The “pull-back” interpretation is interesting. That is what you call the no-update case? In the appendix, Table 2, top 3 cases, the model-free works well, albeit with a lot of steps. Can you compare the overall complexity of the MFRL and USB-PO for these cases, accounting for cost per step of each? Also, is there a hybrid MFRL-MBRL approach?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

There are no negative social issues.

Summary

This paper studies the learning of the dynamics model in model-based reinforcement learning. The authors propose a novel method USB-PO that has provable properties.

Strengths

1. The paper studies an important problem in model-based RL, namely the model shift and model bias balance during model update without heavy dependence on the threshold and a lack of adaptability. 2. The algorithm is straightforward and the theoretical results also match the expectation. 3. The paper is clearly written and easy to follow.

Weaknesses

1. In the pseudocode Algorithm 1, what is the difference between $M$ and $M_2$? Are they supposed to be the same? 2. Can the authors elaborate the MBRL algorithms that have heavy dependences on the threshold? 3. Can phase 1 and phase 2 be integrated to a single joint optimization objective? It would be easier to implement if the algorithm is a trust-region style algorithm that maintains a single model and regulates the updated and the last-iteration model. 4. There are also previous MBRL works [1, 2, 3] that have a dual update procedure, in a very similar way to how the proposed method regulates the model updates. I recommend the authors to also discuss these related works in a later version of the manuscript. [1] Zhang et al. Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning.\ [2] Sun et al. Dual Policy Iteration.\ [3] Levine et al. Learning neural network policies with guided policy search under unknown dynamics.

Questions

See the weakness section above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See the weakness section above.

Reply

The reviewer appreciates the authors replies to the questions posed. Also, the reviewer appreciates the specific changes put forward, including combining algorithm 1 and algorithm 2 statements, and incorporating the additional related work. I am satisfied with my review rating.