The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning

Summary

This paper identifies an overlooked problem in offline model-based reinforcement learning (RL). Offline model-based RL approaches commonly perform online RL inside a learned dynamics model and due to model errors, they generally use truncated $k$ step rollouts, instead of full trajectories (episodes). In this paper, the authors replace the learned dynamics model with the true simulator and show that offline model-based RL approaches fail even when they have access to the true dynamics. This is due to states which can only be reached at the end of the $k$ step rollouts (which the authors coin "edge-of-reach" states) and thus, only appear in the target of the Q update. As such, the Q-function is never updated at these "edge-of-reach" states so any overestimation is gradually propagated over the entire state-action space. The authors provide a nice diagram which provides intuition and shows the effect in a simple environment. They then propose an algorithm which overcomes the so-called "edge-of-reach" problem by using an ensemble of Q functions. They demonstrate good performance in D4RL and V-D4RL.

Strengths

I think this is a good paper that should be accepted. It identifies an overlooked problem in offline model-based RL which seems like it will be an important contribution to the community. The authors have also done a nice job providing intuition for the problem via Figure 2 and the example in Section 4. Based on the identified problem, the authors also propose a novel method which is simple yet effective. They have also demonstrated the effectiveness of their approach on data sets with both proprioceptive and image-based observations.

Weaknesses

I have no major weaknesses. Nevertheless, I do have some comments which I provide to help improve the paper. The main weakness of this paper is that it looks messy and overcrowded. There are multiple reasons for this which I will now detail. The use of italics throughout the text makes the paper look very messy. Sometimes the italics refer to questions whilst other times they just seem to be random words. I would advise the authors to remove all italics. There are too many subsections. For example, in Section 6 there are 5 subsections and each subsection is a single paragraph. I would advise the authors to replace \subsection with \paragraph. Similarly, on Lines 283/293/303 the authors use unnumbered subsections. This looks very messy. I suggest the authors replace this with \paragraph so that the text starts immediately after the bold text and does not have a line break before. There are a lot of incorrect uses of colons, which makes the writing quality feel poor. Please remove the colons on lines 12/90/114/122/137/144/154/176/195/273/305. The authors do not mention how the Q-function is initialized. I think it is very important to include all of this information. For example, if we initialize the Q-function to be zero at all inputs wouldn't that help prevent overestimation bias from "edge-of-reach" states? ## Minor comments and typos - Line 89 - $\\}\_{i=1,\ldots,N}$ should be $\\}\_{i=1}^{N}$ - Line 122 - incorrect use of semicolon. This should just be "and". - Line 137 - the full stop should be outside the quote. - Line 167 - $0\ldots k-1$ should be $0,\ldots,k-1$ - Line 170 - Why is the last sentence of the definition not italicised? - Line 249 - the comma should be outside the quote. - Line 306/308/309/31 - the dataset subscript looks wrong. - Sections shouldn't lead straight into subsections, e.g. 6 into 6.1. - Add a few sentences explaining what you are introducing in this section.

Questions

- What is on the x-axis of Figure 5? What environment is this from? This is not clear from the text or caption.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are adequately discussed.

Summary

The paper presents a novel analysis of the challenges faced by offline model-based reinforcement learning (RL) when the dynamics model becomes increasingly accurate. The central thesis is that existing methods fail under true, error-free dynamics due to a previously unaddressed issue termed the "edge-of-reach" problem. The authors provide a thorough investigation, theoretical analysis, and empirical evidence to support their claims. They propose Reach-Aware Value Learning (RAVL), a method that directly addresses this problem, demonstrating robust performance across benchmarks.

Strengths

1. The paper is well-organized, with a clear abstract, introduction, and conclusion that effectively summarize the contributions and findings. 2. The authors have provided open-source code and detailed hyperparameter settings, facilitating the reproduction of the experiments. 3. Rational experiments on standard benchmarks and a simple environment validate the "edge-of-reach" hypothesis and demonstrate the effectiveness of RAVL.

Weaknesses

The methodology contribution of this paper is limited. While I assume that the "edge-of-reach" problem is significant to the current offline RL literature, the proposed method by the authors fails to demonstrate obvious superiority compared with the SOTA offline model-free or model-based methods (see Table 2 of this paper). One reason, I guess, is that the D4RL benchmark is too simple to underscore the superiority of addressing the "edge-of-reach" problem. On the other hand, I expect that authors can validate their method on more complex and challenging benchmarks and provide more convincing results. I am willing to reconsider my score after seeing this.

Questions

1. I wonder how existing uncertainty-penalty-based offline model-based RL methods perform if I gradually increase the length of rollouts in the estimated dynamics model along with the improved accuracy of the model (a very intuitive trick). 2. Does the "edge-of-reach" hypothesis exist in many online RL scenarios? For example, researchers usually early stop the trajectory according to a pre-defined maximal length of the trajectory in the Gym Mujoco benchmark.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations of their work properly.

Summary

This paper identifies and investigates a previously neglected issue in offline model-based RL called the "edge-of-reach problem", which is due to the truncated rollouts used in model-based RL to mitigate the compounding errors of the dynamics model. The authors proposed Reach-Aware Value Learning (RAVL) to address the edge-of-reach problem using value pessimism. They show RAVL's strong performance on the D4RL and V-D4RL benchmarks.

Strengths

1. The edge-of-reach problem is interesting and previously overlooked in offline model-based RL. This paper is the first to formally investigate the problem. 2. Comprehensive, well-designed experiments to support the claims.

Weaknesses

1. The proposed method doesn't outperform existing baselines according to the reported results on D4RL. Although it's claimed that "RAVL can be easily combined with appropriate dynamics uncertainty penalization", it's not supported by any empirical evidences. Hence, it's unclear how well RAVL work in general when combined with other model-based methods with design choices orthogonal to value pessimism. 2. I found the paper a bit hard to read due to the formatting, e.g. too many long italics phrases.

Questions

1. Why does RAVL not benefit from reduced dynamics errors according to table 1? 2. It would be interesting to show how RAVL's performance change by varying the accuracy of the dynamics model, similar to figure 1.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

The paper investigates a phenomena in model-based offline RL that most existing algorithms fail when they are given the perfect model of the environment. Through multiple experiments, the paper argues the failure is due to the wrong estimated values at the states in the end of model rollouts. These states are called edge of reach and are used for targets of Q values but are never trained themselves. The paper continues to propose a fix in the form of pessimistic target value estimate using ensemble Q networks.

Strengths

- The paper does an admirable job with empirical investigations of the root cause of the observed phenomenon. I find the perfect model experiments, the simple environment example, and the patch oracle fix to be a very convincing set of evidences for the paper's claim. The paper sets a good example for this kind of empirical research. - The introduced fix is practical and simple, and I believe might improve other existing model-based offline RL algorithms as well - The empirical results obtained by the - I find the paper well written and free of typos.

Weaknesses

**1)** The paper could benefit from more investigations on the necessity of the _detection_ of an edge-of-reach state. The current proposed value learning only uses the pessimistic Q target when it decides that the state is an edge-of-reach state. **Q1)** I wonder whether if this distinction is necessary and if we could always use the pessimistic value and get the similar results. I believe this could be doable because in states with low variance, the pessimistic value will not defer from the normal one anyway. Potentially, this removes a hyperparameter which sets the threshold for the choice of value target. It also reduces the number of ensemble networks needed for the algorithm. **2)** The paper could better position itself in the literature. The pessimistic value target is a ubiquitous trick in RL to overcome value estimation. See for example TD3 [1] paper for actor critic methods. The issue where models query Q networks in untrained states has also been pointed out in MBRL research. See for example [2]. **3)** It probably is out of the scope of the paper, but I wonder how much of this issue is shared with online MBRL algorithms. They also usually only create rollouts of a limited length. A discussion on this matter will be welcome. **4)** It could be a personal matter, but the paper has too many future referencing. For example lines 160, 220, 231, 253, 260, 261. I find these confusing as a reader. Jumping to read the future section while reading is not ideal or even possible. Also when the future section is being read, the connection made in the past sections to the current one is probably forgotten. I recommend reducing these references and only make such connections when necessary. A general comment that more discussions about a topic will appear in the future could be enough for the reader without asking them to jump. [1] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018. [2] Jafferjee, Taher, et al. "Hallucinating value: A pitfall of dyna-style planning with imperfect environment models." arXiv preprint arXiv:2006.04363 (2020).

Questions

See Q1, and weakness 3 above.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

I find the limitation discussions sufficient.

Please respond to author response if you haven't

Dear Reviewers, Before the author-reviewer discussion period ends, please make sure that you have read the author responses, acknowledge that your review reflects them, and engage with the authors if necessary. Thank you!

Rebuttal Acknowledgement

I thank the authors for the rebuttal. I have read all the comments and I am maintaining my score.