Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning
Current approaches to model-based offline reinforcement learning often incorporate uncertainty-based reward penalization to address the distributional shift problem. These approaches, commonly known as pessimistic value iteration, use Monte Carlo sampling to estimate the Bellman target to perform temporal difference-based policy evaluation. We find out that the randomness caused by this sampling step significantly delays convergence. We present a theoretical result demonstrating the strong dependency of suboptimality on the number of Monte Carlo samples taken per Bellman target calculation. Our main contribution is a deterministic approximation to the Bellman target that uses progressive moment matching, a method developed originally for deterministic variational inference. The resulting algorithm, which we call Moment Matching Offline Model-Based Policy Optimization (MOMBO), propagates the uncertainty of the next state through a nonlinear Q-network in a deterministic fashion by approximating the distributions of hidden layer activations by a normal distribution. We show that it is possible to provide tighter guarantees for the suboptimality of MOMBO than the existing Monte Carlo sampling approaches. We also observe MOMBO to converge faster than these approaches in a large set of benchmark tasks.
Paper
Similar papers
Peer review
Summary
To address the high variance issue of model-based offline RL methods, which rely on sampling-based uncertainty estimation, this manuscript proposes MOMBO, a model-based policy optimization method based on moment matching. MOMBO learns a Q-function using moment matching and deterministically propagates uncertainties through the Q-function. Theoretically, using moment matching has tighter upper bounds on value approximation than using Monte Carlo sampling. The authors evaluate MOMBO’s performance across various environments.
Strengths
1. MOMBO is a more stable and sample-efficient approach. 2. There is a tight upper bound on value approximation errors using moment matching compared to Monte Carlo sampling.
Weaknesses
1.There is a clear necessity for the authors to enhance their writing proficiency. 2.The manuscript lists many contributions, which obscures its central thesis. The primary contribution is learning a Q-function using moment matching. However, this approach needs to strike me as sufficiently innovative. 3.How can the alignment of moments from one distribution with those from another ensure the approximation or equivalence of the two? 4.OOD data seems more penalized by high variance, whereas IND data does not generally have high variance. Moreover, there is a need for diversification to improve the variance of Ensemble RL (cf. EDAC, NeurIPS 2021). The moment matching proposed by the authors is similar to transforming uncertainty/variance. Furthermore, the experimental results do not prove the method's validity due to the lack of comparison with the latest SOTA method. 5.Section 3, entitled "Limitations of SOTA model-based offline RL approaches," does not quite capture the issue's essence. Ideally, it should be reframed to reflect a specific shortcoming within a methodology, technique, or theoretical framework. Moreover, the term "SOTA" is inherently time-bound, making it an imprudent choice for a section title, as it may quickly become outdated. 6.The authors follow \mu - \beta \cdot \sigma, where \beta is the weight parameter seems to differ little from most offline RL methods based on uncertainty estimation. 7.The core of the proposed method is based on moment matching; however, the citations to moment matching in the manuscript are too old. Is there any recent work? 8.Only some tests on mujoco environments are insufficient. Isn't it common to perform experimental validation on environments like Antmaze, Adroit, etc.? How do the authors explain this? I suspect that the proposed method will not work in these neglected environments.
Questions
See weaknesses.
Rating
3
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
None.
Further Clarifications
1. We promise once again to implement all of your suggestions, which comprise: 1. Shortening the sentences that are hard to read 2. Unifying the allocation of terminology, sticking to their usage in the literature 3. Correcting wrongly constructed sentences 4. Rephrasing sentences written in passive voice in active voice in the most understandable way 5. Correcting improperly used writing conventions throughout the text. We do think they are extremely important to improve the quality of our work. Their implementation does require a thorough and careful pass over the text, which is a tedious task and we are ambitious to fulfill it following your guidance. We kindly express our opinion that all these corrections are typically handled during the preparation of a camera-ready versions of conference papers. We encounter corresponding period in journal publication processes as **accept with minor revision**. 2. Here are our answers to your questions: We would like to first make sure that we agree on the terminology. We study the **variance** of *the Monte Carlo estimators used in the penalty term calculation of model-based offline RL algorithms*. Using an estimator is inevitable while calculating a penalty term based on the Bellman operator. In mathematical terms, there is uncertainty on the next state $s’$ stemming from the error in the learned environment dynamics. This uncertainty needs to be propagated through the state-action value function $Q(s’,a’)$ which appears in the Bellman target calculation. This operation is analytically intractable if $Q$ is a neural network. > Is high variance the only issue with ineffective Monte Carlo sampling? We are sure it is not the only issue. Our claim is that it is one of the many important issues as it plays a significant role in training efficiency. We demonstrate why it is the case by developing concentration inequalities for the Monte Carlo estimator and our moment matching based alternative. > Given that the authors have only analyzed the variance issue, it naturally raises the following concern:” Does the variance problem only arise with Monte Carlo sampling? As we pose the problem in the first place as the estimator variance of a Monte Carlo estimate, the answer is yes. Reducing estimator variance is major issue in a wide range of problems in statistics. It is handled using methods such as introducing control variates or Rao-Blackwellization. See for instance: Lemieux (2017). The state-of-the-art approach to approximate the uncertainty of the state-action value of the next state is to use Monte Carlo integration, which is represented in our paper with the MOBILE algorithm (Sun et al., 2023). Sampling-free approaches do exist, but only for penalty terms that are computed directly on the estimated error of the model dynamics. We represent this alternative in our paper by the MOPO algorithm (Yu et al., 2021). However, as proven in Theorem 4.3 of the MOBILE paper, this option over-penalizes the reward and causes suboptimal results. The tightness experiment we reported in our global response demonstrates that MOPO applies much higher penalty than MOBILE as this theorem suggests, and our MOMBO applies even less penalty, hence brings about much faster training. > Compared to other means of addressing the variance issue, what is the motivation for using a sampling-free approach in this paper There does not exist an alternative way to address this issue yet. Our key novelty is that we are the first to detect its existence and prominence for offline reinforcement learning and provide a solution to mitigate it. ----- Lemieux, C., _Control Variates_ (Wiley StatsRef: Statistics Reference OnlineL 1-8 2017) Yu et al., _COMBO: Conservative offline model-based policy optimization_ (NeurIPS 2021) Sun et al., _Model-bellman inconsistency for model-based offline reinforcement learning_ (ICML 2023)
Summary
This paper proposes a new uncertainty estimation method for model-based offline reinforcement learning, which uses moment matching to deterministically propagate uncertainties through the Q-function, rather than relying on sampling-based uncertainty estimation. The resulting model, Moment Matching Offline Model-Based Policy Optimization (MOMBO), significantly accelerates training compared to its sampling-based counterparts while maintaining performance.
Strengths
- The paper is clearly written, allowing readers to follow the main arguments. - It provides a detailed theoretical analysis to demonstrate why moment matching is better than Monte Carlo sampling.
Weaknesses
- The novelty of this method seems relatively weak. It appears to be a minor modification of the uncertainty quantification method used in MOBILE [1]. Additionally, as noted by the authors in the limitations section, this method is also limited to certain activation functions. I want to clarify that I do not think minor modifications lack value or novelty. Instead, the novelty of a method should be evaluated along with its empirical performance. However, the experiments in this paper do not show a strong improvement over the baseline (MOBILE). - The advantage claimed by the authors is sample efficiency. Firstly, using “sample efficiency” here is incorrect. This criterion is used to evaluate an online RL method. In offline RL, we train an algorithm on a fixed dataset, so “convergence speed” would be more appropriate. Furthermore, this criterion is not critical in an offline setting, where we are more concerned with the asymptotic performance. - According to PEVI [2], the key factor for an uncertainty-based offline RL method is how tightly it can estimate the Bellman error, which directly determines the performance of the derived policy. The authors do not provide direct evidence (theoretical or empirical) to show that MOMBO can provide a tighter estimation of the Bellman error. [1] Sun et al. "Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning" (ICML 2023) [2] Jin et al. "Is Pessimism Provably Efficient for Offline RL?" (ICML 2021)
Questions
To summarize the main points/questions raised in the weaknesses section: - Could the authors evaluate their method on more challenging tasks to show that it can achieve better asymptotic performance compared to MOBILE? - Could the authors provide evidence to demonstrate that MOMBO can provide a more correlated/tighter estimation of the ideal uncertainty, i.e., the Bellman error? I‘m willing to raise my score if all these concerns can be addressed by the authors.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have thoroughly discussed the limitations of their method in the paper.
Summary
** I am unfamiliar with the methods/related works in this paper.** This paper proposed a model-based method for offline RL, based on moment matching. Improved numerical results are presented.
Strengths
The method of moment matching is quite novel, which aims to improve the accuracy of the first two moment estimation.
Weaknesses
I am quite confused about Sec 4. Why the Gaussian RVs are related? I feel the presentation of the paper can be improved.
Questions
The authors mention that the uncertainty-based approach can be overly pessimistic, but some recent paper [1,2] are showing that the uncertainty-based methods achieve the minimax optimal sample complexity, under the tabular setting. Can you explain the reason of such a statement? [1]Li, Gen, et al. "Settling the sample complexity of model-based offline reinforcement learning." The Annals of Statistics 52.1 (2024): 233-260. [2]Wang, Yue, Jinjun Xiong, and Shaofeng Zou. "Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach."
Rating
5
Confidence
2
Soundness
3
Presentation
2
Contribution
3
Limitations
n/a
Summary
This work addresses the issue of sampling for uncertainty propagation that is the standard practice in offline RL and identifies the high variance of sampling-based estimates as an obstacle to better performance of uncertainty-aware offline RL methods. As an alternative, the authors propose propagating uncertainty with moment matching, which is a deterministic procedure. The method involves propagating the uncertainty of state transitions into the value function where the value function is parameterized with a ReLU network. The method, coined MOMBO, is further theoretically supported with tight bounds on value error approximation, and experiments on the standard Mujoco benchmarks indicate that MOMBO is on par or exceed existing approaches in offline RL.
Strengths
- Uncertainty propagation in RL is a relevant problem to which there is not yet a definitive solution, mainly due to the interplay between uncertainty in state transition and uncertainty in value estimation. This work addresses this issue and proposes a method which is well motivated and theoretically supported. While the method itself does not touch upon the method of uncertainty quantification, rather only the propagation of uncertainty, I believe further investigation into the effect of various UQ methods when used with this propagation could be interesting. - Empirical results are fairly strong - Overall the paper is well written
Weaknesses
While the authors conjecture that sampling based approaches to uncertainty propagation has inherent flaws, it would be nice to see more evidence, e.g. empirical evidence that digs into this phenomenon. As such, the proposed method does not seem like a fundamentally novel approach, rather an improvement in estimation of a given framework. Still, the suggestion of moment matching as a scalable and efficient method of propagating uncertainty seems useful to the community and this paper would fit well alongside existing work.
Questions
N/A
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A
Discussion Period
Hi all! Just your friendly area chair checking in. First, thanks to the reviewers for your work so far on this paper. The discussion period has begun, which includes both reviewers and authors. This is our opportunity to clear up misunderstandings, get answers to questions, and generally gather enough information to make an informed decision grounded in the acceptance criteria. It seems there is general agreement that the MOMBO approach is novel and interesting. The main critiques raised seem to concern whether there is clear evidence of meaningful empirical advantages of MOMBO over existing methods, and some clarity issues in the manuscript. Does that seem like a fair high-level summary? Reviewers: please carefully read the other reviews and the authors' responses and let us know what follow-up questions you have and to what extent your evaluation might change as a result of the additional context. Please especially raise any points of disagreement with other reviewers or the authors, as these are opportunities for us to clarify misunderstandings and detect misconceptions.
Thanks for the author's response, which has resolved some of my initial concerns. I have thoroughly read the comments from other reviewers and the author's replies, including the general response. Below, I provide explanations for the points raised by the author and list the concerns that are still unclear to me: 1. Improvement Suggestions for Writing: Here are the areas I believe would benefit from refinement in writing: - Sentences are too long and hard to read: Line 20-23 "Applying traditional …"， line 24-27 "Model-based …"， line 171-174 "As the input …" - Conflation of expressions: distributional shift & distribution shift, behavioral policy & behavior policy. Refer to literatures MOBILE and COMBO. - Sentence Construction: On lines 89-104, the frequent use of "when" and consecutive "while" at the beginning of sentences, coupled with a near absence of connecting words, raises concerns about the flow and coherence of the writing. - Passive Voice: Upon reviewing lines 121-125, it has been observed that the text relies heavily on the passive voice. While the passive voice can be appropriate in certain contexts, an overreliance on it can obscure the clarity and directness of the writing. - Writing Conventions: The placement of "E.g.," (Line 146) and "I.e.," (Lines 174, 226) at the beginning of sentences is not in accordance with standard academic writing practices. While these abbreviations are commonly used to provide examples or clarify meanings, their position at the sentence's onset can disrupt the flow and formality expected in scholarly articles. Improvement of Pronoun Usage: The manuscript exhibits multiple instances of ambiguous or incorrect pronoun usage, which can lead to confusion for the reader (as seen on line 135; a comprehensive list is not provided here for the sake of brevity). - Refinement of Expressions for Scholarly Rigor: The manuscript frequently employs expressions that may not meet the standards of precision expected in academic writing, such as "via" (the specific instances are not exhaustively listed here). In summary, the numerous writing issues are a significant shortcoming of this paper. The abovementioned concerns may only encompass some areas that require attention; thus, the authors must invest further effort into meticulously reviewing the entire manuscript. 2. Clarification and Enhancement of Contribution Statement: The conciseness of the stated contributions needs to be improved. The author's response appears somewhat superficial and has not yet captured the essence of the contributions. If I understand correctly, this paper's core contribution lies in employing a moment-matching method to address the variance issues arising from Monte Carlo sampling. However, the authors have not adequately reflected these critical elements, especially considering using the term "bottleneck" as a substitute. 3. Clarification of Research Motivation: The motivation behind the study is somewhat ambiguous. If I have understood correctly, the paper's logic is as follows: Monte Carlo sampling leads to variance issues, which in turn prompts the proposal of a sampling-free method. It seems the authors aim to improve upon Monte Carlo sampling. However, I have concerns: - Is high variance the only issue with ineffective Monte Carlo sampling? Given that the authors have only analyzed the variance issue, it naturally raises the following concern: - Does the variance problem only arise with Monte Carlo sampling? - Compared to other means of addressing the variance issue, what is the motivation for using a sampling-free approach in this paper? In summary, the current manuscript's analysis of motivation is incomplete. I have not grasped the authors' central focus, which is why solving the variance problem necessitates using a sampling-free approach.
4. Further Experimental Validation and Concerns: I appreciate the additional experiments provided by the authors. However, even with the selected tasks, I fail to discern the advantages of MOMBO. The performance of the proposed method does not surpass MOBILE; the only advantage seems to be a slightly smaller standard deviation. However, the high standard deviation in the pen-cloned scenario contradicts the motivation of this paper. Moreover, the authors have corrected the expression from sample efficiency to convergence rate. However, in Figure 1 of the manuscript, it appears that only the halfcheetah-random task shows a faster convergence, with no significant advantage in the convergence rate for the other three tasks when compared to the high performance achieved by MOBILE. In the halfcheetah-medium-replay task, although it starts converging faster, it experiences significant fluctuations later, reflecting instability in the training process. Additionally, the standard deviation in the appendix's Figure 4 for both the halfcheetah-medium-replay and walker2d-medium-replay tasks is also significant. In summary, I believe there is room for improvement in this paper's experimental validation, including, but not limited to, a broader comparison of experimental tasks, verification of method effectiveness (for example, modifying linear or ReLU layers to construct ablation studies), and the reliability of statistical results. These concerns also prevent me from confidently assigning a high score.
Clarifications about Experiments and Results
> 4. Further Experimental Validation and Concerns: We would like to reiterate our reply to Reviewer BmgH about the interpretation of our key experimental findings. In a realistic offline RL scenario, it is not possible to measure the evaluation-time performance, hence not trivial to decide how long a model should be trained. Hence, we argue that the steepness of the learning curve is a crucial performance criterion, and the area under the learning curve is an appropriate means to quantify its steepness. As visible in the right-most column of Table 1, MOMBO reaches a significantly steeper learning curve than MOBILE, which is what we would expect from an offline RL algorithm that calculates its reward penalty with less approximation error. The normalized reward results, in turn, serve as a sanity check. They demonstrate that our scheme maintains the original MOBILE performance when trained long enough. Notably, we expect an algorithm with a higher AUC score to deliver better results for a fixed amount of computational time or training data. We would also like to bring the uncertainty quantification results we reported in the global response to your attention. These results provide direct evidence about why moment matching would address the sampling error more effectively. > Figure 1 of the manuscript, it appears that only the halfcheetah-random task shows a faster convergence, with no significant advantage in the convergence rate for the other three tasks when compared to the high performance achieved by MOBILE. In the halfcheetah-medium-replay task, although it starts converging faster, it experiences significant fluctuations later, reflecting instability in the training process. We believe there is a misunderstanding here about the implications of the results we presented. It is true that MOMBO fluctuates after 1 million steps during halfcheetah-medium-replay training a little more than MOBILE. However, this fluctuation is still within the one standard deviation distance of MOBILE's mean performance during the far majority of the training process. The eventual AUC scores of the two models are nearly identical even with similar standard deviations across experiment repetitions: 68.7 ± 3.9 for MOMBO and 67.9 ± 2.8 for MOBILE. We would like to remind that halfcheetah-medium-replay is among the few environments where MOMBO fluctuates more than MOBILE. In the majority of the tasks, MOMBO delivers clearly higher AUC than MOBILE, providing strong evidence of its improved training stability and supporting the central claim of our work. > Additionally, the standard deviation in the appendix's Figure 4 for both the halfcheetah-medium-replay and walker2d-medium-replay tasks is also significant. As reported in Table 1, MOMBO delivers smaller standard deviations across experiment repetitions than other models in six of the twelve environments with respect to final reward, while MOBILE does the same for only four. However, one can see MOMBO's advantage over MOBILE more directly from the AUC scores, where it outperforms MOBILE in all environments but one. The legends of Figures 2, 3, and 4 convey the same information as Table 1 for individual environments. You can see in the final and AUC scores presented in the Figure 4 legends that MOMBO's variance across experiment repetitions is comparable to MOBILE, as expected. > Adroit experiments We reported new experiments only on the two Adroit environments for which the official GitHub source of the MOBILE paper provides sufficient details regarding the configuration. We did not select the tasks based on other criteria.
Hi reviewer igqb, Thanks so much for your lively engagement with the authors so far. We are about to lose access to them in the discussion period (August 13, 11:59pm AoE). I'm hoping that you might be able to weigh in one more time regarding your concerns about the comparison between MOBILE and MOMBO. The authors have made several points in their most recent comment. They argue that the key improvements are learning with fewer training steps and improved uncertainty estimates (see the PDF from the rebuttal); they have expanded upon the halfcheetah results; and they have clarified how the Adroit environments were selected. Could you take a look and let us know the extent to which your concerns have been alleviated by these responses? Thanks!
1. Could you please clarify why only minor writing changes are considered? If there is no substantial evidence to counter the multiple shortcomings I have highlighted, then my concerns remain unresolved. 2. The authors have yet to adequately address my concerns regarding the motivation behind the research. I kindly request that you respond directly to the three questions I posed in the Clarification of Research Motivation. 3. Based on my understanding, a lower standard deviation typically indicates a faster rate of convergence. However, I am concerned that from Figures 2 through 5, the proposed method demonstrates a fast convergence rate in only a few tasks in 4/12 (Figure 1: HalfCheetah, Figure 2: HalfCheetah, Walker, and Figure 3: Hopper). Given that the convergence rate is not consistently guaranteed, fluctuations are expected, and performance does not exceed that of MOBILE, what is the significance of a low standard deviation? From the current graphs, it appears that MOMBO, despite having a high AUC, does not outperform MOBILE when considering the same number of training steps.
Further Clarifications
3. “Based on my understanding, a lower standard deviation typically indicates a faster rate of convergence.“ Can you specify the problem with treating the area under learning curve scores we reported in Table 1 as a quantitative measure of convergence rate and drawing conclusions accordingly? We do understand the merit of looking carefully into individual figures, however, as scientists, we believe more in numbers than bare eye analysis. From the fact that in 11 of the 12 tasks MOMBO delivers higher AUC and similar cross-experiment variance compared to MOBILE, it converges faster. MOMBO and MOBILE perform similarly only when both models are trained for 3 million steps. This training time is typically chosen by performing online evaluations. However, in a realistic offline RL setup, we do not have access to the real environment on which we can check whether our model is sufficiently trained. Furthermore, a model that converges faster would fit better to data when its capacity is increased. The significance of a low standard deviation is improved training robustness. This comes as a side benefit of eliminating a factor of randomness from the training process and an explanation of why our method reaches faster convergence, which is our main result. MOBILE and MOPO perform comparably only when both are trained long enough. This outcome serves as a consistency check and is not the main point of our contribution. We hope we were able to answer your questions. Thanks again for the truly enjoyable scientific discussion we had together. We fully trust your fair scientific judgment and respect any score you give in your final evaluation.
Clarifications about Contribution Statement and Motivation
Thanks for your meticulous review and valuable suggestions. We believe there remains a few points calling for further clarification, but we also fully respect your final decision. > 1. Improvement Suggestions for Writing: We agree with your points and appreciate your rigor in pointing them out greatly. We promise to handle them in the revised version of our work. However, we kindly express our view that their scope stays within the limits of a minor revision. > 2. Clarification and Enhancement of Contribution Statement: > 3. Clarification of Research Motivation: We agree about the need for further improvement here. We believe presenting our core contribution in Section 1, `Our contribution` paragraph as follows would address both points 2 and 3, as well as your concern about the central focus of our work: Estimating tight $\xi$-uncertainty quantifiers and using them for reward penalization is a provable recipe for improved offline reinforcement learning (see Theorem 4.2 of Jin et al. (2021)). Reward penalties calculated on the Bellman target are provably tighter $\xi$-uncertainty quantifiers than those calculated on the environment model (see Theorem 3.6 of Sun et al. (2023)). However, the calculation of this quantity necessitates propagating the uncertainty of the estimated next state through a non-linear value function, which is typically modeled by a deep neural net. The commonplace approach is to approximate this quantity by Monte Carlo sampling. Our key novelties include: i) detecting the problem of approximation errors on the reward penalty for the first time in the literature ii) characterizing the problem theoretically by deriving novel concentration inequalities on their commonplace Monte Carlo estimator, iii) proposing an original solution to this problem by using a moment matching approach originally developed for Bayesian neural net inference for the first time in the context of reward penalty calculation, iv) quantifying the approximation error incurred by the Monte Carlo estimate on a large set of benchmarks, showing a significant reduction of this error by our proposed moment matching approach, and demonstrating the impact of the error on convergence speed (improved AUC score). ----- Jin et al., _Is Pessimism Provably Efficient for Offline RL?_ (ICML 2021) Sun et al., _Model-bellman inconsistency for model-based offline reinforcement learning_ (ICML 2023)
Thank you for the response. The experiments do show some improvement, but I am still not convienced that the previous methods are too pessimistic. Considering the different counstruction of [Li et al. 2024] and the MOPO baseline, I am wondering if the MOPO is the SOTA of current uncertainty-based approaches.
Clarifications about Pessimism and SOTA
Thank you indeed for your response. > Thank you for the response. The experiments do show some improvement, but I am still not convienced that the previous methods are too pessimistic. We believe the tightness scores we reported in the global response provide direct and strong evidence in favor of our claim that MOMBO applies less penalty than both MOPO and MOBILE. The experiments demonstrate that MOPO applies significantly higher penalty than MOMBO and MOBILE, which is an expected consequence of Theorem 3.6 of Sun et al. (2023). To remind, tightness score measures the difference between the penalty applied by a model and the actual error the Bellman estimator makes. See our global response for details. If there still remains an open issue regarding the relative pessimism of the models in comparison, we are happy to discuss further. > Considering the different counstruction of [Li et al. 2024] and the MOPO baseline, I am wondering if the MOPO is the SOTA of current uncertainty-based approaches. We actually consider MOBILE as the SOTA of the model-based offline RL models, not MOPO, given its publication date (ICML, 2023) and the lack of a widely known method that improves its performance using commensurate resources. That said, MOPO is still a reference model representing the option of reward penalization based on environment dynamics. MOBILE improves on MOPO by deriving a penalty term based on a sampling-based estimate of the Bellman target. Our contribution is to identify and characterize the problems this sampling-based approach causes and present a novel solution that addresses these problems both theoretically (see our Theorems 1 and 2, and comparison paragraph between lines 247-258.) and empirically (see our AUC results in Table 1 that demonstrate faster convergence). ----- Sun et al., _Model-bellman inconsistency for model-based offline reinforcement learning_ (ICML 2023)
Thank you for the additional experiments. I maintain my original rating, thank you.
Decision
Accept (poster)