Library

Adversarial Model for Offline Reinforcement Learning

We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with "any" admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model-based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.

cs.LGcs.AIAdversarial Robustness in Machine LearningReinforcement Learning in RoboticsComputer Science
Source

Paper

References (48)

01Mastering Diverse Domains through World ModelsDanijar Hafner, J. Pašukonis, Jimmy Ba et al.2023 · arXiv.org · 1.3k citations In Library
02RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement LearningMarc Rigter, Bruno Lacerda, Nick Hawes2022 · Neural Information Processing Systems · 168 citations In Library
03When Is Partially Observable Reinforcement Learning Not Scary?Qinghua Liu, A. Chung, Csaba Szepesvari et al.2022 · Annual Conference Computational Learning Theory · 135 citations In Library
04Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample ComplexityLaixi Shi, Gen Li, Yuting Wei et al.2022 · International Conference on Machine Learning · 115 citations In Library
05Adversarially Trained Actor Critic for Offline Reinforcement LearningChing-An Cheng, Tengyang Xie, Nan Jiang et al.2022 · International Conference on Machine Learning · 160 citations In Library
06Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, S. Levine2021 · International Conference on Learning Representations · 1.6k citations In Library
07Provable Benefits of Actor-Critic Methods for Offline Reinforcement LearningA. Zanette, M. Wainwright, E. Brunskill2021 · Neural Information Processing Systems · 139 citations In Library
08Pessimistic Model-based Offline Reinforcement Learning under Partial CoverageMasatoshi Uehara, Wen Sun2021 · International Conference on Learning Representations · 181 citations In Library
09Bellman-consistent Pessimism for Offline Reinforcement LearningTengyang Xie, Ching-An Cheng, Nan Jiang et al.2021 · Neural Information Processing Systems · 336 citations In Library
10A Minimalist Approach to Offline Reinforcement LearningScott Fujimoto, S. Gu2021 · Neural Information Processing Systems · 1.2k citations In Library
11Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement LearningTengyang Xie, Nan Jiang, Huan Wang et al.2021 · Neural Information Processing Systems · 201 citations In Library
12On the Optimality of Batch Policy Optimization AlgorithmsChenjun Xiao, Yifan Wu, Tor Lattimore et al.2021 · International Conference on Machine Learning · 39 citations In Library

Scroll for more · 36 remaining

Similar papers

ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline DataTengyang Xie, Mohak Bhardwaj, Nan Jiang et al.2022 · arXiv (Cornell University) In Library
Model-Based Offline Reinforcement Learning with Adversarial Data AugmentationHongye Cao, Fan Feng, Jing Huo et al.2025 · arXiv (Cornell University) · 2 citations In Library
RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement LearningMarc Rigter, Bruno Lacerda, Nick Hawes2022 · arXiv (Cornell University) · 168 citations In Library
Model-Based Offline Reinforcement Learning With Adversarial Data AugmentationHongye Cao, Fan Feng, Jing Huo et al.2025 · IEEE Transactions on Neural Networks and Learning Systems In Library
Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and DefensesThanh Nguyen, Tung M. Luu, Tri Ton et al.2024 · arXiv (Cornell University) · 6 citations In Library
Model-based Offline Policy Optimization with Adversarial NetworkJunming Yang, Xingguo Chen, Shengyuan Wang et al.2023 · arXiv (Cornell University) · 4 citations In Library
Model-Based Offline Policy Optimization with Adversarial NetworkJunming Yang, Xingguo Chen, Shengyuan Wang et al.2023 · Frontiers in artificial intelligence and applications In Library
Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement LearningJiayu Chen, Le Xu, Aravind Venugopal et al.2025 · arXiv (Cornell University) · 2 citations In Library
Robust Offline Reinforcement Learning from Low-Quality DataWenjie Shi, Tianchi Cai, Shiji Song et al.2021 In Library
RORL: Robust Offline Reinforcement Learning via Conservative SmoothingRui Yang, Chenjia Bai, Xiaoteng Ma et al.2022 · arXiv (Cornell University) · 116 citations In Library

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview
Reviewer sNsB6/10 · confidence 4/52023-06-18

Summary

A fundamental principle in offline RL is pessimism, which however is not free from the performance degradation with respect to the baseline reference policies, i.e., the currently running policies in the system. In viewing this issue, this paper proposes Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies that improve upon an arbitrary reference policy, regardless of the data quality. Specifically, in ARMOR, the authors extend the technique of relative pessimism by adversarially training an MDP model, to achieve robust policy improvement (RPI) with arbitrary reference policies, regardless of whether they collected the data or not. Based on the theory, a scalable implementation of ARMOR is developed, which achieves competitive performance on D4RL benchmarks, while using only a single model rather than the widely-used ensembles.

Strengths

1. The proposed method is theoretically supported. 2. Practical implementation, including the selection of key hyperparameters, is discussed in details. 3. Achieving competitive performance while using only a single model worth praising, since this is important for utilizing more sophisticated dynamics model.

Weaknesses

1. The assumptions, in particular the realizability of the model class $\mathcal{M}$ and $\mathcal{M_\alpha}$, and the policy class $\Pi$, maybe demanding. It is unclear if or how these assumptions holds in practice. It is also unclear if the theoretical results still hold with a misspecified model and/or policy class, e.g., $M^* \notin \mathcal{M}$, $M^* \notin \mathcal{M_\alpha}$ , and/or $\pi_{\mathrm{ref}} \notin \Pi$. 2. The authors draw a distinction between the reference policy and the data-collecting behavior policy. But in the experiments, the reference policy is simply set as the behavior-cloned policy on the same or a very similar offline dataset, e.g., expert v.s. medium. Therefore, this conceptual distinction seems artificial. * In practice/experiments, can the learned policy still improve upon the reference when the behavior policy (offline dataset) behaves significantly different from the reference policy? * And more importantly, why couldn't we simply learn from the data generated by the "reference policy"? 3. The authors claim that the proposed method is robust to hyperparameters within a known set, i.e., the RPI property. But there is no experiment directly showing this. I would love to see more evidence and details pertaining to, for examples, * How to select this set? * How large can this set be? * How many hyperparameters can this set incorporate? 4. Except the proposed method, the practical implementation (Section 4) shows many differences from the implementations of the baseline methods, such as MOPO, COMBO, CQL, IQL and so on. It is therefore unclear if the performance gain is due to the proposed method or due to these different choices of implementation details. This can be seen from the fact that the closely-related model-free method ATAC has performance similar to ARMOR and better than baselines on several datasets.

Questions

1. Can existing model-based methods, such as MOPO, MoREL or COMBO, utilize the reference policy as well? For example, a quick (and naive) thought may be adding an extra behavior-cloning term towards the reference policy on model-generated states (similar to L153-155). I am curious what preventing them from doing so? Otherwise, with such an augmentation, are they still underperform ARMOR? 2. Could you elaborate more on the distinction between the reference policy and behavior policy? Are there any practical setting where the reference policy is significantly different from the policy that collect the offline dataset? 3. In Eqn. (1) and (2), why do you introduce the $\alpha$ parameter? Maybe I miss something, but I don't find $\alpha$ being used later on, except something like "for $\alpha$ large enough".

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors adequately addressed the limitations.

Reviewer pF8E4/10 · confidence 3/52023-07-07

Summary

This paper introduces ARMOR (Adversarial Model for Offline Reinforcement Learning), a novel model-based framework for offline reinforcement learning (RL) that addresses the challenge of performance degradation. Offline RL allows learning decision-making policies from logged data without requiring new data collection. ARMOR robustly learns policies that improve upon a reference policy by adversarially training a Markov decision process (MDP) model. The framework optimizes policies for worst-case performance relative to the reference policy, ensuring robust policy improvement regardless of data coverage. The authors provide theoretical proofs that ARMOR competes with the best policy within data coverage and never degrades the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate their claims, the authors present a scalable implementation of ARMOR that achieves state-of-the-art performance on D4RL benchmarks without using model ensembles. This makes ARMOR suitable for using high-capacity world models. The empirical results support the robust policy improvement property of ARMOR.

Strengths

Overall, ARMOR offers a promising solution for offline RL, providing robust policy improvement over a broad range of hyperparameter choices and regardless of data coverage. The framework has practical implications for real-world problems where collecting diverse or expert-quality data is expensive or infeasible. The paper contributes theoretical analysis, a scalable implementation, and empirical validation.

Weaknesses

1. Inadequate Discussion of Existing Limitations: The paper briefly mentions that offline RL algorithms have not been widely adopted beyond academic research, but it fails to provide a comprehensive discussion of the existing limitations and challenges that hinder their practical adoption. Without a clear identification and discussion of the existing limitations, the paper's motivation may appear less compelling. 2. Conflicting results: There is uncertainty regarding the claim that the policy learned by the proposed approach outperforms different hyper-parameter settings. In Figure 1, significant variations in performance can be observed for the different hyper-parameter choices. It is important to address and discuss this variation in more detail to provide a clearer understanding of the robustness and reliability of the proposed approach across different hyper-parameter settings.

Questions

None

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Reviewer suos7/10 · confidence 4/52023-07-08

Summary

The paper propose a new model-based offline RL framework, called ARMOR, which can robustly learn policies that improve upon an reference policy by adversarially training a Markov decision process (MDP) model. ARMOR aims to optimize for the worst-case relative performance over uncertainty. In experiment, ARMOR implementation achieves good performance on D4RL benchmarks using only a single model.

Strengths

1. The idea is novel. The paper give us a new perspective to learn a dynamics model in offline RL, which is valuable to the community; 2. The paper is overall well-written and easy to follow for me; 3. The implementation is generally reasonble in intuition.

Weaknesses

1. The description in Section 4.1 is quite confusing, with many details tucked away in the appendix. This arrangement is not particularly reader-friendly. I suggest that the authors better link the following points: (1) why the objective in Line 228 can substitute for Eq. (1); (2) how Line 228 is transformed into the goal of Line 199 and Algorithm 1. You don't need to elaborate on the relationship, but at least some intuition or motivation should be provided to the reader. Additionally, an optional suggestion: the authors might consider restructuring the overall arrangement by first introducing the part of Line 228 and then describing how it is implemented, which may result in a more natural discourse. 2. The notation used for expressing errors is confusing here, with many instances of $\mathcal{E}$ and $\mathcal{L}$ representing loss, lacking in distinctiveness. A classic example is the symbols in Line 228, $\mathcal{E}\_{D}$ and $\mathcal{E}_{\hat D}$, both error symbols share the same structure, only the parameters differ. But they represent completely different losses, with the former being the maximum likelihood, and the latter the Bellman error. I suggest the authors revise the notations used, making it clear to readers what kind of loss it is without having to inspect the parameter differences. 3. Figure 2 could be further polished: Without reading the main text, readers would not understand why ARMOR chooses the two models on the right, as the figure (including the caption) does not demonstrate that the reference policy is actually moving rightward. I believe the authors could add this information, allowing readers to better grasp the authors' motivation. 3. Although D4RL is a fairly popular benchmark, I think the tasks from D4RL do not adequately highlight the advantages of the proposed algorithm. As the authors mentioned, this algorithm is motivated by real-world application scenarios, like "Usually, the systems we apply RL to have currently running policies, such as an engneeded autonomous driving rule or a heuristic-based system for diagnosis, and the goal of applying a learning algorithm is often to further improve upon these baseline reference policies...". However, the data in the selected D4RL datasets mostly come from mixed policies. The medium dataset aligns with this scenario, but the baselines already perform well, making it hard to discern the advantage of the proposed method. The recently released neorl [1] dataset might be more suitable for this work, as it was specifically designed for the scenarios the authors proposed, with all datasets collected by a single working policy, a concept similar to the authors' reference policy. [1] NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning

Questions

The implementation of Equation (4) somewhat contradicts my intuition. Specifically, - $M$'s initial goal was to enhance $\pi_{\rm ref}$'s $J$ and reduce $\pi$'s $J$. However, in the final implementation, none of $M$'s optimization aspects involve $\pi_{\rm ref}$. This makes it unclear how the losses employed in practice can meet the original optimization objective of Equation (1). - I'm unclear on why the optimization process for the critic necessitates a pessimistic loss, e.g., $\mathcal{L}_{D_M}$? The need for pessimism about the critic's estimates isn't evident in the objective Equation (1). In Table 1, I'm curious on why several cloned datasets don't work by ARMOR? This seems to be the datasets very suitable for ARMOR to show better performance. In Figure 3, the algorithm shows sensitivity to $\beta$. It would be beneficial if the authors could elucidate the underlying reasons for this and provide any principles to guide researchers. The experiments in Section 5.2 could be improved. The authors claim that "ARMOR can robustly learn policies that improve upon an arbitrary reference policy..." which is important. However, in the experiments, only performance improvements under a specific reference policy are demonstrated. It would be more constructive to verify this claim by picking various policies as reference policies and conduct experiments under one or two environments. I think DOPE [1] can be referenced as a work to conduct this experiment, which preserves many policies that can be used as reference policies. Regarding related work, a discussion could be made with the work [2] that utilizes adversarial approaches for offline environment learning. The optimization objective of this work coincidently using an opposite objective to the Equation (1) of this paper for model learning, which is "$\max_M \min_\pi \ell(M, \pi)$". Also suddenly inspired by the work [2], I would like to propose an open question for discussion: Can we also consider using adversarial policies further with ARMOR, such as $\max_\pi \min_M \max_{\pi_{\rm ref}}$? Since constructing $\pi_{\rm ref}$ from the offline data isn't a necessity, following Theorem 3, if we can create a sufficient number of $\pi_{\rm ref}$ in an adversarial manner, could we theoretically ensure that the final $\hat \pi$ approximates the optimal policy in any situation? [1]Benchmarks for Deep Off-Policy Evaluation. [2] Adversarial Counterfactual Environment Model Learning.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NAN

Reviewer V62y6/10 · confidence 3/52023-07-12

Summary

The paper introduces the Adversarial Model for Offline Reinforcement Learning (ARMOR), a model-based offline RL framework. ARMOR uses adversarial training to robustly learn and improve policies over any given reference policy, regardless of data quality. The framework utilizes the concept of 'relative pessimism' for worst-case optimization, ensuring that it either maintains or improves upon the performance of the reference policy, a property known as Robust Policy Improvement (RPI). The authors have also shown, theoretically and empirically, that ARMOR is robust to hyperparameter choices, and can outperform or be on par with state-of-the-art offline RL methods.

Strengths

* The paper is well-written and technically sound. The RPI in this paper is stronger than the RPI property in the literature, which only guarantees to be no worse than the behavior policy that collected the data. * The experimental results show the effectiveness of the proposed method and it outperforms existing baselines in many environments.

Weaknesses

* I think the proposed method highly depends on the quality of the MDP model $M$. For the results in Table 1, in some environments, the proposed method does not outperform baselines. I think some explanation and intuitions are needed. I am wondering if the $M$'s structure is not optimal in those cases. * The model fitting loss in Equation (3) is not well-defined. Specifically, $r$ is not mentioned. Is it the same as $R^*$ as in Definition 1? * Minor: Please refrain from only using color to distinguish bars in Figure 3, as it is not friendly to readers with color blindness. Also, in Figure 2, not all illustrations are valid MDPs. I suggest the authors re-draw those toy examples.

Questions

Please refer to the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations.

Reviewer sNsB2023-08-15

Response to the authors

Dear authors, Thank you so much for the responses, which clarify some of my previous concerns. I do have some follow-up questions pertaining to your responses. 1. Regarding the statement in the global response: *"a policy cloned on the expert dataset ... is the hardest policy to show RPI against - a near-expert policy that lies outside the data support"*. I am a bit confused about why it is the "hardest" to show RPI against? And why such a near-expert policy lies outside the data support? For the latter, wouldn't "a policy cloned on the expert dataset" automatically lies in the data support? 2. For *"it is straightforward to handle the case of misspecified model and policy classes by additional additive terms"*, could you briefly discuss how would you plan to handle? I understand that there is a word-limit in the response, so it is fine to only give a brief overview. 3. Is the RPI shown on Figure 3 significant? Or, maybe a more fair question is: in what percentage of the settings the RPI is significant over the reference policy? I can see that on many settings, the error bars of the reference policy overlaps substantially with ARMOR (under various $\beta$). Examples may be hammer-human, door-human, hopper-med-replay, and so on. * In any case, I highly recommend the authors to re-draw Figure 3 and set the $y$-range separately for different datasets so that the distinction between REF and the ARMOR variants can be more apparent. 4. *"in general it is hard to directly compare model-based and model-free methods over implementation details"* --- I agree with this. But I would like to kindly point out that both MOPO and COMBO are model-based methods. I think a more apple-to-apple comparison between ARMOR and these baselines can better demonstrate the gain of the proposed method. Otherwise, the current Table 1 looks like the improvement of ARMOR may just come from a better backbone, which ARMOR may not surpass either. * If an apple-to-apple comparison is impossible, can the authors briefly discuss what are the main obstacles (except for using dynamic ensembles which should not be a major blocker)?

Authorsrebuttal2023-08-17

Response to the Reviewer (1/2)

Thank you for the response. Please find our responses to your additional questions below: **Question 1** Regarding the statement in the global response: "a policy cloned on the expert dataset ... is the hardest policy to show RPI against - a near-expert policy that lies outside the data support". Before answering the reviewer’s question, we want to highlight that, in the RPI experiments, the data that ARMOR uses and the data that was used (by BC) to create the expert reference are **different**. In other words, ARMOR needs to learn to be competitive to the reference policy, **without** using the data that created the reference or using data running the reference would generate. **Why the expert reference is the hardest** Since the reference is near optimal, the learned policy also needs to learn to be near optimal to demonstrate the RPI property. This is highly nontrivial, because the learner doesn’t have the reference’s data (as highlighted above) and we have seen that running standard offline RL on the data the learner has access to cannot reach near optimal performance. If ARMOR achieves RPI here, it means that it must be doing something special (i.e., be effective in leveraging the reference policy). In contrast, if we were to use a reference policy with performance that is attainable by offline RL (ORL) using the training data, we wouldn’t be able to clearly establish whether the improvement over the reference is due to typical (ORL) effects, or due to RPI. **Why the near-expert policy lies outside the data support** ARMOR takes as input two things - an offline RL dataset and a reference policy. In the RPI experiments, the dataset is either the medium or medium-replay, and the reference is the “policy cloned on expert dataset”. As highlighted above, ARMOR does not have access to the actual expert dataset for training. We can further ascertain that the near-expert reference lies outside data support by looking at the performance of the best policy extracted by running ORL on medium and medium-replay datasets. This serves as a rough proxy for coverage because if the near-expert policy was covered by the dataset, ORL would recover it. Since, the best ORL policy is not near-expert we can approximately say that it lies outside data support. We realize that this is not a mathematically principled statement, but it is only meant to elucidate the overall point. **Question 2** The key idea for dealing with misspecified classes is to set the best in-class candidate as the learning goal, and pay for the gap between best in-class candidate and the ground truth (i.e., misspecification). For adapting to the model misspecification (modifying Theorem 2): Define $M_{\rm class}^\star$ to be the best in-class candidate, which has small total variation error from the ground truth $M^\star$. Then it is straightforward to get a slightly different version of Theorem 2 which bounds $J_{M_{\rm class}^\star}(\pi^\dagger) - J_{M_{\rm class}^\star}(\widehat\pi)$ rather than $J_{M^\star}(\pi^\dagger) - J_{M^\star}(\widehat\pi)$ (original version), and we pay for the difference between $2\max_{\pi} |J_{M_{\rm class}^\star}(\pi) - J_{M^\star}(\pi)|$ (i.e., misspecification, bounded by the small total variation error assumed before) to obtain what we want. Adapting to the policy misspecification (modifying Theorem 3) is similar. We define $\pi_{\sf ref}^{\rm class}$ to be the best in-class behavior estimator and then 1) using $\pi_{\sf ref}^{\rm class}$ as an intermediate comparator (i.e., $J(\pi_{\sf ref}) - J(\widehat\pi) = J(\pi_{\sf ref}) - J(\pi_{\sf ref}^{\rm class}) - [J(\widehat\pi) - J(\pi_{\sf ref}^{\rm class}])$); 2) $J(\pi_{\sf ref}) - J(\pi_{\sf ref}^{\rm class})$ bounded by policy misspecification; 3) $\pi_{\sf ref}^{\rm class}$ is in class now, we just need to follow the similar argument of dealing with model misspecification as above.

Authorsrebuttal2023-08-17

Response to the Reviewer (2/2)

**Question 3** First, we would like to clarify that RPI is defined as the learned policy being **not worse** than reference policy. RPI does not mean that the learned policy will be **strictly better** than the reference. For the RPI experiments in the main paper, we considered 7 environments with a total of 14 datasets (2 per environment). Overall, we observe the following metrics: **In *14/14* datasets, ARMOR shows RPI (i.e., ARMOR policy is not worse than the reference policy)** (Criterion: the confidence intervals overlap) As we mentioned above, the difference between ORL and REF is a rough indication of whether the reference policy is in the data support. Here we note that in 12/14 cases REF is strictly better than ORL and in **all** those cases ARMOR demonstrates RPI. **In **5/14** datasets, the ARMOR policy is *strictly* better than the reference.** (Criterion: the lower confidence of ARMOR performance is better than upper confidence of REF ) Note here that this metric is highly dependent on the quality of the reference policy. Since the reference is near-expert in these experiments, it can be hard for some environments to improve significantly over the reference. Thank you for the suggestion to re-draw Fig. 3 with a different y-axis range. We will try to include this in the final draft. **Question 4** Regarding an “apples-to-apples” comparison with MOPO and COMBO, we assume the reviewer means implementing all three algorithms using a common backbone. In this regard, we would like to comment on how the conceptual differences among the algorithms led to us choosing ATAC as the backbone. MOPO uses SAC as the underlying RL algorithm. It modifies the reward function for SAC using a pessimism bonus based on dynamics uncertainty and trains the policy on a mixture of model-free and model-based data. In contrast, COMBO’s implementation essentially runs CQL on the mixture data. Note that in both cases the dynamics model training is **unaffected** by policy optimization. However, according to the theory in ARMOR, we need to train an **adversarial** model for the policy. In terms of an actor-critic framework, this means the Bellman error of the critic has to be changeable by the model (which is similar to back-propagation-through-time, but implemented via a critic). This is why the model in ARMOR also minimizes the Bellman error estimate, unlike COMBO and MOPO which train the model and policy+critic in two separate stages. In addition, Appendix D in (Cheng et al, 2022) also shows that the pessimism objective in CQL cannot facilitate adversarial training. Therefore, although ARMOR, COMBO and MOPO, all run actor-critic style algorithms on mixture data, we cannot implement ARMOR based on their backbones CQL or SAC (and vice versa). In fact, the changes (from CQL/SAC) required to implement the adversarial training step needed here essentially make the underlying RL backbone equivalent to ATAC. Therefore, we chose to build upon that.

Reviewer sNsB2023-08-17

Response to the authors

Dear authors, Thank you so much for the response, which addresses most of my concerns. I will increase my rating to 6.

Reviewer suos2023-08-18

Thank you for your response. Most of my concerns have been addressed. Regarding the open questions: 1. I reviewed Appendix E, but I'm still struggling to understand the relationship between $\max_\pi \min_M \max_{\pi_{ref}} J_M(\pi) - J_M(\pi_{ref})$ and Regret Minimization. How does this objective potentially lead to an overly conservative policy? 2. Assuming that $\max_\pi \min_M \max_{\pi_{ref}} J_M(\pi) - J_M(\pi_{ref})$ results in an overly conservative policy, I'm trying to reconcile this with my understanding of Theorem 3. Specifically, if we can generate a sufficient number of $\pi_{ref}$ adversarially, wouldn't we theoretically ensure that the final policy approximates the optimal policy in any situation? This appears to be a direct implication of your Theorem 3. Have I overlooked something?

Authorsrebuttal2023-08-18

Thanks for your further question. First apologies. We made a typo. What we intended to mean in the rebuttal was $\arg\max\_\pi \min\_M {\color{red} \min\_{\pi\_{ref}}} J\_M(\pi) - J\_M(\pi\_{ref}) = \arg\max\_\pi \min\_M J\_M(\pi) - \max\_{\pi\_{ref}} J\_M(\pi\_{ref})$, otherwise the innermost optimization would try to pick the easiest reference policy (i.e., the worst policy in $M$) to compare with and is no longer adversarial. We hope that this is the same as what the reviewer meant earlier. After this correction, suppose the innermost $ \min\_{\pi\_{ref}}$ is over a sufficiently rich policy class, then it's easy to see that the minimizer is $\pi\_{ref} = \pi\_M^\star$, that is, $\max\_\pi \min\_M { \min\_{\pi\_{ref}}} J\_M(\pi) - J\_M(\pi\_{ref}) = \max\_\pi \min\_M J\_M(\pi) - J\_M(\pi\_M^\star)$, thus reducing to the regret minimization case discussed in Corollary 10, bullet point 3. We said the “this algorithm can be overly conservative” because the term $J\_M(\pi) - J\_M(\pi\_M^\star)$ above is always greater than or equal to the performance difference for any fixed reference policy $\pi_{ref}$, i.e., $J\_M(\pi) - J\_M(\pi\_{ref})$. As a result, the policy learned in this way does not necessarily have RPI to a given $\pi_{ref}$. And to reconcile with Theorem 3, note that Theorem 3 applies to ARMOR which has **a fixed $\pi\_{ref}$**, but the $\pi\_{ref}$ above corresponds to a policy that changes as we consider different models $M$ in the version space. When $\pi\_{ref}$ is fixed, it is easy to show (under the assumptions we make) that the objective value of ARMOR is never negative (i.e., RPI), so we never have degenerate performance. The problem with regret minimization and using multiple $\pi\_{ref}$ in general is that the objective value can become negative, which is a sign that the optimization is ill-posed and we are being overly conservative about the performance difference to $\pi_{ref}$. Related to this point, the reviewer conjectured that "[with] sufficient number of $\pi\_{ref}$, [we can] theoretically ensure that the final policy approximates the optimal policy". This is not true. Here is a minimal counterexample: Consider a 2-armed multi-armed bandit. Suppose our version space is $(1, 0), (0, 1)$, i.e., we know one arm is good (reward 1) and one arm is bad (reward 0), but we don't know which is which. In this case, if we choose the two deterministic policies as the 2 reference policies and plug them into the formulation above, the optimal objective is $-1/2$ and the best policy is choosing between the two arms uniformly randomly. Such a policy is suboptimal in both model instances, and in general no policy can yield good improvement in such a case. In general, being able to “approximate the optimal policy” without further assumptions is too good to be true in offline RL since there is model uncertainty that we cannot resolve without access to further (online) data.

Reviewer suos2023-08-19

Thanks for the response. The meaningful adversarial objective should be $\max_\pi \min_\pi \min_{\pi_{ref}} J_M(\pi) - J_M(\pi_{ref})$. Sorry for my mistake. After reading the response, I have recognized that the objective will lead to an overly conservative policy based on the intuitive example. But after recapping the assumptions in the main body, I still cannot understand the statement you made "It is easy to show (under the assumptions we make) ... conservative about the performance difference to $\pi_{ref}$". How can the assumptions (Assumption 1 and Assumption 2) you make derive the conclusion that "the objective value of ARMOR is never negative" and how does the negative value make the optimization ill-posed?

Authorsrebuttal2023-08-19

> "The objective value of ARMOR is never negative" For this we only need the realizability of $\pi\_{ref}$ in Assumption 2, that $\pi\_{ref} \in \Pi$. This way, if we simply choose $\pi = \pi\_{ref}$, the objective is $0$ ($J\_M(\pi) - J\_M(\pi\_{ref})$ is always $0$ regardless of $M$), and since we maximize $\pi$ the optimal choice must have an objective no lower than that of $\pi = \pi\_{ref}$. > "how does the negative value make the optimization ill-posed" This is not a strict statement but more of an intuition. Roughly speaking this means that the improvement goal is too hard to achieve, and in the worst case not only we cannot guarantee improvement, but our policy may be significantly worse than the reference policy. We recommend going through the 2-armed bandit example we provided in the previous response for concreteness.

Reviewer suos2023-08-19

Thanks for the response. If so, I think something is missing in the theorem since, as you said, "the negative value makes the optimization ill-posed" is just from an intuition, instead of a necessity in the process of theoretical analysis. Thus, if I understand correctly, we can derive a contradictory conclusion from intuition and your theorem? In addition, in my previous review, my concerns are major in the written issue, thus I recommend the authors give a specific revision plan (with Line numbers or Section/paragraph). I am glad to increase my score if the written issue is addressed well and the above concern is solved.

Authorsrebuttal2023-08-19

From reading the reviewer’s response, we found that there is likely a confusion, which might have been due to this thread touching on multiple topics. Please let us clarify. > something is missing in the theorem We would like to emphasize there is no contradiction between our theorem and the discussions above. We clarify the following logical steps: 1. The theorems in the paper are for ARMOR, which takes a **fixed** $\pi_{ref}$ as input to the algorithm. 2. Under Assumption 2, the optimal objective of ARMOR is non-negative (as we explained in the previous response). 3. The proof of Theorem 3 on RPI **crucially** relies on this non-negative property (see line 500 in Appendix C). Our current analysis is **not** applicable to other algorithms such using multiple reference policies, or adapting the reference policy during learning, as we discussed in this thread of **open** questions. For these non-ARMOR algorithms, the optimal objective value can be negative (as we explained in the previous response), which breaks the proof of Theorem 3 about RPI (Step 3 above). Thus, we cannot show these non-ARMOR algorithms have RPI to a given $\pi_{ref}$ using the current analysis. Further, we believe these non-ARMOR algorithms don’t have RPI in general, because we can construct counterexamples (as we demonstrated using the bandit example in the previous response). We didn’t give a precise definition of what “ill-posed” means in our first response, which is why we said it’s “more of an intuition”. But we highlight that all the statements in our paper are formally proved. In addition, we point out above non-negativity is crucial to establish the RPI property. So by “ill-posed” what we meant more precisely is that it breaks the current proof and therefore we cannot establish performance guarantees for these non-ARMOR algorithms in the same way we did for ARMOR. We hope this discussion would resolve the reviewer’s remaining concerns. > revision plan We thank the reviewer for suggestions on improving the writing clarity. We will implement them according to our initial rebuttal (e.g., about sec 4.1, loss notation, figures, etc.). We also think this line of “open-question” discussion can be informative and helpful to future readers. We will incorporate it to improve the paper’s clarity (such as highlighting that ARMOR takes a fixed reference policy, and why non-negativity resulting from Assumption 2 is important to establish RPI). We also welcome any further suggestions from the reviewer.

Reviewer suos2023-08-20

If the non-negative property is a necessity in the process of theoretical analysis, it is ok for me as the theorem is consistent with the intuition. Could you check the line you mentioned to show the non-negative property, since I found line 500 is at the end of Appendix A.4 instead of Appendix C, and it is an uninformative sentence "This completes the proof"?

Authorsrebuttal2023-08-20

Line 500 is correct. But we made a typo; you're right it's Appendix A.4, not Appendix C. Sorry for the confusion. What we referred to is the proof just below line 500, which shows "Proof of Theorem 3"; recall Theorem 3 is the RPI theorem of ARMOR. The proof uses the non-negativity of the optimal objective value: $ \max_{\pi \in \Pi} \min_{M \in \mathcal{M}} [ J_M(\pi) - J_M(\pi_{ref})] \geq \min_{M \in \mathcal{M}} [ J_M(\pi_{ref}) - J_M(\pi_{ref})] = 0$, where the inequalities uses $\pi_{ref} \in \Pi$ which is Assumption 2. (This is the last two lines of the proof below line 500). We hope this is clear now!

Reviewer suos2023-08-20

Everything is clear now :) About the revision plan: 1. I think Section 4.1 can be restructured to improve readability. Currently, readers might need to repeatedly check all of the terms among Equations in Line 199, Eq (4-5), and the objective in Line 228 to sort out their relationship. After reading the rebuttal, in my opinion, we can place the "Connection to the Theoretical Formulation" first, which transits the story from Eq (1) to the real implementation and give the whole and real objective, then give the description and insights of terms (like Line 199), and finally give the whole algorithm and other implementation details. 2. I would like to see more discussion on related studies that use the adversarial way for model learning. At least the study [2] is worth to be discussed as a related work since the study also considers an objective of adversarial model learning, in an opposite way. Thanks for your detailed response. I will increase my score.

Reviewer pF8E2023-08-21

Thank you for the response. I still don't find the reasoning for the weaknesses convincing, and will keep the score of 4.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC