Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Data

Summary

The authors introduced the concept of sparsified MDP. Based on this concept, they proposed a new algorithm that takes as input a dataset, uses it to design and deploy a non-reactive exploratory policy, and then outputs a locally near-optimal policy. A nearly minimax-optimal upper bound for the sample complexity is also established to learn a local ε-optimal policy using this algorithm.

Strengths

- This paper proposed a new setting where the agent is access to an offline dataset and can further explore the environment online with a non-reactive policy. In this way, the agent can utilize the offline dataset while avoid the engineering costs associated with switching policies. - Sparsified MDP provides an perspective on combining both optimism and pessimism principle, which is interesting. - The authors obtain the sample complexity bound of the new approach, and the bound is tighter than previous method. - The paper is well-written and easy to follow.

Weaknesses

- The algorithm is based on tabular setting, which limits its application. - The exploratory policy only explore in the sparsified MDP. In this way, the final policy is strictly limited by the offline dataset even if the budget for online interaction is large. - The transitions that cannot achieve the threshold is dropped directly, even though they contain some information of the environment.

Questions

- Is it possible for the exploratory policy to explore out of the coverage of offline dataset? I know it is impossible if there is no data at all, but it seems that using the information of the dropped transitions may help.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are not explicitly discussed by the paper.

Summary

The paper explores reinforcement learning applications where a pre-existing dataset of collected experience is available, and suggests the possibility of obtaining additional online data to enhance policy quality. To avoid the costs associated with switching policies, the authors propose utilizing a single non-reactive exploration policy for gathering supplementary data. They present an algorithm with provable guarantees that leverages an offline dataset to design such a policy. The algorithm is analyzed theoretically, and the authors evaluate the final policy's quality based on the local coverage of the original dataset and the amount of additional data collected. Overall, the research contributes to improving reinforcement learning by optimizing data acquisition and policy design.

Strengths

The advantages of this paper are as follows: 1. Although I haven't carefully derived each equation, the proofs in this paper are expected to be accurate, with complete steps and rigorous derivations. 2. This paper represents the pioneering effort in terms of theoretical rigor, addressing the challenge of designing an experiment in reinforcement learning for online passive exploration, leveraging a dataset comprising pre-collected experiences. The setting is novel and has some practical significance. 3. The proof method presented in this paper is innovative and the sample complexity mentioned in the conclusion is also tight. It will introduce new approaches to the theoretical research of RL. Additionally, the conclusions of this paper also reveal some interesting insights that will enrich the existing theoretical achievements in the offline-to-online field.

Weaknesses

1. The basic assumption of this article is that the offline-trained policy cannot be switched during the online phase, and it allows for the collection of an unlimited number of samples using this policy. I have two concerns. Firstly, from a theoretical research perspective, this assumption narrows down the problem to a very specific setting, so even with rigorous mathematical proofs, the generalizability of the conclusions may be compromised. Secondly, from a practical application standpoint, if the offline-trained policy itself is poor but cannot be switched during the online phase, and a large number of online samples need to be collected using this policy, there will be even greater security issues. As a result, the advocated security considerations in this paper will no longer exist. I appreciate the mathematical methods used in this paper, but the lack of persuasive motivation will impact the significance of this paper. 2. The proof process in this paper is too lengthy and difficult to understand. Although the main text provides some introduction to the overall proof logic, there are too many specific terms involved in the proof without providing intuitive explanations for their generation, which increases the difficulty of understanding. I suggest that the author can provide a more concise version, even if the resulting sample complexity is not optimal, but it can be used to understand the overall proof framework. 3. Another limitation is that it currently only applies to smaller S and A. When S and A are larger, the application will become difficult and the sample complexity will be significant. So I'm curious to know if this approach can be extended to scenarios involving function approximation. If it is possible, what additional considerations or processing steps would be required? If it is not feasible, what challenges exist?

Questions

See weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors didn't discuss the limitation of their work.

Summary

This paper proposes an algorithm for policy fine-tuning in reinforcement learning using a dataset of pre-collected experience. The algorithm leverages the dataset to design a non-reactive exploratory policy and outputs a locally near-optimal policy. The paper makes theoretical contributions in analyzing the quality of the policy and establishing sample complexity bounds. The work is motivated by the practical need for non-reactive exploration in domains where switching policies is costly and impractical. The paper presents innovative ideas and provides a novel solution to the problem.

Strengths

- The paper introduces an algorithm that addresses the problem of non-reactive policy design in reinforcement learning and provides provable guarantees for the quality of the resulting policy. - The concept of sparsified MDP is introduced and effectively used in the algorithm and theoretical analysis. - The paper rigorously establishes a nearly minimax-optimal upper bound for the sample complexity needed to learn a local ε-optimal policy using the proposed algorithm. - The paper addresses a practical need for non-reactive exploration in domains where policy switches are costly and provides a solution that can be valuable in such scenarios.

Weaknesses

- The paper only considers discrete state and action spaces. - The paper provides no empirical evaluations or demonstrations of the proposed algorithm. Neither does it shed light on the design of practical algorithms.

Questions

- Can you give more details on the non-reactive property of a policy? It seems that most RL policies in an MDP will be non-reactive, as long as they take only the current state $s_t$ as input. How is the example in line 30-32 related to the non-reactive property? - Online exploration may lead to safety violations. How safe will the online phase be in the original MDP? - It seems contradictory to be both optimistic and pessimistic at the same time. How to determine the region that the agent knows how to navigate? - Can you provide some empirical evaluations of the proposed algorithm, compared with fully offline or fully online algorithms?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The paper proposes an algorithm that, given a previously collected dataset of transitions from an MDP, produces a non-reactive policy that can effectively collect additional data that enables a near-optimal policy to be obtained for any possible reward function. The algorithm is model-based and combines elements of optimism (exploration bonuses) and pessimism (early termination at OOD states/actions). Suboptimality is bounded relative to the optimal policy that stays within the “sparsified MDP” which is the subset of the original MDP that is sufficiently covered by the data.

Strengths

* The paper presents positive, novel (to my knowledge) theoretical results in a well-motivated setting of practical interest. * The interaction protocol is new AFAIK, but closely related to existing areas such as reward-free RL. It may inspire more work in the future. * The paper is clear and not hard to follow, despite its technicality.

Weaknesses

* No experiments, despite having “Design of Experiments” in the title :) (But this is a theory paper so I think it is okay) * IMO the Related Work should cite the MOReL paper [1], which uses a pessimistic MDP construction similar to your sparsified MDP, in which low-density states/actions lead to a special absorbing state. Of course, they are tackling a different setting (offline RL with a particular reward function) and use a somewhat different termination criterion, but the idea is the same.

Questions

I’m curious if it is known that better sample complexity can be obtained if you allow the policy to be adaptive? (Although I understand the engineering-related reasons for not doing so.) If so, it could be useful context to briefly comment on in the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

No, limitations are not addressed.

Summary

The work proposes a method to create a non-reactive exploratory policy from an initial input dataset. Then, leveraging the new data, the algorithm generates a locally near-optimal policy. The relevance of the algorithm is in the low-switching algorithms, where it is assumed there is a cost to changing a deployed policy. The proposed algorithm uses the initial data to build a sparsified MDP, an approximation of the original MDP that keeps only the transitions (s, a, s') for which there are at least Φ transitions. The next part of the algorithm is building the exploratory policy using the sparsified MPD using a value iteration iterative strategy with an exploration bonus instead of actual rewards. The exploration bonus balances optimism and pessimism, boosting exploration towards less explored states while avoiding spending too much time in unknown parts of the MDP. Finally, additional data is collected from the environment using this exploring policy, and a value iteration algorithm is employed on the combined datasets of experiences to build the final policy. The authors provide optimality bounds, describing the conditions for the algorithm to discover an epsilon-suboptimal policy.

Strengths

The strengths of this paper lie in the originality of the setup and in the mathematical work to describe the conditions that need to be met for sub-optimality guarantees.

Weaknesses

The weakness of the paper is the need for empirical. For example, it would be helpful to see how well the algorithm performs given initial data sets of different sizes and coverage.

Questions

Could some experiments be added to see the performance of the algorithm in practice, maybe compared to other offline RL methods?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

no

Summary

The paper considers the setting where it is possible to leverage a dataset of transitions, together with the possibility of deploying a policy to collect additional information. The question then lies into what kind of policy should be deployed and what kind of data should be gathered. The authors argue that deploying an exploratory policy that switches, e.g. a policy that learns and adapts from its experience, leads a great engineering costs. As such, they propose to follow the principle of pessimism together with a non reactive policy that would be constrained to a sub-region of the MDP with enough transitions. They also make use of the principle of optimism to derive the exploratory actions taken within the subMDP. The authors provide a near optimal minimax-optimal upper bound for learning an epsilon-optimal policy.

Strengths

The contributions and assumptions are made clear in the introduction (however, the concept of reactive policy is only clearly explained in plain a words a bit late). An intuition section is also provided which helps the reader follow the paper. The authors provide theoretical guarantees both for the sparsified MDP together with the full MDP. In particular for the results on the full MDP it seems like a reduction in the epsilon coefficient is an important contribution. The paper emphasizes a hybrid approach, exploring a setting that combines elements of both offline and online methods, hence offering a more practical and adaptable framework for various real-world applications.

Weaknesses

Although the setting of interest is important (mix of online and offline) the paper doesn’t stress enough the importance of a non-reactive policy. Why is learning from the generated experience such a bad idea in practice? It seems like a slowly changing deployed policy (where changes perhaps happen through a trust region) would be a better choice. I understand the paper is essentially a theoretical one, however it would be interesting to present some empirical evidence as to the practicability of the proposed algorithm. Indeed, it is not clear if the current dependencies on state, action, epsilon and such quantities in the bounds would provide a meaningful difference. The paper misses an opportunity to provide a comprehensive conclusion, one that effectively synthesizes the results, giving the reader a clear understanding of the study's overall implications and potential future directions.

Questions

How good of a policy (trained from scratch) could be obtained with the number of additional samples required in Corollary 5.2? Although the dependence on the desired accuracy is reduced in Corollary 5.2, how much of a difference can this make given the concentrability coefficient?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The limitations are not clearly stated throughout the work. Some of the questions above try to probe into this.

Response

I acknowledge the authors' rebuttal and I remain my rating towards acceptance.