Robust Knowledge Transfer in Tiered Reinforcement Learning

Summary

The paper presents an extension of "Tiered-RL", a multi-fidelity RL framework where a "low-fidelity" environment is executed in parallel with the "high-fidelity" environment, with the purpose of training faster while keeping near-optimal regret. The paper is a theoretical exploration without empirical evaluation

Strengths

- The paper presents a deep theoretical evaluation of a setting that is very relevant yet not very methodologically-explored: When we have related tasks of varied importances to be solved (such as sim2real), and one task can be leveraged to learn faster another one. - An algorithm able to guarantee near-optimal regret while training in multiple tasks might be of use in security-critical applications such as robotics or medical domains.

Weaknesses

- While the theoretical results sound exciting, I would expect at least a simple empirical evaluation of the proposed algorithm to be provided to show how hard it is to actually implement the algorithm in a practical domain. - The "Tiered RL" setting sounds very similar (if not exactly the same) as the multi-fidelity RL modeling, that wasn't even cited by the authors. While there is still a novelty in the exact way the problem is solved by running all "tiers" (or fidelities) in parallel, the problem formulation seems exactly like multi-fidelity MDPs to me, and I would suggest to use the same multi-fidelity MDP formulation to keep consistency in the literature. Multi-fidelity MDPs are explicitly modeled in this paper: Silva, Felipe Leno, et al. "Toward Multi-Fidelity Reinforcement Learning for Symbolic Optimization.", Adaptive and Learning Agents (ALA) workshop, 2023. And the multi-fidelity RL problem has been explored in a similar way in the following papers Sami Khairy and Prasanna Balaprakash. 2022. Multifidelity reinforcement learning with control variates. arXiv preprint arXiv:2206.05165 (2022). Mark Cutler, Thomas J Walsh, and Jonathan P How. 2014. Reinforcement learning with multi-fidelity simulators. In 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3888–3895. At the very least all of those papers should have been included to your related works section.

Questions

- A doubt regarding the setting. Would the agent care about the performance in all tiers? Or is it only important to optimize the performance in the highest tier? If only the highest tier matter, the setting is exactly the same as in multi-fidelity RL as said in the "weaknesses" section, otherwise, I would need more clarification of in which practical application this modeling would be useful.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

No foreseeable negative societal impact

Summary

The authors look at the tiered reinforcement learning setting, which is a parallel transfer learning framework where the goal is to transfer knowledge from a low-tier source task to a high-tier target task in order to reduce the exploration risk of the high-tier task. Additionally, these tasks are solved in parallel. Contrary to previous related work, the authors do not assume the low-tier and high-tier tasks share dynamics or reward functions and focus on robust knowledge transfer without prior knowledge on task similarity. The authors use a condition called the “Optimal Value Dominance” to propose novel online learning algorithms that can achieve constant regret on partial states depending on the task similarity and near-optimal regret when the two tasks are dissimilar. Furthermore, for low-tier tasks, these algorithms keep near-optimal regret at very little cost. The authors also study the scenario when multiple low-tier tasks are present and propose a novel transfer source selection algorithm that can gather knowledge from all low-tier tasks and produce benefits on a much larger state-action space.

Strengths

* The regret analysis of the robust tiered multi-armed bandit (MAB) models was very thorough.

Weaknesses

* No experiments that compared performance with other tiered RL algorithms * It would have nice to have had the related work in the manuscript instead of as a supplement.

Questions

* What type of data have others used on this problem or similar RL problems? Can it be leveraged in this setting as well?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

* The authors did not address the fact that they did not perform any experiments with either synthetic or real data on their model.

Summary

The authors propose a robust parallel knowledge transfer reinforcement learning algorithm for single or multiple source tasks without knowledge on model similarity using the previously defined Tiered Reinforcement Learning framework. The paper remove the limitation on prior knowledge about the task similarity to generalize the framework. The main contribution is three-fold: 1) Establish necessary conditions for lower regret bound, 2) propose robust parallel transfer algorithms for reinforcement learning and its special case, multi-armed bandits, and 3) describe a new source task selection mechanism that guarantees constant regret for larger space of state-action pairs using multiple low-tier tasks.

Strengths

Originality. Removing the assumption on task similarity poses new challenges to guarantee lower regret bounds that are not trivial. The new algorithm to solve the general knowledge transfer require learning whether low-tier and high-tier tasks are similar from observed data and balance between exploration and exploration from the low-tier model at the same time. The lower bounds from simplifying the general case simplify to the prior work under the similarity conditions between low- and high-tier tasks. Therefore, the lower bounds are sound. To extend the approach for the high-tier task to leverage multiple low-tier tasks, the authors describe a method called “trust till failure” that still guarantees lower bounds on regret. Quality & Clarity. The contribution and novelty is well defined. The definitions and equations are sound. Significance. The proposed approach generalizes the previous work to a larger class of problems while still maintaining lower bounds on regret minimization.

Weaknesses

Originality & Significance. To remove the limitation of task similarity for generalization is not trivial. However, the motivation for generalization is lacking with few references to potential robotics applications. The paper can benefit from providing an illustrative example or toy problems to motivate new class of problems the proposed method can be applied to. Quality. I am assuming the authors made a mistake and uploaded the incomplete paper. The related work is supposed to be in the Appendix but there are no Appendices attached to the submitted paper. Without these Appendices, it is difficult to compare how the new method compared to prior work. There are several grammatical errors and the conclusion section does not talk about the limitations of the current approach. The future work is also in Appendix which is not included in the paper. Clarity. It is not clear how parallel knowledge transfer is any different from meta-learning and/or why it is not a special case of meta-learning. It will be good to discuss when the learning methods overlap or differ under which conditions to better motivate the approach. Specifically, for the use case of multiple low-tier learners with a single high-tier learner.

Questions

Please provide the appendices to complete the evaluation of the work on soundness and its relation to prior work.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors do not provide limitations of the proposed approach. There are no societal issues related to this work.

Summary

This paper studies the tiered reinforcement learning setting, a transfer learning setting where the goal is to transfer information from a low tier task to a higher tier one while learning both to solve both tasks in parallel. Contrary to prior work the author do not assume that both tasks share the same rewards and dynamics, they show it is still possible to benefit from the low tier task to learn the higher tier one. The author also extend their work by considering multiple low tier tasks and present selection mechanism which can gather information the different tasks.

Strengths

Overall this is a solid paper, while technical the paper is well written and properly organized. Removing the assumption that the low and high tier sources share the same dynamics and rewards makes this setting much more applicable and interesting. The setup with multiple low tier tasks is also valuable could have many applications.

Weaknesses

It would have been interesting to add empirical evaluations of the proposed algorithms to understand their actual performance.

Questions

In the multiple source tier setting is there a point where adding more tasks with a low similarity with the high tier tasks can hurt performance?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations.

Summary

The paper extends the work of Huang et al (2022) on Tiered Reinforcement Learning (where the objective is to transfer knowledge from the low tier (risk-tolerant) source task to a high-tier (risk-averse) target task while solving the two two tasks in parallel) by relaxing the assumptions of identical reward and transition functions in the source and target tasks. The paper first identifies ‘Optimal Value Dominance’ a necessary condition to keep the source algorithm near-optimal while achieving provable benefits for the regret in the high-tier algorithm when Low and High tier tasks are similar. The paper then proposes algorithms for tiered multiarmed bandit and tiered reinforcement learning with regret lower bounds depending on the similarity between the low- and high-tier tasks, with an improved lower bound for the case when the tasks are the same compared to Huang et al (2022). Finally, the paper also considers a case when there are multiple low-tier tasks, in which the most similar states to the high-tier task are chosen from any of the low tier tasks for an additional log W factor in regret.

Strengths

Originality The paper extends the framework (multiple low-tier tasks) of and relaxes some of the assumptions (non-identical reward and transition function among the low- and high-tier tasks) of a previous paper on Tiered RL. Quality The paper is mostly well-written, clear and concise. The paper motivates the problem setting well in the introduction, and places it well in the literature. The concepts are introduced and explained in a logical order. Clarity The paper has a similar structure to Huang et al (2022). It is mostly clear and easy to follow, although certain parts do feel a bit rushed. (I will mention them in the Weaknesses part.) Significance The paper contributes a few findings in Tiered RL that could be useful to the community: it proves a tighter lower bound for the case when the lower and higher tier tasks are the same, and proves lower bounds for the case when the reward and transition functions differ depending on the difference in the gaps. It also introduces to setting when there are multiple low-tier tasks.

Weaknesses

The work in this paper results from a minor relaxation of certain assumption in Huang et al (2022), and a lot of possible extensions and improvements are left for future work. The RL results depend on a hyperparameter, lambda that needs to be chosen wisely. Minor typos/grammar/style issues that did not affect the rating of this paper: Lines 37, 41 and 52: “In [13]” is considered bad style, either write Author et al (year) [13], or better yet just cite the claim. Lines 62: conceptions -> findings Lines 67-68, starting with while: that subsentence makes no sense, please rephrase The sentence on lines 125-127 seems like it belongs to the Frequently Used Notations subsection, not to 2.1 Line 165: no need for ‘actually’ Line 166: ‘leave it to the future work’ -> ‘leave it for future work’ Line 176: remove ‘kind of’ or replace with ‘may [explain]’ Line 177 is -> are, literatures -> papers/works/publications

Questions

What do you mean by ‘back-propagation process of value iteration’ (line 255)? I am familiar with the concepts of backpropagation and value iteration individually, but their combination is rather ambiguous to me. How would one go about choosing the lambda hyperparameter before beginning to solve the tasks?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors are upfront about the limitations, such as the unique optimal policy assumption, and mention in the appendix that the avoidance of the lower bound knowledge about the minimal gap would be beneficial. Concerns for negative societal impact in this theoretical paper are not applicable.