Summary
This paper studies the problem of adaptation in Reinforcement Learning (RL) and proposes an adaptation strategy to improve training efficiency. This method is compatible with pre-trained neural networks and rule-based agents. The authors provide experiments in the nanoRTS environment to demonstrate the strength of their method.
Strengths
(1) Interesting topic: Adaptation is an interesting and motivating area for reinforcement learning.
Weaknesses
(1) Lack of related works discussion. The authors state that adaptation for RL is a largely unexplored area, while I would recommend doing a further literature review on similar topics. Some examples contain [1, 2, 3]
(2) Insufficient experiments. The provided experiment results are limited. I would recommend doing more experiments with diverse tasks. For example, if the domain is focused on Reinforcement learning, you may try other tasks in the OpenAI gym [4].
(3) Lack of README file to run the code. I appreciate the authors for providing the code. However, it would further benefit the reviewers, and potential users with a README file to guide how to run the code.
Reference:
[1] Nagabandi, Anusha, et al. "Learning to adapt in dynamic, real-world environments through meta-reinforcement learning." arXiv preprint arXiv:1803.11347 (2018).
[2] Arndt, Karol, et al. "Meta reinforcement learning for sim-to-real domain adaptation." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.
[3] Guez, Arthur, David Silver, and Peter Dayan. "Efficient Bayes-adaptive reinforcement learning using sample-based search." Advances in neural information processing systems 25 (2012).
[4] Brockman, Greg, et al. "Openai gym." arXiv preprint arXiv:1606.01540 (2016).
Questions
(1) What is the relationship between the proposed method with adaptive RL such as [5, 6, 7]
Reference:
[1] Nagabandi, Anusha, et al. "Learning to adapt in dynamic, real-world environments through meta-reinforcement learning." arXiv preprint arXiv:1803.11347 (2018).
[2] Arndt, Karol, et al. "Meta reinforcement learning for sim-to-real domain adaptation." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.
[3] Guez, Arthur, David Silver, and Peter Dayan. "Efficient Bayes-adaptive reinforcement learning using sample-based search." Advances in neural information processing systems 25 (2012).
[4] Brockman, Greg, et al. "Openai gym." arXiv preprint arXiv:1606.01540 (2016).
[5] Khan, Said G., et al. "Reinforcement learning and optimal adaptive control: An overview and implementation examples." Annual reviews in control 36.1 (2012): 42-59.
[6] Rigter, Marc, Bruno Lacerda, and Nick Hawes. "Risk-averse bayes-adaptive reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 1142-1154.
[7] Eghbal-zadeh, Hamid, Florian Henkel, and Gerhard Widmer. "Context-adaptive reinforcement learning using unsupervised learning of context variables." NeurIPS 2020 Workshop on Pre-registration in Machine Learning. PMLR, 2021.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.