Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes

Continuously learning to solve unseen tasks with limited experience has been\nextensively pursued in meta-learning and continual learning, but with\nrestricted assumptions such as accessible task distributions, independently and\nidentically distributed tasks, and clear task delineations. However, real-world\nphysical tasks frequently violate these assumptions, resulting in performance\ndegradation. This paper proposes a continual online model-based reinforcement\nlearning approach that does not require pre-training to solve task-agnostic\nproblems with unknown task boundaries. We maintain a mixture of experts to\nhandle nonstationarity, and represent each different type of dynamics with a\nGaussian Process to efficiently leverage collected data and expressively model\nuncertainty. We propose a transition prior to account for the temporal\ndependencies in streaming data and update the mixture online via sequential\nvariational inference. Our approach reliably handles the task distribution\nshift by generating new models for never-before-seen dynamics and reusing old\nmodels for previously seen dynamics. In experiments, our approach outperforms\nalternative methods in non-stationary tasks, including classic control with\nchanging dynamics and decision making in different driving scenarios.\n