Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention

Reinforcement Learning (RL) algorithms can in principle acquire complex\nrobotic skills by learning from large amounts of data in the real world,\ncollected via trial and error. However, most RL algorithms use a carefully\nengineered setup in order to collect data, requiring human supervision and\nintervention to provide episodic resets. This is particularly evident in\nchallenging robotics problems, such as dexterous manipulation. To make data\ncollection scalable, such applications require reset-free algorithms that are\nable to learn autonomously, without explicit instrumentation or human\nintervention. Most prior work in this area handles single-task learning.\nHowever, we might also want robots that can perform large repertoires of\nskills. At first, this would appear to only make the problem harder. However,\nthe key observation we make in this work is that an appropriately chosen\nmulti-task RL setting actually alleviates the reset-free learning challenge,\nwith minimal additional machinery required. In effect, solving a multi-task\nproblem can directly solve the reset-free problem since different combinations\nof tasks can serve to perform resets for other tasks. By learning multiple\ntasks together and appropriately sequencing them, we can effectively learn all\nof the tasks together reset-free. This type of multi-task learning can\neffectively scale reset-free learning schemes to much more complex problems, as\nwe demonstrate in our experiments. We propose a simple scheme for multi-task\nlearning that tackles the reset-free learning problem, and show its\neffectiveness at learning to solve complex dexterous manipulation tasks in both\nhardware and simulation without any explicit resets. This work shows the\nability to learn dexterous manipulation behaviors in the real world with RL\nwithout any human intervention.\n