Reinforcement learning has the potential to automate the acquisition of\nbehavior in complex settings, but in order for it to be successfully deployed,\na number of practical challenges must be addressed. First, in real world\nsettings, when an agent attempts a task and fails, the environment must somehow\n"reset" so that the agent can attempt the task again. While easy in simulation,\nthis could require considerable human effort in the real world, especially if\nthe number of trials is very large. Second, real world learning often involves\ncomplex, temporally extended behavior that is often difficult to acquire with\nrandom exploration. While these two problems may at first appear unrelated, in\nthis work, we show how a single method can allow an agent to acquire skills\nwith minimal supervision while removing the need for resets. We do this by\nexploiting the insight that the need to "reset" an agent to a broad set of\ninitial states for a learning task provides a natural setting to learn a\ndiverse set of "reset-skills". We propose a general-sum game formulation that\nbalances the objectives of resetting and learning skills, and demonstrate that\nthis approach improves performance on reset-free tasks, and additionally show\nthat the skills we obtain can be used to significantly accelerate downstream\nlearning.\n