Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives

Despite the potential of reinforcement learning (RL) for building\ngeneral-purpose robotic systems, training RL agents to solve robotics tasks\nstill remains challenging due to the difficulty of exploration in purely\ncontinuous action spaces. Addressing this problem is an active area of research\nwith the majority of focus on improving RL methods via better optimization or\nmore efficient exploration. An alternate but important component to consider\nimproving is the interface of the RL algorithm with the robot. In this work, we\nmanually specify a library of robot action primitives (RAPS), parameterized\nwith arguments that are learned by an RL policy. These parameterized primitives\nare expressive, simple to implement, enable efficient exploration and can be\ntransferred across robots, tasks and environments. We perform a thorough\nempirical study across challenging tasks in three distinct domains with image\ninput and a sparse terminal reward. We find that our simple change to the\naction interface substantially improves both the learning efficiency and task\nperformance irrespective of the underlying RL algorithm, significantly\noutperforming prior methods which learn skills from offline expert data. Code\nand videos at https://mihdalal.github.io/raps/\n