HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

Discrete-continuous hybrid action space is a natural setting in many\npractical problems, such as robot control and game AI. However, most previous\nReinforcement Learning (RL) works only demonstrate the success in controlling\nwith either discrete or continuous action space, while seldom take into account\nthe hybrid action space. One naive way to address hybrid action RL is to\nconvert the hybrid action space into a unified homogeneous action space by\ndiscretization or continualization, so that conventional RL algorithms can be\napplied. However, this ignores the underlying structure of hybrid action space\nand also induces the scalability issue and additional approximation\ndifficulties, thus leading to degenerated results. In this paper, we propose\nHybrid Action Representation (HyAR) to learn a compact and decodable latent\nrepresentation space for the original hybrid action space. HyAR constructs the\nlatent space and embeds the dependence between discrete action and continuous\nparameter via an embedding table and conditional Variantional Auto-Encoder\n(VAE). To further improve the effectiveness, the action representation is\ntrained to be semantically smooth through unsupervised environmental dynamics\nprediction. Finally, the agent then learns its policy with conventional DRL\nalgorithms in the learned representation space and interacts with the\nenvironment by decoding the hybrid action embeddings to the original action\nspace. We evaluate HyAR in a variety of environments with discrete-continuous\naction space. The results demonstrate the superiority of HyAR when compared\nwith previous baselines, especially for high-dimensional action spaces.\n