Regularizing Action Policies for Smooth Control with Reinforcement Learning

A critical problem with the practical utility of controllers trained with\ndeep Reinforcement Learning (RL) is the notable lack of smoothness in the\nactions learned by the RL policies. This trend often presents itself in the\nform of control signal oscillation and can result in poor control, high power\nconsumption, and undue system wear. We introduce Conditioning for Action Policy\nSmoothness (CAPS), an effective yet intuitive regularization on action\npolicies, which offers consistent improvement in the smoothness of the learned\nstate-to-action mappings of neural network controllers, reflected in the\nelimination of high-frequency components in the control signal. Tested on a\nreal system, improvements in controller smoothness on a quadrotor drone\nresulted in an almost 80% reduction in power consumption while consistently\ntraining flight-worthy controllers. Project website: http://ai.bu.edu/caps\n