Developmental Reinforcement Learning of Control Policy of a Quadcopter UAV with Thrust Vectoring Rotors

In this paper, we present a novel developmental reinforcement learning-based\ncontroller for a quadcopter with thrust vectoring capabilities. This multirotor\nUAV design has tilt-enabled rotors. It utilizes the rotor force magnitude and\ndirection to achieve the desired state during flight. The control policy of\nthis robot is learned using the policy transfer from the learned controller of\nthe quadcopter (comparatively simple UAV design without thrust vectoring). This\napproach allows learning a control policy for systems with multiple inputs and\nmultiple outputs. The performance of the learned policy is evaluated by\nphysics-based simulations for the tasks of hovering and way-point navigation.\nThe flight simulations utilize a flight controller based on reinforcement\nlearning without any additional PID components. The results show faster\nlearning with the presented approach as opposed to learning the control policy\nfrom scratch for this new UAV design created by modifications in a conventional\nquadcopter, i.e., the addition of more degrees of freedom (4-actuators in\nconventional quadcopter to 8-actuators in tilt-rotor quadcopter). We\ndemonstrate the robustness of our learned policy by showing the recovery of the\ntilt-rotor platform in the simulation from various non-static initial\nconditions in order to reach a desired state. The developmental policy for the\ntilt-rotor UAV also showed superior fault tolerance when compared with the\npolicy learned from the scratch. The results show the ability of the presented\napproach to bootstrap the learned behavior from a simpler system\n(lower-dimensional action-space) to a more complex robot (comparatively\nhigher-dimensional action-space) and reach better performance faster.\n