Variable Gain Gradient Descent-based Robust Reinforcement Learning for Optimal Tracking Control of Unknown Nonlinear System with Input-Constraints

In recent times, a variety of Reinforcement Learning (RL) algorithms have been proposed for optimal tracking problem of continuous time nonlinear systems with input constraints. Most of these algorithms are based on the notion of uniform ultimate boundedness (UUB) stability, in which normally higher learning rates are avoided in order to restrict oscillations in state error to smaller values. However, this comes at the cost of higher convergence time of critic neural network weights. This paper addresses that problem by proposing a novel tuning law containing a variable gain gradient descent for critic neural network that can adjust the learning rate based on Hamilton-Jacobi-Bellman (HJB) error and instantaneous rate of variation of Lyapunov function along augmented system trajectories. By allowing high learning rate the proposed variable gain gradient descent tuning law could improve the convergence time of critic neural network weights. Simultaneously, it also results in tighter residual set, on which trajectories of augmented system converge to, leading to smaller oscillations in state error. A tighter bound for UUB stability of the proposed update mechanism is proved. In order to obviate the requirement of nominal dynamics, a neural network based identifier is chosen from existing literature that precedes the RL controller. Numerical studies are then presented to validate the effectiveness of the combined identifier and robust Reinforcement Learning control scheme in controlling a continuous time nonlinear system.