Smoothed functional-based gradient algorithms for off-policy reinforcement learning: A non-asymptotic viewpoint

We propose two policy gradient algorithms for solving the problem of control\nin an off-policy reinforcement learning (RL) context. Both algorithms\nincorporate a smoothed functional (SF) based gradient estimation scheme. The\nfirst algorithm is a straightforward combination of importance sampling-based\noff-policy evaluation with SF-based gradient estimation. The second algorithm,\ninspired by the stochastic variance-reduced gradient (SVRG) algorithm,\nincorporates variance reduction in the update iteration. For both algorithms,\nwe derive non-asymptotic bounds that establish convergence to an approximate\nstationary point. From these results, we infer that the first algorithm\nconverges at a rate that is comparable to the well-known REINFORCE algorithm in\nan off-policy RL context, while the second algorithm exhibits an improved rate\nof convergence.\n