New Methods for Optimal Operational Control of Industrial Processes Using Reinforcement Learning on Two Time Scales

Abstract

Current challenges in industrial processes control include achieving optimum operation for systems with two-time-scale dynamics and unknown models. This paper presents, for the first time, the integration of singular perturbation theory and reinforcement learning to solve this problem. To this end, an optimal operational control (OOC) problem with two time scales is formulated to reach the desired operational indices. Then, a singularly perturbed dynamics for two-time-scale industrial operational processes is developed by introducing a perturbed scale, resulting in the separation of the original system dynamics. Thus, the original optimization problem is decomposed into a reduced slow subproblem and a boundary fast subproblem. The fact that the sum of the separate solutions of these subproblems is approximately equal to the solution of the OOC problem is proven. Then, two $Q$-learning algorithms are proposed to obtain a composite feedback control. Finally, an industrial thickener example is employed to show the effectiveness of the proposed method.