Deep Reinforcement Learning Combustion Optimization System Using Synchronous Neural Episodic Control
Neural episodic control (NEC) is an emerging deep reinforcement learning algorithm, the dynamic-scale memory buffer of which is updated by appending experience in a steady stream. NEC has greater ability to utilize the information of the memory and outperforms other deep reinforcement learning agents. However, the scale of memory consumption increases dramatically along with the training episodes. It is even worse that, to query the large-scale memory buffer, kd-trees based approximate nearest neighbors algorithm has to be repeatedly performed. These defects of NEC consequently hinder the further application on the industrial control units with low-performance hardware; e.g. the thermal power plant. To overcome this problem, in this paper, a deep reinforcement learning system based on the synchronous neural episodic control (SNEC) algorithm is proposed for combustion optimization tasks on limited memory resources. The proposed system consists of two major components: 1) a predictor on the basis of convolutional neural networks and LSTM module; 2) an SNEC optimizer agent. The main research and innovations are as follows: 1) a deep neural network model is developed to predict system states accurately and serves as a proxy for the optimizer; 2) a fixed-scale memory buffer is designed specifically for the control agents with low-performance hardware; 3) a straightforward mechanism for lookup and update is established to eliminate the computation of kd-trees search algorithm; 4) different from the original one, only one public differentiable neural dictionary (PDND) is required for SNEC agent, and thus the Q-value for all the actions are updated synchronously. The experimental results show that the proposed SNEC algorithm outperforms the standard deep Q-networks on limited memory resources; with the application of this system, the power efficiency turns out to be improved, and original gas emissions are reduced. In addition, the control actions are designed to be simple enough for the practical implementation.
Paper
Full text
Deep Reinforcement Learning Combustion Optimization System Using Synchronous Neural Episodic Control
Semantic Scholar · Engineering · 2018
Abstract
Neural episodic control (NEC) is an emerging deep reinforcement learning algorithm, the dynamic-scale memory buffer of which is updated by appending experience in a steady stream. NEC has greater ability to utilize the information of the memory and outperforms other deep reinforcement learning agents. However, the scale of memory consumption increases dramatically along with the training episodes. It is even worse that, to query the large-scale memory buffer, kd-trees based approximate nearest neighbors algorithm has to be repeatedly performed. These defects of NEC consequently hinder the further application on the industrial control units with low-performance hardware; e.g. the thermal power plant. To overcome this problem, in this paper, a deep reinforcement learning system based on the synchronous neural episodic control (SNEC) algorithm is proposed for combustion optimization tasks on limited memory resources. The proposed system consists of two major components: 1) a predictor on the basis of convolutional neural networks and LSTM module; 2) an SNEC optimizer agent. The main research and innovations are as follows: 1) a deep neural network model is developed to predict system states accurately and serves as a proxy for the optimizer; 2) a fixed-scale memory buffer is designed specifically for the control agents with low-performance hardware; 3) a straightforward mechanism for lookup and update is established to eliminate the computation of kd-trees search algorithm; 4) different from the original one, only one public differentiable neural dictionary (PDND) is required for SNEC agent, and thus the Q-value for all the actions are updated synchronously. The experimental results show that the proposed SNEC algorithm outperforms the standard deep Q-networks on limited memory resources; with the application of this system, the power efficiency turns out to be improved, and original gas emissions are reduced. In addition, the control actions are designed to be simple enough for the practical implementation.