Structural learning in artificial neural networks using sparse optimization

Abstract

Sparse optimization have been applied to simultaneously estimate the weights and model structure of an artificial neural network.The problem has been formulates as an 0-norm optimization problem, which is approximatively solved with an iterative reweighting procedure.The proposed method reduces the complexity an artificial neural network by finding a sparse representation of it, i.e. a solution where as many weights as possible are equated to zero.The proposed algorithms have successfully been applied to several benchmark problems and to a case study for estimating waste heat recovery in ships. In this paper, the problem of simultaneously estimating the structure and parameters of artificial neural networks with multiple hidden layers is considered. A method based on sparse optimization is proposed. The problem is formulated as an 0-norm minimization problem, so that redundant weights are eliminated from the neural network. Such problems are in general combinatorial, and are often considered intractable. Hence, an iterative reweighting heuristic for relaxing the 0-norm is presented. Experiments have been carried out on simple benchmark problems, both for classification and regression, and on a case study for estimation of waste heat recovery in ships. All experiments demonstrate the effectiveness of the algorithm.