Spectral Dynamics of Learning Restricted Boltzmann Machines

The restricted Boltzmann machine, an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Based on empirical observations, we propose a generic statistical ensemble for the weight matrix of the RBM and characterize its mean evolution, with respect to common learning procedures as a function of some statistical properties of the data. In particular we identify the main unstable deformation modes of the weight matrix which emerge at the beginning of the learning and unveil in some way how these further interact in later stages of the learning procedure.