Crystalizing Effect of Simulated Annealing on Boltzmann Machine

Abstract

This paper proposes a method to estimate the posterior distribution of a Boltzmann machine. Due to high feature extraction ability, a Boltzmann machine is often used for both of supervised and unsupervised learning. It is expected to be suitable for multimodal data because of its bi-directional connection property. However, it needs a sampling method to estimate the posterior distribution, which becomes a problem during an inference period because of the computation time and instability. Therefore, it is usually converted to feedforward neural networks, which means to lose its bi-directional property. To deal with these problems, this paper proposes a method to estimate the posterior distribution of a Boltzmann machine fast and stably without converting it to feedforward neural networks. The key idea of the proposed method is to estimate the posterior distribution using a simulated annealing on non-uniform temperature distribution. The advantage of the proposed method against Gibbs sampling and conventional simulated annealing is shown through experiments with artificial dataset and MNIST. Furthermore, this paper also gives the mathematical analysis of Boltzmann machine’s behaviour with regard to temperature distribution.