Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey

This is a tutorial and survey paper on factor analysis, probabilistic\nPrincipal Component Analysis (PCA), variational inference, and Variational\nAutoencoder (VAE). These methods, which are tightly related, are dimensionality\nreduction and generative models. They assume that every data point is generated\nfrom or caused by a low-dimensional latent factor. By learning the parameters\nof distribution of latent space, the corresponding low-dimensional factors are\nfound for the sake of dimensionality reduction. For their stochastic and\ngenerative behaviour, these models can also be used for generation of new data\npoints in the data space. In this paper, we first start with variational\ninference where we derive the Evidence Lower Bound (ELBO) and Expectation\nMaximization (EM) for learning the parameters. Then, we introduce factor\nanalysis, derive its joint and marginal distributions, and work out its EM\nsteps. Probabilistic PCA is then explained, as a special case of factor\nanalysis, and its closed-form solutions are derived. Finally, VAE is explained\nwhere the encoder, decoder and sampling from the latent space are introduced.\nTraining VAE using both EM and backpropagation are explained.\n