Disentangled Representation Learning and Generation with Manifold Optimization

Disentanglement is a useful property in representation learning which\nincreases the interpretability of generative models such as Variational\nautoencoders (VAE), Generative Adversarial Models, and their many variants.\nTypically in such models, an increase in disentanglement performance is\ntraded-off with generation quality. In the context of latent space models, this\nwork presents a representation learning framework that explicitly promotes\ndisentanglement by encouraging orthogonal directions of variations. The\nproposed objective is the sum of an autoencoder error term along with a\nPrincipal Component Analysis reconstruction error in the feature space. This\nhas an interpretation of a Restricted Kernel Machine with the eigenvector\nmatrix-valued on the Stiefel manifold. Our analysis shows that such a\nconstruction promotes disentanglement by matching the principal directions in\nthe latent space with the directions of orthogonal variation in data space. In\nan alternating minimization scheme, we use Cayley ADAM algorithm - a stochastic\noptimization method on the Stiefel manifold along with the ADAM optimizer. Our\ntheoretical discussion and various experiments show that the proposed model\nimproves over many VAE variants in terms of both generation quality and\ndisentangled representation learning.\n