VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

Energy-based models (EBMs) have recently been successful in representing\ncomplex distributions of small images. However, sampling from them requires\nexpensive Markov chain Monte Carlo (MCMC) iterations that mix slowly in high\ndimensional pixel space. Unlike EBMs, variational autoencoders (VAEs) generate\nsamples quickly and are equipped with a latent space that enables fast\ntraversal of the data manifold. However, VAEs tend to assign high probability\ndensity to regions in data space outside the actual data distribution and often\nfail at generating sharp images. In this paper, we propose VAEBM, a symbiotic\ncomposition of a VAE and an EBM that offers the best of both worlds. VAEBM\ncaptures the overall mode structure of the data distribution using a\nstate-of-the-art VAE and it relies on its EBM component to explicitly exclude\nnon-data-like regions from the model and refine the image samples. Moreover,\nthe VAE component in VAEBM allows us to speed up MCMC updates by\nreparameterizing them in the VAE's latent space. Our experimental results show\nthat VAEBM outperforms state-of-the-art VAEs and EBMs in generative quality on\nseveral benchmark image datasets by a large margin. It can generate\nhigh-quality images as large as 256$\\times$256 pixels with short MCMC chains.\nWe also demonstrate that VAEBM provides complete mode coverage and performs\nwell in out-of-distribution detection. The source code is available at\nhttps://github.com/NVlabs/VAEBM\n