Constraining Variational Inference with Geometric Jensen-Shannon Divergence

We examine the problem of controlling divergences for latent space\nregularisation in variational autoencoders. Specifically, when aiming to\nreconstruct example $x\\in\\mathbb{R}^{m}$ via latent space $z\\in\\mathbb{R}^{n}$\n($n\\leq m$), while balancing this against the need for generalisable latent\nrepresentations. We present a regularisation mechanism based on the\nskew-geometric Jensen-Shannon divergence\n$\\left(\\textrm{JS}^{\\textrm{G}_{\\alpha}}\\right)$. We find a variation in\n$\\textrm{JS}^{\\textrm{G}_{\\alpha}}$, motivated by limiting cases, which leads\nto an intuitive interpolation between forward and reverse KL in the space of\nboth distributions and divergences. We motivate its potential benefits for VAEs\nthrough low-dimensional examples, before presenting quantitative and\nqualitative results. Our experiments demonstrate that skewing our variant of\n$\\textrm{JS}^{\\textrm{G}_{\\alpha}}$, in the context of\n$\\textrm{JS}^{\\textrm{G}_{\\alpha}}$-VAEs, leads to better reconstruction and\ngeneration when compared to several baseline VAEs. Our approach is entirely\nunsupervised and utilises only one hyperparameter which can be easily\ninterpreted in latent space.\n