Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models

Many applications of generative models rely on the marginalization of their\nhigh-dimensional output probability distributions. Normalization functions that\nyield sparse probability distributions can make exact marginalization more\ncomputationally tractable. However, sparse normalization functions usually\nrequire alternative loss functions for training since the log-likelihood is\nundefined for sparse probability distributions. Furthermore, many sparse\nnormalization functions often collapse the multimodality of distributions. In\nthis work, we present $\\textit{ev-softmax}$, a sparse normalization function\nthat preserves the multimodality of probability distributions. We derive its\nproperties, including its gradient in closed-form, and introduce a continuous\nfamily of approximations to $\\textit{ev-softmax}$ that have full support and\ncan be trained with probabilistic loss functions such as negative\nlog-likelihood and Kullback-Leibler divergence. We evaluate our method on a\nvariety of generative models, including variational autoencoders and\nauto-regressive architectures. Our method outperforms existing dense and sparse\nnormalization techniques in distributional accuracy. We demonstrate that\n$\\textit{ev-softmax}$ successfully reduces the dimensionality of probability\ndistributions while maintaining multimodality.\n