Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS

Mutual information (MI) is an information-theoretic measure of dependency\nbetween two random variables. Several methods to estimate MI, from samples of\ntwo random variables with unknown underlying probability distributions have\nbeen proposed in the literature. Recent methods realize parametric probability\ndistributions or critic as a neural network to approximate unknown density\nratios. The approximated density ratios are used to estimate different\nvariational lower bounds of MI. While these methods provide reliable estimation\nwhen the true MI is low, they produce high variance estimates in cases of high\nMI. We argue that the high variance characteristic is due to the uncontrolled\ncomplexity of the critic's hypothesis space. In support of this argument, we\nuse the data-driven Rademacher complexity of the hypothesis space associated\nwith the critic's architecture to analyse generalization error bound of\nvariational lower bound estimates of MI. In the proposed work, we show that it\nis possible to negate the high variance characteristics of these estimators by\nconstraining the critic's hypothesis space to Reproducing Hilbert Kernel Space\n(RKHS), which corresponds to a kernel learned using Automated Spectral Kernel\nLearning (ASKL). By analysing the aforementioned generalization error bounds,\nwe augment the overall optimisation objective with effective regularisation\nterm. We empirically demonstrate the efficacy of this regularization in\nenforcing proper bias variance tradeoff on four variational lower bounds,\nnamely NWJ, MINE, JS and SMILE.\n