KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support

We study the gradient flow for a relaxed approximation to the\nKullback-Leibler (KL) divergence between a moving source and a fixed target\ndistribution. This approximation, termed the KALE (KL approximate lower-bound\nestimator), solves a regularized version of the Fenchel dual problem defining\nthe KL over a restricted class of functions. When using a Reproducing Kernel\nHilbert Space (RKHS) to define the function class, we show that the KALE\ncontinuously interpolates between the KL and the Maximum Mean Discrepancy\n(MMD). Like the MMD and other Integral Probability Metrics, the KALE remains\nwell defined for mutually singular distributions. Nonetheless, the KALE\ninherits from the limiting KL a greater sensitivity to mismatch in the support\nof the distributions, compared with the MMD. These two properties make the KALE\ngradient flow particularly well suited when the target distribution is\nsupported on a low-dimensional manifold. Under an assumption of sufficient\nsmoothness of the trajectories, we show the global convergence of the KALE\nflow. We propose a particle implementation of the flow given initial samples\nfrom the source and the target distribution, which we use to empirically\nconfirm the KALE's properties.\n