Variational inference (VI) is widely used as an efficient alternative to\nMarkov chain Monte Carlo. It posits a family of approximating distributions $q$\nand finds the closest member to the exact posterior $p$. Closeness is usually\nmeasured via a divergence $D(q || p)$ from $q$ to $p$. While successful, this\napproach also has problems. Notably, it typically leads to underestimation of\nthe posterior variance. In this paper we propose CHIVI, a black-box variational\ninference algorithm that minimizes $D_{\\chi}(p || q)$, the $\\chi$-divergence\nfrom $p$ to $q$. CHIVI minimizes an upper bound of the model evidence, which we\nterm the $\\chi$ upper bound (CUBO). Minimizing the CUBO leads to improved\nposterior uncertainty, and it can also be used with the classical VI lower\nbound (ELBO) to provide a sandwich estimate of the model evidence. We study\nCHIVI on three models: probit regression, Gaussian process classification, and\na Cox process model of basketball plays. When compared to expectation\npropagation and classical VI, CHIVI produces better error rates and more\naccurate estimates of posterior variance.\n