Connections and Equivalences between the Nyström Method and Sparse Variational Gaussian Processes

We investigate the connections between sparse approximation methods for\nmaking kernel methods and Gaussian processes (GPs) scalable to large-scale\ndata, focusing on the Nystr\\"om method and the Sparse Variational Gaussian\nProcesses (SVGP). While sparse approximation methods for GPs and kernel methods\nshare some algebraic similarities, the literature lacks a deep understanding of\nhow and why they are related. This may pose an obstacle to the communications\nbetween the GP and kernel communities, making it difficult to transfer results\nfrom one side to the other. Our motivation is to remove this obstacle, by\nclarifying the connections between the sparse approximations for GPs and kernel\nmethods. In this work, we study the two popular approaches, the Nystr\\"om and\nSVGP approximations, in the context of a regression problem, and establish\nvarious connections and equivalences between them. In particular, we provide an\nRKHS interpretation of the SVGP approximation, and show that the Evidence Lower\nBound of the SVGP contains the objective function of the Nystr\\"om\napproximation, revealing the origin of the algebraic equivalence between the\ntwo approaches. We also study recently established convergence results for the\nSVGP and how they are related to the approximation quality of the Nystr\\"om\nmethod.\n