Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent

Summary

This paper focuses on the theoretical understanding and algorithmic contributions of Gaussian-Stein Variational Gradient Descent (Gaussian-SVGD) for Gaussian Variational Inference (GVI). The paper discusses the dynamics of Gaussian-SVGD and provides convergence rates in both mean-field and finite-particle settings when the target is Gaussian. It also shows that Gaussian-SVGD converges to the best Gaussian approximation to the target in KL-divergence for non-Gaussian targets. The authors propose two algorithmic frameworks for density-based and particle-based implementations of Gaussian-SVGD, which encompass various GVI algorithms proposed in the literature as special cases. The Gaussian-SVGD, and other related algorithms, are compared against one another on three examples: Gaussian target, Bayesian logistic regression, and a Gaussian mixture model. The authors highlight future directions for research, including establishing convergence results for particle-based Gaussian-SVGD algorithms under log-concave and more general targets, and exploring acceleration techniques.

Strengths

The paper is very well written and the mathematical content is clearly articulated. The authors have done a good job of relating this work to other similar works from the literature. The paper is well-structured and the authors' particular contributions to this line of research are clearly outlined. The technical detail, as far as I can see, is correct and the contribution of this work to the ML community, in particular the new convergence rates, are of significant interest. The paper is closely linked to other related works but is still highly original. The paper provides a nice balance between establishing important theoretical results and providing a usable algorithm for users to implement in practice.

Weaknesses

This is a nice piece of work and there are no areas where I can see significant weaknesses. However there are a few things that the authors may wish to consider to improve their paper: * Starting from a very simple question, "Why should I use Gaussian-SVGD?" I think the authors have skipped some of the motivation for this work. If you know that the target is Gaussian, then of course you would just fit a Gaussian to it and would not require a particle-based variational inference approach. If the target is non-Gaussian, then why would you use Gaussian-SVGD over standard SVGD? * It is not clear why regularized SVGD is introduced. What is the motivation here? Is it simply to show that you can derive a Gaussian-SVGD algorithm with the regularized Stein metric? * The kernel $K(\cdot,y)$ is introduced in Def 2.2 without being defined. Additionally, I think in eq. (1) you need to replace $\rho$ with $\rho_t$. * The simulation section is unfortunately a bit weak compared to other similar papers. The authors only consider three simple models and the details are largely glossed over, (e.g. what is $d$? How many particles? etc.). The authors seem to have run out of space and only provide a short simulation study for the logistic regression example. It would be better to include a more detailed simulation study in the main paper and to consider more challenging models. * Additionally, on the point of experiments, it is a bit surprising that SVGD is not also included in this list.

Questions

* This sort of relates to my previous question approach motivation. It is well-known that SVGD does not scale well to high-dimensional target distributions. Is it possible to show, either theoretically (i.e. results like Theorem 3.7) or empirically, that Gaussian-SVGD is better than SVGD when $d$ is large? * Similarly, what is known about how many particles are needed compared to SVGD? * Particle-based methods are preferred over standard variational methods which fit a Gaussian distribution to the target because of their nonparametric nature. In the experiments section, it seems like the Gaussian mixture model would have been the more interesting example to include in the paper, particularly in the context of how good the approximation to the target is compared against SVGD. Could this be included?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The paper could discuss more the theoretical limitation of this work. However, from an ethical perspective, I do not believe that this paper touches on issues that would have a negative societal impact.

Summary

This paper delves into an examination of Gaussian-SVGD through the analysis of mean-field PDE and discrete particle systems. It offers evidence for finite particle convergence and supports these findings with empirical validations.

Strengths

Through clear modeling, the article provides a comprehensive analysis for Gaussian-SVGD. The paper gives new interpretations for Wasserstein Gradient Flow and SVGD under linear systems, which are very interesting.

Weaknesses

The setup of the paper is relatively artificial, because in actual scenarios, linear kernels are rarely used to implement SVGD. The paper does not put much effort into justifying the significance of Gaussian variational inference, such as making comparisons with other distribution classes. If there were clear conclusions, it would have a significant impact on the significance of this paper.

Questions

Can you elaborate some advantages of Gaussian-SVGD compared with RBF-SVGD?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

The setup of the article is somewhat artificial. It would be more meaningful if the significance of this linear system could be demonstrated.

Summary

This paper studies the Stein variational gradient descent and its variants with a bilinear kernel on the space of Gaussian measures. The authors prove the rate of convergence of the dynamics, proposed finite particle algorithms and proved a uniform-in-time propagation of chaos, and finally prove the convergence rate of the finite particle algorithms.

Strengths

1. The study of the dynamics properties of SVGD and its variants on Gaussian space seems to be interesting, which is a natural analog to the Bures-Wasserstein gradient flow. This paper gives a comprehensive analysis of such algorithms, including the well-posedness of the dynamics and the rate of convergence. 2. The proof and the results seem to be correct. I didn't check all the details in the proof, but both the proof and the results make sense to me. 3. The paper is in general well-written, the theorems are stated properly, and I think it is not hard for readers to understand the main idea.

Weaknesses

I do not observe an obvious weakness of this paper, but I do have a few questions, please refer to the "Questions" part.

Questions

1. Intuitive explanation of the convergence rate: as proved in the paper, the rate of convergence of SVGD for centered Gaussian is $O(e^{- t})$, which do not depend on $\lambda$. This seems to be interesting, but I'm wondering if there's an intuitive explanation for this result? The dependence of $O(e^{- t/\lambda})$ in the Wasserstein gradient flow case makes more sense to me intuitively, since larger $\lambda$ will give a more "flat" potential. 2. Regarding the non-Gaussian target: - It is shown in Theorem 4.1 that when $\rho_{\theta^*}$ be the unique Gaussian measure that minimize $D_{KL}(\rho_{\theta^*}||\rho^*)$, then the Gaussian-SVGD will converge to $\rho_{\theta^*}$. But when there are multiple $\rho_{\theta^*}$ that achieve the minimization, will the dynamic converge to one of them? - Does the particle dynamics Algorithm 2 yield a similar uniform-in-time propagation of chaos results as in Theorem 3.7 with the same rate for Gaussian-SVGD? Minor points and typos: 1. Some ambiguity in definitions： - The definition of the kernels on page 3 (lines 111-118) is not entirely clear to me, in particular, what is $\mu$ and $\Sigma$ here? If I understand the latter part correctly, I think $\mu$ and $\Sigma$ will be the mean and covariance of $\rho_t$, which is actually time variant. I suggest the authors clarify this point in the definition. - The same problem appears in equation (26) when defining $\widehat{\nabla V}$, since $\widehat{\nabla V}$ depends on $t$, which is a time-dependent vector field. 2. Definition 2, line 101: $G_{\rho}^{Wass}$ should be $G_{\rho}^{Stein}$

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No limitations and potential negative societal impact.

Summary

This work performs a theoretical investigation of Gaussian SVGD, a special case of SVGD restricted to the submanifold of Gaussian densities by means of a bilinear kernel. The authors characterize the mean-field dynamics of Gaussian SVGD, with a particular focus on Gaussian targets and obtain finite-particle guarantees in certain settings. Moving beyond Gaussian targets, the authors show that the dynamics of mean-field Gaussian SVGD converges to the best Gaussian approximation (In KL divergence) to the target density, and obtain convergence rates for the mean-field dynamics with the target density is log-smooth and log-concave

Strengths

The work provides a thorough treatment of the mean-field dynamics of Gaussian SVGD. The experimental evaluation is also satisfactory.

Weaknesses

My primary concerns are as follows : 1. Most of the results presented in this work consider continuous-time mean-field dynamics under the assumption of a Bilinear kernel and Gaussian target density. In my opinion, these assumptions adversely impact the significance of these results, as they fail to explain one of the major appeals of SVGD, i.e., the fact that it provides a non-parametric approximation to a large class of target densities. (e.g. [1] shows discrete-time mean-field guarantees for subgaussian densities and [2] extends it to the class of densities satisfying a generalized Talagrand’s inequality). While the exponentially fast convergence guarantees for the mean-field dynamics are seemingly appealing, they fall short of providing a satisfactory explanation of the behavior of SVGD in practically relevant settings. 2. Despite the simplifying assumption of Gaussian targets and bilinear kernels, the work does not provide any quantitative discrete-time finite-particle convergence rates. To the best of my understanding, Theorem 3.9 is the only result that considers finite-particle discrete-time dynamics, but it does not provide any quantitative rates. 3. It is not clear why one should prefer Gaussian SVGD over the Bures-Wasserstein Gradient Flow [3] since the only setting where it seems to outperform Bures-Wasserstein Gradient Flow is that of centered Gaussians, which is a highly restrictive condition. 4. The results in Section 4 leave much to be desired. Considering the fact that the best known results for Bures-Wasserstein gradient flow [3] cover both log-concave and strongly log-concave densities (see also [4] for a computable JKO discretization of the Bures-Wasserstein flow with convergence guarantees for log-concave and log-strongly concave densities), it is not at all clear what the benefits of Gaussian SVGD are when the target is not Gaussian. 5. The overall presentation of the results requires a lot of work. For instance, it is not clearly stated where exactly in the Appendix each Theorem is proved. 6. In addition to the limited applicability of these results in explaining the behavior of SVGD (as the most practically relevant case is that of non-logconcave targets) and the absence of satisfactory finite-particle discrete-time guarantees, I am somewhat concerned about the technical novelty of these results. To the best of my understanding, It seems like the mean-field continuous-time guarantees of this work can be easily obtained from the well-established results on SVGD [5,6] by restricting to the (finite-dimensional) Gaussian submanifold. While limited theoretical contribution in itself is not a major weakness, I find it difficult to recommend acceptance given the unsatisfactory practical usefulness of these results. [1] Salim et. al., “A Convergence Theory for SVGD in the Population Limit under Talagrand's Inequality T1” [2] Sun et. al., “Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Condition” [3] Lambert et. al., “Variational inference via Wasserstein gradient flows” [4] Diao et. al., “Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space” [5] Duncan et. al., “On the geometry of Stein variational gradient descent” [6] Liu, “Stein Variational Gradient Descent as Gradient Flow”

Questions

1. Could you obtain discrete-time finite-particle rates for Gaussian SVGD under the settings considered in your results. If not, could you highlight what would be the technical challenges involved in obtaining such a result? 2. Could you clarify why the mean-field continuous-time Gaussian SVGD dynamics is of interest, when (to the best of my knowledge) : 1) It does not satisfactorily explain the behavior of SVGD in practical scenarios, 2) It is outperformed by Bures-Wasserstein Gradient Flows (as well as the practically implementable Bures-JKO scheme) in most settings 3. Could you comment on the technical novelty of these results? To the best of my understanding, the mean-field guarantees can be obtained by adapting the existing analysis of the Gradient Flow induced by SVGD [5,6] to the submanifold of Gaussian densities (a technique which is well-established for Wasserstein gradient flows in [3]). 4. I would recommend reworking the overall presentation. Explicitly stating where each theorem is proven in the Appendix would be a good starting point.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

I found the discussion on the limitations of this work to be somewhat unsatisfactory. The authors state that limitations are discussed on Section 6 (Other Related Works) and Section 7 (Conclusion). Section 6 discusses some of the prior literature and Section 7 highlights some future directions. Neither of these sections adequately discuss the limitations of this work (e.g. applicability is primarily restricted to centered Gaussian targets for most of the results, finite-particle discrete-time rates are absent in most settings)

Summary

This work studies the behavior of Gaussian-SVGD (SVGD projected to the space of Gaussian distributions, using bilinear kernels) and its variants. The authors studied the behavior of the mean-field PDEs and established for Gaussian targets finite-particle convergence results which are significantly better than previous results for nonparametric kernels and targets. They have also derived density- and particle-based implementations of Gaussian-SVGD for general targets, which are shown to generalize recent works on Gaussian variational inference (GVI) with algorithmic guarantees. One of the proposed algorithms outperform recent works on a Bayesian logistic regression experiment.

Strengths

- The theoretical results appear interesting (to me as a non-expert, and note that I didn't check the proofs): the mean-field results sometimes improve over recent works on Gaussian variational inference with algorithmic guarantees, and the finite-particle results apply to standard SVGD, albeit with the other restrictions. - The discussions on connections between Gaussian SVGD and existing GVI approaches are also interesting, and one of the proposed methods appear promising empirically.

Weaknesses

- While the restrictions to Gaussian families and affine kernels are similar to some of the recent works, it inevitably limits the scope of this work as much of the interest around SVGD arises from its flexibility. - While Gaussian variational inference in general has demonstrated competitive empirical performance, it is unclear if the proposed approach maintains this property, as it is not clear if the more complex algorithms interact nicely with common modifications such as minibatching. This is also relevant because while some of the recent works (e.g., [17, 36]) also studied alternative GVI approaches, they come with full algorithmic guarantees, whereas for the present method there is no qualitative convergence guarantees for the discrete-time, finite-particle case.

Questions

For the experiments it would be helpful to compare with ordinary gradient descent on the variational parameters, to provide some ideas on the practical utility.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

update/feedback

Dear Reviewer EWuB, As the deadline for the discussion phase is fast approaching, we were curious if you had any feedback for our response. Please also let us know if you have any further questions. Thank you and looking forward to hearing from you. Sincerely, Authors

update/feedback

Dear Reviewer 1EPQ, As the deadline for the discussion phase is fast approaching, we were curious if you had any feedback for our response. Please also let us know if you have any further questions. Thank you and looking forward to hearing from you. Sincerely, Authors

Reviewer 1EPQ

Dear AC, Apologies for the second reminder. However, Reviewer 1EPQ hasn't responded yet to our rebuttal, even after sending a reminder. While they provided the lowest score to your submission, we have answered all their questions and clarified several misunderstandings on their part. Furthermore, the discussion period is inching towards the deadline (Aug 21st) with also the weekend in between, which makes us even more anxious about their non-response. We are sending this email to you and also tagging the reviewers in the hope that we get a response. In the event that Reviewer 1EPQ doesn't respond in a timely manner, we would greatly appreciate it if you could ignore their review when making the final decisions regarding our submission. Thanks for your time and consideration. Sincerely, Authors