Summary
This paper offers a convergence proof for the stochastic optimization problem inherent in full-rank Gaussian variational inference when the log-density of the target is concave. The primary challenge of the convergence proof lies in managing the non-smoothness present in the entropy term of Gaussian VI. This issue is addressed in the current work by considering proximal and projected stochastic gradient descent. To evaluate the validity of the imposed assumptions, case studies involving Bayesian (generalized) linear regression problems are provided.
Strengths
I appreciate the clarity of the writing of the present work; it clearly descibes the scope of the problem studied and states the main challenge of showing the convergence. The layout of each section forms a smooth flow of the proof strategies, in addition to a nice organization of the mathematical proofs, which makes the reading enjoyable.
The case study provided in appendix (sec 7.3) is helpful for reasoning imposed assaumptions.
The proximal operator introduced for optimizing the covariance matrix is novel to me. Although it's a standard techniques in optimization literature---splitting non-strongly smooth (but closed convex) part of the objective function using proximal operator, it's the first time I see in Gaussian VI literature. (This could of course be due to my lack of knowledge in this field) Additionally, the weighted telescope summation used when $-logp$ is only convex(rather than strongly convex) is very interesting, which is a setting that rarely considered in literature as far as I'm aware.
Even though the paper limits the scope to dense Gaussian VI problem, I think the present techniques can be applied to general location-scale variational families (as long as the variance quadratic bounds still holds).
Weaknesses
In my view, the principal limitation of this work is the somewhat restricted scope of the Variational Inference (VI) problem it examines. Specifically, the variational family under consideration is: (1.) Gaussian (albeit full-rank), and (2.) the target distribution is log-concave or strongly-log-concave. Additionally, (3.) data subsampling on $\log p$ is not taken into account. The proof techniques necessitated by this setting, in my opinion, are fairly standard within the stochastic optimization literature. While I acknowledge that any relaxation of condition (2) would likely preclude anything beyond convergence to some stationary point, and that the quadratic variance bound is heavily reliant on condition (1) (which is probably not a major concern for the broader community in regards to more complex VI families), offering a convergence result when data subsampling is implemented could greatly increase the impact of this work. I would be inclined to raise the score to 8 if data subsampling was considered, or if existing proof techniques could easily address it (though I doubt the current approach is effective in this case, please correct me if I'm mistaken).
Regarding originality, even though I concur that explicit results may be lacking for this specific problem setting, I don't think this is the **first** optimization theory guarantee for full-rank Gaussian VI. For instance, [Xu & Campbell, 2022] studies the convergence of full-rank Gaussian VI without the assumption of a log-concave target (even though they utilize posterior asymptotics to somewhat reduce the underlying optimization problem on a strongly-log-concave target, they also employ a special scaling operator (Eq(8)) to handle the non-Lip-smoothness of the entropy term $\log \text{det}$). I recommend that the authors integrate an optimization analysis of the scaled stochastic gradient descent proposed in [Xu & Campbell, 2022], and carefully compare the results to those provided in [Xu & Campbell, 2022] within a strongly convex setting at least (ignoring the data asymptotics and merely assuming that $\log p$ is strongly-log-concave should align their work with the current setting).
Finally, I encourage the authors to provide additional case studies, perhaps involving more Bayesian GLMs or even Bayesian sparse regression with a Horseshoe prior. From what I understand, most of these models yield a log-concave target, but the application of the variance quadratic bound is unclear to me. If negative results were to appear, for example, some models not having a quadratically bounded gradient noise, these examples would still significantly benefit the community.
At the end, I hope to stress that despite the aforementioned weakness/limitations, I think this is a really good work for this niche research direction.
Reference:
[Xu&Campbell, 2022]: The computational asymptotics of Gaussian variational inference and the Laplace approximation, Statistics and Computing
Questions
1. Compare to [Xu&Campbell, 2022]: As I mentioned earlier, this is a very relavant past work on this topic. I wonder specifically, if we ignore all the asymptotic stuff and assumes $\log p$ is strongly-log-concave, does [Xu&Campbell, 2022] achieve similar convergence rate ($O(1/T)$) as the present paper?
2. I wonder if the proximal operator (line 264) has been noted in past VI literature? Or it is proposed by the author.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.