Summary
The authors introduce a novel algorithmic approach to fitting semi-implicit variational approximations. This method is based on discretizing a suitable gradient flow, and the paper provides a comprehensive theoretical analysis to support it. The approach, called Particle Variational Inference (PVI), directly optimizes over the space of probability distributions for the mixing distribution rather than optimizing a specific parametric form for the mixing distribution, as in previous SIVI approaches.
Numerical examples demonstrate improvements compared to existing semi-implicit variational inference methods. The paper provides a solid theoretical foundation and a practical algorithm with beneficial properties over existing methods in the literature.
Strengths
This paper is a very strong submission, and I believe it is worthy of acceptance at this conference. The strengths that I would like to highlight are the following:
- **High degree of novelty:** Semi-implicit variational inference (SIVI) methods have existed for some time. Existing approaches faced challenges due to the nature of their design, often leading to optimizing bounds on the ELBO or difficult algorithms. This paper's essential contribution is constructing an objective function that can be directly optimized by its gradient flow. This is a significant advancement in the SIVI literature, enabling direct optimization of the ELBO.
- **Theoretical foundations:** The paper provides a solid theoretical analysis of the proposed method. This includes the study of a related gradient flow, establishing existence and uniqueness of solutions, and providing propagation of chaos results. These theoretical underpinnings give a rigorous basis for the practical algorithm.
- **Practical algorithm:** The particle variational inference (PVI) algorithm is derived as a practical implementation by discretizing the gradient flow of the proposed objective function. This direct link between theory and practice is a strength of the paper, as it provides a clear path from the mathematical formulation to a computationally feasible method.
Weaknesses
The following points, particularly the limited experimental validation, prevent me from giving the submission a higher rating (8/9):
- **Limited scope of numerical experiments:** The paper's major weakness lies in its experimental section, which represents the minimum set of experiments for an acceptable paper. Specifically: a. Lack of empirical demonstration of expressiveness: Despite claiming in Section 2 that PVI can learn potentially more expressive variational approximations than other SIVI methods, no experiments empirically demonstrate this advantage. b. Insufficient exploration of optimization stability: The paper misses an opportunity to address a key challenge in SIVI methods - the difficulty in getting objective functions/algorithms to converge (or meaningfully work). Experiments showing that PVI provides consistently more stable optimization than other SIVI methods or converges on complex models where other SIVI methods struggle would have significantly strengthened the paper. Additionally, comparing performance accuracy on Bayesian Neural Networks (BNNs) for different SIVI methods doesn't show meaningful advantages to this method.
- **Theoretical-practical gap:** The theoretical analysis is for a modified gradient flow (with γ > 0), but the practical algorithm uses γ = 0. However, this discrepancy is mitigated by the authors' justification that this approach is relatively common in the literature, and empirical results did not show obvious discrepancies between theory and practice.
- **Clarity in explaining the flexibility of PVI's variational approximation:** The novelty in PVI's approach - not optimizing a specific parametric form of the mixing distribution - was not immediately apparent to me from the explanations in Sections 2 and 3 upon first read. In particular, the last sentence in Section 2 (lines 103-105) is confusing. While it becomes more understandable after reading Section 3, the intuition behind why Q_{YuZ} might not reduce to other parameterizations when fit with PVI is not fully explained.
Questions
1. How sensitive is PVI to the choice of kernel and number of particles?
2. Can the theoretical analysis be extended to cover the γ = 0 case used in practice?
3. How does PVI scale to higher-dimensional problems or larger datasets?
4. Are there specific types of problems where PVI is expected to significantly outperform existing methods?
5. How does the choice of preconditioner affect PVI's performance?
Limitations
The authors have adequately addressed limitations.