Summary
This paper introduces Pseudo-Likelihood Inference (PLI) as a novel method inside the simulation-based inference (SBI) family. SBI methods are used to estimate model parameters when the likelihood function is unknown or intractable. Existing SBI methods, such as Approximate Bayesian Computation (ABC) and Sequential Neural Posterior Estimation (SNPE), either approximate the likelihood or directly model the posterior. The contribution by the authors, PLI, combines neural approximation with ABC to achieve competitive performance in challenging Bayesian system identification tasks. The authors make use of integral probability metrics and introduce a smooth likelihood kernel with an adaptive bandwidth that is updated based on information-theoretic trust regions. This formulation allows for optimizing neural posteriors via gradient descent and does not rely on summary statistics. PLI performance improves the more data is available, particularly in scenarios involving stochastic simulations and multi-modal posterior landscapes. The effectiveness of PLI is evaluated through experiments conducted on four classical SBI benchmark tasks as well as a highly dynamic physical system. The results demonstrated particular advantages of PLI in handling stochastic simulations and multi-modal posterior landscapes.
Strengths
* PLI is a novel approach addressing the limitations of existing methods that achieves competitive performance in challenging Bayesian system identification tasks. In fact, the proposed method outperforms SNPEs and other likelihood-free models, particularly when more data is available.
* The usage of the adaptive bandwidth for the smooth likelihood kernel allows for better adaptation to different problem settings and improves the accuracy of parameter estimation.
* The method enables to directly use gradient descent optimizing neural posteriors using gradient descent directly on raw observations or data points without requiring any intermediate summarization step.
* The effectiveness of PLI is evaluated in multiple experiments conducted on four classical Simulation-Based Inference benchmark tasks and in a highly dynamic physical system, demonstrating its applicability across various domains.
Weaknesses
* The changes induced by replacing the KL divergence with the MMD and the Wasserstein distance should further explored. Consequences on changing the KL divergence for other metrics has been studied before in works such as [1], where an adversarial approach is used to evaluate the KL divergence and other divergences using only samples. I strongly suggest discussing this possibility in the article, since it seems that this alternative can be useful here.
* To provide more robust experimental results, when comparing different models I would suggest also using other proper scoring rules [2].
* Since efficiency is an important aspect when dealing with high-dimensional tasks, there is limited discussion about computational efficiency throughout the paper. Considering that likelihood-free inference methods can be computationally demanding, it would be valuable to discuss any potential limitations or considerations regarding computational resources required by PLI compared to other SBI approaches
* While the paper highlights PLI's advantages over SNPEs, it would be valuable to understand how PLI performs against other popular approaches s.a. Markov Chain Monte Carlo methods. A comparative analysis would provide a more comprehensive understanding of PLI's strengths and weaknesses relative to existing techniques.
* (_minor_) Due to the formulation of several function-space Bayesian inference models, if you deemed it fitting it could be interesting discussion on the function-space approaches such as [3-6]. I think most of these models can be encompassed as (S)NPE methods and that could enrich the discussion in the paper.
* (_minor_) Please correct typos, s.a. "posteriror" in line 143, "divergnce" in line 156.
* (_minor_) The red hyperlink boxes to equations appear with an extra line inside, which difficults reading. Please check this.
---
**References**:
[1] Santana, S. R., & Hernández-Lobato, D. (2022). Adversarial α-divergence minimization for Bayesian approximate inference. Neurocomputing, 471, 260-274.
[2] Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477), 359-378.
[3] Ma, C., & Hernández-Lobato, J. M. (2021). Functional variational inference based on stochastic process generators. Advances in Neural Information Processing Systems, 34, 21795-21807.
[4] Rodrı́guez Santana, S., Zaldivar, B., & Hernandez-Lobato, D. (2022, June). Function-space Inference with Sparse Implicit Processes. In International Conference on Machine Learning (pp. 18723-18740). PMLR.
[5] Ma, C., Li, Y., and Hernández-Lobato, J. M. (2019). “Variational implicit processes”. In: International Conference on Machine Learning, pp. 4222–4233.
[6] Sun, S., Zhang, G., Shi, J., and Grosse, R. (2019). “Functional variational Bayesian neural networks”. In: International Conference on Learning Representations.
Questions
* What are the limitations and potential challenges associated with implementing PLI? The paper presents PLI as an effective method for challenging Bayesian system identification tasks, but it is important to address potential limitations or practical considerations when applying this approach in real-world scenarios. For example, what computational resources are required by PLI? Are there specific problem domains where its performance may be less favorable?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.