Summary
The authors suggest a new objective function for simulation-based inference that adds a penalty term to the “expected score” objective that is used by many other works. This penalty term encourages the resulting posterior approximations to be well-calibrated in the sense that the $1-\alpha$ highest probability density regions contain the ground-truth parameter $100(1-\alpha)$% of the time. The penalty term is an extension of the Kolmogorov-Smirnov test statistic for goodness of fit between a $U(0,1)$ distribution and the rank statistics of eq. 7, which are asymptotically uniformly distributed when $p(\theta \mid x)$ is calibrated. Minimizing this penalty term alongside the usual objective function should yield posteriors that are calibrated while substantially different from the degenerate case of the prior (which is trivially calibrated).
Strengths
* The proposed penalty/regularization term is relatively lightweight computationally, and can be tacked on to many existing simulation-based inference algorithms.
* The intuition is simple; the construction of the statistics $\hat{\alpha}$ is clever, and the Kolmogorov-Smirnov test statistic is well-understood.
* The method is assessed on a variety of problems from the test suite of benchmarks provided by Lueckmann et al. (2021), and performs favorably in that the resulting posteriors are either calibrated or tend to be conservative.
Weaknesses
* Although synthetic likelihood and ABC approaches are mentioned in the introduction, the proposed method seems prohibitively costly in these scenarios, and seems geared toward amortized methods only, where eq. 7 can be computed rapidly.
* For ratio estimation in particular, the non-regularized NRE is better calibrated than the corrected version in some cases (e.g., spatial SIR), suggesting that it is at least possible the penalty term can worsen calibration.
* In practice, it appears that attempts at calibration often result in conservative rather than calibrated posteriors; while this is likely still preferable to many users to the alternative of *not* adding this correction, it suggests a bit of a misnomer.
Questions
* The method called neural posterior estimation (NPE) has been shown in other work to be equivalent to minimizing a forward KL divergence between exact and approximate posteriors (Reweighted Wake-Sleep (Bornschein and Bengio, 2015), and a similar work “Revisiting Reweighted Wake Sleep…” (Le et al., 2018)); it’s been argued in these that the resulting neural posterior estimates tend to be overdispersed or conservative as a result. 1) How does this impact the relevance of this work if the baseline is usually already conservative? 2) The NPE row of Figure 1 doesn’t seem to reflect this intuition in practice; is there any explanation for why this is so? Is NPE indeed the same as sleep-phase in reweighted wake-sleep in the experiments?
* What are the advantages and disadvantages of using either the sorting-based computation or the direct computation method? Is the use of the latter solely due to the belief that backprop through indicator functions is somehow worse? As $\hat{\alpha}$ itself requires indicator functions, I suppose the backprop operation can’t be too problematic. It would be interesting to see both methods implemented in the experiments.
* The form of eq. 9 suggests self-normalized importance sampling, but as $\hat{p}(\theta \mid x^*)$ is already normalized, this doesn't seem necessary. Maybe this is a misunderstanding of notation. To my reading, though, just the numerator of eq. 9 is a valid importance sampling estimator. Is so-called “standard” or self-normalized importance sampling nonetheless used?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Generally the settings where this work may be applied are clear, but as mentioned above discussing the use of the method in amortized vs. non-amortized settings may make more clear the costs associated with with this method when $\hat{p}(\theta_j \mid x^*)$ can’t be computed quickly.