L-C2ST: Local Diagnostics for Posterior Approximations in Simulation-Based Inference

Many recent works in simulation-based inference (SBI) rely on deep generative models to approximate complex, high-dimensional posterior distributions. However, evaluating whether or not these approximations can be trusted remains a challenge. Most approaches evaluate the posterior estimator only in expectation over the observation space. This limits their interpretability and is not sufficient to identify for which observations the approximation can be trusted or should be improved. Building upon the well-known classifier two-sample test (C2ST), we introduce L-C2ST, a new method that allows for a local evaluation of the posterior estimator at any given observation. It offers theoretically grounded and easy to interpret -- e.g. graphical -- diagnostics, and unlike C2ST, does not require access to samples from the true posterior. In the case of normalizing flow-based posterior estimators, L-C2ST can be specialized to offer better statistical power, while being computationally more efficient. On standard SBI benchmarks, L-C2ST provides comparable results to C2ST and outperforms alternative local approaches such as coverage tests based on highest predictive density (HPD). We further highlight the importance of local evaluation and the benefit of interpretability of L-C2ST on a challenging application from computational neuroscience.

Paper

References (49)

Scroll for more · 37 remaining

Similar papers

Peer review

Reviewer Ppab6/10 · confidence 4/52023-06-23

Summary

This paper defines a new diagnostic to check posteriors learned from simulation-based inference. The main issue in this setting is that one does not have access to likelihoods, and so one needs diagnostics that do not rely on being able to calculate likelihoods. Here the authors present L-C2ST, which essentially tries to train a classifier to distinguish the _joint_ space over both parameters _and_ observations into whether a given point was more likely under the true joint distribution, $p(x, \theta)$ or under a variational approximation $q(\theta | x) p(x)$. It turns out that a Bayes-optimal (probabilistic) classifier will assign equal weight to the two distributions for all $\theta$ for a given $x_0$ if and only if $q(\theta | x_0) = p(\theta | x_0)$ (up to some technical considerations). As a result, the authors propose training such a classifier and then seeing how much its weights deviate from $0.5$ and using that as a measure of how well $q$ matches $p$. Finally, the authors develop a hypothesis testing framework (to account for randomness in the trained classifier) to determine how significant a given deviation is.

Strengths

* Trustworthy checks of how good a given variational posterior is for real settings where we do not have access to the true posterior are sorely needed, and this work is an important step in that direction. * The paper is well-written, well-motivated, and clearly explained. * The theoretical results complement and motivate the proposed algorithm well (but see below for some minor technical concerns). * The proposed algorithm is extremely simple to describe, making it practical and elegant.

Weaknesses

* My major concern is that I am not totally convinced that learning the proposed classifier is any easier than learning the variational posterior. As such, I would be concerned that diagnostics showing a good posterior fit could be driven by a far-from-optimal classifier. My intuition for why this should be a hard problem is as follows: for _any_ amortized posterior $q(\theta | x)$, if we can learn a Bayes-optimal classifier $d^*$, then one could recover the true posterior for any $x$ via $p(\theta | x) = q(\theta | x) / (1-d^*(\theta, x))$. As a result, learning $d^*$ for _any_ $q$ is equivalent to learning the true posterior. This is somewhat borne out in the third column of figure 2, where one needs roughly the same number of samples to learn a good posterior as one needs in order to learn a good classifier. I'm wondering if somehow the proposed approach is essentially equivalent to training two different amortized variational posteriors on independent data and then seeing how well they match. Some further simulations showing that one problem is easier than the other would alleviate some of my concerns. I should also say that I think that even if my concerns are valid, the diagnostic is still interesting and useful. * I felt that the benchmarking could have been improved slightly. In particular, some empirical relationship between the proposed measure of fit and existing measures would be nice. For example, the authors could take the true posterior in a toy model where it is known, generate $q$ by perturbing that posterior in a systematic way and compare how increasingly large perturbations affect both the L-C2ST measure as well as things like KL, reverse KL, TV, and so on. The results in Figures 1 and 2 hint at this, but there are a lot of moving parts (especially in Figure 2) making it difficult to determine whether to contribute power (or lack thereof) to failure of the SBI (i.e., $q$ is bad) or the classifier (i.e. $r$ is good). * A minor technical point throughout the proofs of the theorems is that they rely on the null hypothesis $\mathcal{H}_0$ holding on a set of strictly positive measure. That is, if $\mathcal{H}_0$ fails on a set of measure zero then the proofs would break down. In particular, in that case $\mathcal{A}$ would have measure zero and the final integral after line 454 would be exactly zero. I tend to feel that failures on sets of measure zero have no practical implications, and so this is a minor point. But, if one is going to state theorems and proofs, it would be good to be rigorous. * Similarly, Lemma 1 should not be phrased in terms of "points" as those would have zero measure, and instead should say "If there exists a set $\mathcal{A} \subseteq \mathcal{S}$ with positive measure such that $p(\theta) > q(\theta)$ for all $\theta \in \mathcal{A}$, then there exists a set $\mathcal{B}$ with positive measure such that $p(\theta') < q(\theta')$ for all $\theta'\in \mathcal{B}$. * Finally, there are similar (and equally minor) issues appearing in the proofs of Theorems 2 and 3. Typos: * Line 103: "may lack of" --> "may lack" * Line 240: "in average" --> "on average" * Line 251: Missing period before "The local PP-plot"

Questions

* Using the same $X_n$'s in Algorithm 1 for draws from $p(\cdot | X_n)$ and $q(\cdot | X_N)$ introduces dependencies in the training data for the classifier. Can that somehow be leveraged when training the classifier (similar to how in linear regression, we would want to perform GLS instead of OLS to increase the power when data points have shared structure like this). Along similar lines, if it is much cheaper to sample from $q(\theta | x)$ than it is to sample from $p(x, \theta)$, then could it help to somehow incorporate more samples from $q(\theta | X_n)$ in the algorithm (or perhaps the whole distribution somehow)? Obviously this would require accounting for the dependencies (perhaps via some kind of downweighting) in the training of the classifier. This is obviously not necessary to implement for this paper, but I am curious to hear the authors thoughts. * Algorithm 1 also seems computationally intensive due to the permutation step. Would it be possible to do something like split-sample conformal inference to speed up the p-value calculations? Again, obviously not necessary for the paper, I'm just curious.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

I don't see any potential for direct societal harm from this work. The authors may wish to include a limitations section that highlights failure modes of their method.

Reviewer ukES5/10 · confidence 4/52023-07-02

Summary

The authors present a new diagnostic tool for simulation-based inference. The method learns the C2ST without knowledge of the true posterior distribution. If the density estimator is a normalizing flow, the authors propose to perform the classification in latent space.

Strengths

**Originality**: The method tackles and important issue for simulation-based inference. The method is new and potentially very useful. The extension for normalizing flows is interesting and creative. If L-C2ST works well, I could see this be adopted by the community. **Quality**: The method is rigorously derived. **Clarity**: Everything is explained well and with sufficient detail. Figures are clear.

Weaknesses

**Quality**: My main concern with this paper is that it does not sufficiently demonstrate how well the method actually works. In particular, I believe that the following crucial questions remain unanswered: - Does the method indeed capture the true C2ST for different `x`? For example, the (decent) TPRs shown in figure 2 could also be obtained if L-C2ST converges onto the average C2ST across x_o. The central claim of the authors is that L2CT is local, but this claim needs substantially more empirical evidence. To address this concerns, I would (for example) suggest to add a scatter plot of L-C2ST vs true C2ST on a task where the ground truth is available by MCMC or analytically - How well does the method scale with parameter and data dimensionality Since C2ST is trained on theta and x as input, I could imagine it to not scale well to high-D x or theta (or require **even more** simulations). All tasks in the paper are low-D though. I suggest that the authors add an additional benchmark task with higher data and parameter dimensionality. - Is the method actually better than local-HPD On one task, L-C2ST performs better than local-HPD and worse on another one. Which method should be preferred? The paper also claims that local-HPD is much less efficient than C2ST but this is never empirically shown. In particular, while a naive implementation of local-HPD might be slow, amortizing over the confidence alpha should be trivial and make local-HPD as fast as C2ST. Please correct me if I am wrong or, otherwise, clarify this in the paper. To address this, I recommend adding an analysis of the runtimes of the algorithms and ideally have more than only two benchmark tasks. I believe that the above concerns are critical to the usefulness of the method, but I would be willing to significantly increase my score if these things are addressed.

Questions

- Do I understand correctly that, for large N_H, one has to train the classifier multiple times? If yes, couldn’t this make the method less efficient than local-HPD?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors state limitations of their method.

Reviewer Esye7/10 · confidence 2/52023-07-04

Summary

An algorithm for evaluating amortized posterior estimators $q(\theta|x)$ in simulation-based inference (SBI) is proposed. In the setting considered, we have the ability to sample $p(\theta)$ and to sample $p(x|\theta)$ but not to evaluate its density. The algorithm modifies the common classifier two-sample test by learning to discriminate between samples $(\theta,x)$ sampled ancestrally (though simulation) from $p(\theta,x)$ and those where $\theta$ is then resampled from $q(\theta|x)$. An extension to normalizing flows is also considered, where the discriminator works in the latent space rather than in the theta space. There are strong results on several SBI benchmarks and real-world problems in terms of statistical power and runtime.

Strengths

Please note that this is a relative outsider's point of view, as I do not directly work on SBI. I find the paper well-written and was able to understand the main points and the math. In particular, the first few pages are a good introduction to the problem area that will be accessible to any reader who is familiar with Bayesian inference and hypothesis testing. The results and their significance are explained well and give intuitions about why the proposed algorithm works well on the chosen problems. The provided code is a helpful addition.

Weaknesses

I do not see any major weaknesses, but have a few questions. - Could you please comment on the scalability of the algorithm? What difficulties do you foresee when scaling to high-dimensional theta or problems when the prior is wide? (Same question about high-dimensional observations. Is more risk of overfitting to the class split of $x$?) - It would be interesting to understand better the benefit of the -NF version of the algorithm beyond the reasons described in L180, which seem to explain only computation cost improvements. Is it expected that the classification boundary is smoother in the latent space than in the theta space? Maybe this could be illustrated, e.g., on the two moons example.

Questions

Please see above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

yes

Reviewer ukES2023-08-10

Concerns on accuracy of l-C2ST

Thanks a lot for the detailed response and for the additional experiments, they helped in clarifying Q2, Q3 (although I still think that the gains over HPD are rather small), and Q4. Q1: However, I am worried about the results shown in the new pdf. In Figure 2, for any given training budget N_train, the oracle C2ST seems to correlate very weakly (if at all) with l-C2ST. How, then, should l-C2ST indeed be a **local** measure for calibration?

Authorsrebuttal2023-08-13

Thank you for your response. **General remark on the performance of $\ell$-C2ST w.r.t. *local*-HPD:** Our goal with $\ell$-C2ST is to present a new, **alternative** method to *local*-HPD, which to our knowledge is the only other existing local diagnostic (please see **Remark 1** in the response to Reviewer `Ppab`, for further motivations regarding our choice to base our work on **C2ST**). Our research and experiments show that $\ell$-C2ST is theoretically valid and works on several datasets, sometimes even outperforming *local*-HPD. It is true that our method does not work as well on all examples, but a big advantage of $\ell$-C2ST is that one can directly use literature and advancements from the *classification* field in order to adapt and enhance it for any given dataset/task (e.g. see response to **Question 1** for Reviewer `Ppab`). This makes $\ell$-C2ST a competitive alternative with great potential. **Regarding the accuracy of $\ell$-C2ST and it actually being a **local** method:** - The method is based on solid mathematical reasoning and it is **local by definition**. Moreover, it has **if and only if** guarantees for consistency, a remarkable property for this kind of statistical test, which *local*-HPD does not have. - The suggested experiment with the scatterplots showing the "accuracy" of $\ell$-C2ST are really interesting! Our results in the attached PDF show that there is some correlation between the values of test statistics for $\ell$-C2ST and oracle-C2ST. This correlation becomes weaker when $N_{\mathrm{train}}$ becomes larger, since the test statistics in these cases tend to zero and can start to be confused with noise. We have carried out standard tests for the statistical significance of the correlation indices between the scores. The results are shown at the end of this comment. - The fact that $\ell$-C2ST does not track the exact values of the oracle C2ST on all examples and for every observation does not mean necessarily that it is not a good method for **detecting local posterior inconsistencies**. Our results on Figure 3 and Figure 4 (those on the `JRNMM` data) clearly show how the statistical test varies for each choice of observation $x_{\mathrm{o}}$. Also, it should be noted that while we want to be as close as possible to the oracle C2ST, the more adapted performance metric, measuring the capacity of **detecting local inconsitencies**, is the statistical error of the test (i.e. power and type 1 error): Figure 2 (resp. 1) in the main paper (resp. attached PDF) show that in more than one example, $\ell$-C2ST reaches maximum power and outperforms *local*-HPD. **P-values of the Pearson test of non-correlation between the oracle C2ST and the $\ell$-C2ST( / -NF) MSE test statistic**. Obtained for 100 observations (as plotted in Figure 2.b.) using `scipy.stats.pearsonr`. Bold values indicate the cases for which the Pearson test rejects the null hypothesis of non-correlation with 95% confidence: `Two Moons`: - $N_{\mathrm{train}}=100$: **10e-27** / **10e-9** - $N_{\mathrm{train}}=1000$: **0.0006** / 0.19 - $N_{\mathrm{train}}=10000$: **10e-16** / **10e-11** - $N_{\mathrm{train}}=100000$: 0.052 / **10e-5** `SLCP`: - $N_{\mathrm{train}}=100$: **0.0001** / 0.12 - $N_{\mathrm{train}}=1000$: **0.0009** / **0.03** - $N_{\mathrm{train}}=10000$: 0.31 / 0.82 - $N_{\mathrm{train}}=100000$: 0.21 / 0.40 `Gaussian Mixture`: - $N_{\mathrm{train}}=100$: **10e-8** / **10e-12** - $N_{\mathrm{train}}=1000$: **10e-7** / **0.01** - $N_{\mathrm{train}}=10000$: **0.006** / 0.35 - $N_{\mathrm{train}}=100000$: **10e-14** / **0.006** `Gaussian Linear Uniform`: - $N_{\mathrm{train}}=100$: **10e-13** / **10e-12** - $N_{\mathrm{train}}=1000$: 0.07 / **10e-9** - $N_{\mathrm{train}}=10000$: 0.42 / **0.002** - $N_{\mathrm{train}}=100000$: 0.68 / 0.87 `Bernoulli GLM`: - $N_{\mathrm{train}}=100$: **10e-8** / **10e-5** - $N_{\mathrm{train}}=1000$: **10e-10** / **0.0002** - $N_{\mathrm{train}}=10000$: 0.67 / 0.18 - $N_{\mathrm{train}}=100000$: 0.39 / 0.31 `Bernoulli GLM Raw`: - $N_{\mathrm{train}}=100$: **0.03** / 0.37 - $N_{\mathrm{train}}=1000$: **10e-8** / **0.0004** - $N_{\mathrm{train}}=100$: **0.0001** / **0.04** - $N_{\mathrm{train}}=100$: 0.92 / 0.06

Reviewer ukES2023-08-14

Quick response

(Longer response later, but) would you mind also sharing the correlation coefficient itself (not only the p-value and confidence interval). Thanks!

Authorsrebuttal2023-08-14

Of course! Below you will find the **Pearson correlation coefficients** associated to the above p-values. This statistic varies between -1 and 1, 0 indicating zero-correlation between the oracle C2ST and $\ell$-C2ST( / -NF). In bold the cases for which the associated p-value is $<0.05$ (see above results). `Two Moons`: - $N_{\mathrm{train}}=100$: **0.84** / **0.57** - $N_{\mathrm{train}}=1000$: **0.34** / -0.13 - $N_{\mathrm{train}}=10000$: **0.71** / **0.61** - $N_{\mathrm{train}}=100000$: 0.20 / **0.40** `SLCP`: - $N_{\mathrm{train}}=100$: **0.39** / 0.16 - $N_{\mathrm{train}}=1000$: **0.34** / **0.23** - $N_{\mathrm{train}}=10000$: -0.11 / 0.02 - $N_{\mathrm{train}}=100000$: 0.13 / -0.09 `Gaussian Mixture`: - $N_{\mathrm{train}}=100$: **0.52** / **0.63** - $N_{\mathrm{train}}=1000$: **0.49** / **0.25** - $N_{\mathrm{train}}=10000$: **0.27** / 0.09 - $N_{\mathrm{train}}=100000$: **-0.68** / **0.27** `Gaussian Linear Uniform`: - $N_{\mathrm{train}}=100$: **0.64** / **0.63** - $N_{\mathrm{train}}=1000$: 0.18 / **0.55** - $N_{\mathrm{train}}=10000$: 0.08 / **0.31** - $N_{\mathrm{train}}=100000$: -0.04 / 0.02 `Bernoulli GLM`: - $N_{\mathrm{train}}=100$: **0.54** / **0.42** - $N_{\mathrm{train}}=1000$: **0.58** / **0.37** - $N_{\mathrm{train}}=10000$: -0.04 / 0.14 - $N_{\mathrm{train}}=100000$: -0.09 / -0.10 `Bernoulli GLM Raw`: - $N_{\mathrm{train}}=100$: **0.22** / 0.09 - $N_{\mathrm{train}}=1000$: **0.53** / **0.35** - $N_{\mathrm{train}}=100$: **0.37** / **0.20** - $N_{\mathrm{train}}=100$: -0.01 / -0.19

Reviewer ukES2023-08-14

Response

Thanks for the additional explanation and experiments! Another quick question upfront: how large is $N_{cal}$ in the experiments above? I.e., how many simulations is the classifier trained on? Apart from this, I mostly agree with what the authors wrote, but I do disagree with this sentence: ```[oracle C2ST not matching l-C2ST] does not mean necessarily that it is not a good method for detecting local posterior inconsistencies.``` Of course, it does not have to track the oracle C2ST perfectly to be useful. I also agree that, even with a correlation coefficient of zero, the method might estimate the **average** C2ST and still have decent statistical power compared to HPD. However, I do think that the claim of the method being local is extremely misleading in scenarios where the correlation between the oracle C2ST and l-C2ST is almost 0. Some of the reported correlations are very low and warrant an extended paragraph on limitations of the method which highlights regimes in which the classifier does not report **meaningful** local C2ST estimates and essentially produces noisy samples of the average C2ST (or shows very weak correlation with the true oracle C2ST). For example, I believe that it might be useful to provide ballpark estimates of how large $N_{cal}$ must be in comparison to $N_{train}$ in order to reach a certain correlation coefficient. Nonetheless, I have increased my score to 5 because I believe that the method can be useful. I hope that, if the paper gets accepted or resubmitted, the authors will add the analyses from this review process to the paper and will add a discussion of the limitations on how faithful the estimated l-C2ST is. Nonetheless, congrats on nice (and hopefully useful) paper!

Authorsrebuttal2023-08-14

Hello! Quick response: $N_{\mathrm{cal}} = 10^4$ in the experiment above and in all experiments where $N_{\mathrm{train}}$ is varying (from the main paper and the PDF attached to this rebuttal). Thank you for the constructive questions and remarks, they are much appreciated. We are glad that we could answer most of your questions and will discuss your last comments. We will make sure to include the additional experiments of this rebuttal, as well as a paragraph on the limitations / failure modes of $\ell$-C2ST!

Reviewer Ppab2023-08-10

Thank you very much for the detailed response. I found the paper interesting and enjoyable, but I stand by my initial score.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC