Summary
This paper defines a new diagnostic to check posteriors learned from simulation-based inference. The main issue in this setting is that one does not have access to likelihoods, and so one needs diagnostics that do not rely on being able to calculate likelihoods. Here the authors present L-C2ST, which essentially tries to train a classifier to distinguish the _joint_ space over both parameters _and_ observations into whether a given point was more likely under the true joint distribution, $p(x, \theta)$ or under a variational approximation $q(\theta | x) p(x)$. It turns out that a Bayes-optimal (probabilistic) classifier will assign equal weight to the two distributions for all $\theta$ for a given $x_0$ if and only if $q(\theta | x_0) = p(\theta | x_0)$ (up to some technical considerations). As a result, the authors propose training such a classifier and then seeing how much its weights deviate from $0.5$ and using that as a measure of how well $q$ matches $p$. Finally, the authors develop a hypothesis testing framework (to account for randomness in the trained classifier) to determine how significant a given deviation is.
Strengths
* Trustworthy checks of how good a given variational posterior is for real settings where we do not have access to the true posterior are sorely needed, and this work is an important step in that direction.
* The paper is well-written, well-motivated, and clearly explained.
* The theoretical results complement and motivate the proposed algorithm well (but see below for some minor technical concerns).
* The proposed algorithm is extremely simple to describe, making it practical and elegant.
Weaknesses
* My major concern is that I am not totally convinced that learning the proposed classifier is any easier than learning the variational posterior. As such, I would be concerned that diagnostics showing a good posterior fit could be driven by a far-from-optimal classifier. My intuition for why this should be a hard problem is as follows: for _any_ amortized posterior $q(\theta | x)$, if we can learn a Bayes-optimal classifier $d^*$, then one could recover the true posterior for any $x$ via $p(\theta | x) = q(\theta | x) / (1-d^*(\theta, x))$. As a result, learning $d^*$ for _any_ $q$ is equivalent to learning the true posterior. This is somewhat borne out in the third column of figure 2, where one needs roughly the same number of samples to learn a good posterior as one needs in order to learn a good classifier. I'm wondering if somehow the proposed approach is essentially equivalent to training two different amortized variational posteriors on independent data and then seeing how well they match. Some further simulations showing that one problem is easier than the other would alleviate some of my concerns. I should also say that I think that even if my concerns are valid, the diagnostic is still interesting and useful.
* I felt that the benchmarking could have been improved slightly. In particular, some empirical relationship between the proposed measure of fit and existing measures would be nice. For example, the authors could take the true posterior in a toy model where it is known, generate $q$ by perturbing that posterior in a systematic way and compare how increasingly large perturbations affect both the L-C2ST measure as well as things like KL, reverse KL, TV, and so on. The results in Figures 1 and 2 hint at this, but there are a lot of moving parts (especially in Figure 2) making it difficult to determine whether to contribute power (or lack thereof) to failure of the SBI (i.e., $q$ is bad) or the classifier (i.e. $r$ is good).
* A minor technical point throughout the proofs of the theorems is that they rely on the null hypothesis $\mathcal{H}_0$ holding on a set of strictly positive measure. That is, if $\mathcal{H}_0$ fails on a set of measure zero then the proofs would break down. In particular, in that case $\mathcal{A}$ would have measure zero and the final integral after line 454 would be exactly zero. I tend to feel that failures on sets of measure zero have no practical implications, and so this is a minor point. But, if one is going to state theorems and proofs, it would be good to be rigorous.
* Similarly, Lemma 1 should not be phrased in terms of "points" as those would have zero measure, and instead should say "If there exists a set $\mathcal{A} \subseteq \mathcal{S}$ with positive measure such that $p(\theta) > q(\theta)$ for all $\theta \in \mathcal{A}$, then there exists a set $\mathcal{B}$ with positive measure such that $p(\theta') < q(\theta')$ for all $\theta'\in \mathcal{B}$.
* Finally, there are similar (and equally minor) issues appearing in the proofs of Theorems 2 and 3.
Typos:
* Line 103: "may lack of" --> "may lack"
* Line 240: "in average" --> "on average"
* Line 251: Missing period before "The local PP-plot"
Questions
* Using the same $X_n$'s in Algorithm 1 for draws from $p(\cdot | X_n)$ and $q(\cdot | X_N)$ introduces dependencies in the training data for the classifier. Can that somehow be leveraged when training the classifier (similar to how in linear regression, we would want to perform GLS instead of OLS to increase the power when data points have shared structure like this). Along similar lines, if it is much cheaper to sample from $q(\theta | x)$ than it is to sample from $p(x, \theta)$, then could it help to somehow incorporate more samples from $q(\theta | X_n)$ in the algorithm (or perhaps the whole distribution somehow)? Obviously this would require accounting for the dependencies (perhaps via some kind of downweighting) in the training of the classifier. This is obviously not necessary to implement for this paper, but I am curious to hear the authors thoughts.
* Algorithm 1 also seems computationally intensive due to the permutation step. Would it be possible to do something like split-sample conformal inference to speed up the p-value calculations? Again, obviously not necessary for the paper, I'm just curious.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
I don't see any potential for direct societal harm from this work. The authors may wish to include a limitations section that highlights failure modes of their method.