Discriminative calibration: Check Bayesian computation from simulations and flexible classifier

Summary

This paper presents a classifier based approach to provide a measure of miscalibration in Bayesian computation, including for methods such as Approximate Bayesian Computation (ABC) and Simulation-Based Inference (SBI) methods like neural posterior estimation. The method enables the test statistic to be learned from data and provides an interpretable divergence measure. The method is a form of two sample testing applied in the amortized and simulation based inference settings. Beyond standard approaches to using classifiers for two sample testing, the authors develop several classifier based approaches which use a form a "label mapping", and provide theoretical work on the validity and efficacy of these approaches. Experiments are provided to test the method for posterior inference in cases where the posterior is known, and provide empirical verification of the theoretical results. The paper also compares to the commonly used simulation based calibration method, and show clear improvements with the methods proposed in the paper. Experiments on cosmological data are also explored.

Strengths

This is a good paper, it provides: - a clear description of the challenge they hope to address, specifically providing a better and more statistically interpretable measure of miscalibration over simulation-based calibration - a clear description of the proposed methods, including how to use label mapping to develop calibration based diagnostics, and how this differs from other approaches - theoretical grounding for the calibration based measures, including the expected behaviors in large sample limits - useful discussions on implementation details - useful discussion on the legitimacy and power of the tests.

Weaknesses

While the idea of using classifiers for two sample testing is not new, the authors do develop new methods based on label mapping. While the new classification approaches are discussed and will likely be quite useful for simulation based inference, some attempts at using classifiers for testing posterior inference quality have been performed before (albeit not directly with some of the new classification methods discussed in the work). For instance, Vandegar et. al "Neural Empirical Bayes: Source Distribution Estimation and its Applications to Simulation-Based Inference" AISTATS 2021, use the AUC of a classifier as a diagnostic in a simulation based setting. Further development of the experiments would be quite useful. A very simple experiment is shown, allowing an exploration of the methods, but some experiments more clearly showing the quality of the method in a controlled but more complex simulation based inference setting could provide useful insights to the readers. For instance, a common example is the Simple Likelihood, Complex Posterior (SLCP) problem from Papamakarios et. al, "Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows", AISTATS 2019. As it stands now, it is difficult to asses the quality of the method in the more complex cosmological data set experiment.

Questions

Could you provide more details on the posterior model in the multi-variate Gaussian example? Could you provide experiments on more complex but controlled examples, for instance the SLCP problem from Papamakarios et. al? This could provide more evidence on the utility of the method in SBI settings. How do these methods scale with feature and parameter dimension? Does this affect the ability to attain a useful diagnostic? You mention that the presence of nuisance parameters do not affect the quality of the diagnostics. However, the Neyman-Pearson lemma does not guarantee that the likelihood ratio is the uniform most powerful test for a given size in the presence of nuisance parameters. Perhaps I am not fully understanding the text, could you explain if this has any ramifications for the proposed tests? Could you add references to other simulation based inference work using classifiers for diagnostics? In addition, recent work on using neural networks for two sample testing (e.g. Grosso et. al, https://arxiv.org/abs/2305.14137) may be relevant to discuss in the related work.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors provide clear discussions of limitations. It would be interesting to also know if there are limitations dependent on the dimension of the features or parameters, and how this may impact the number of samples needed to attain a useful test.

Summary

In this paper the authors focus on the challenge of comparing two conditional distributions p, q from their samples. In particular, this is useful as a check for bayesian computations. To achieve this, the authors propose the use of a probabilistic classification approach where they create a new dataset combining samples from p and q and labels related to the particular distribution. Here, the authors propose different approaches and make connections to previous work. Given the combined dataset, a classifier tries to predict the label from the features. Failing to do so, indicates that the classifier could not discriminate between the two distributions. The performance of the probabilistic classifier is then used to estimate the divergence between the distributions and for hypothesis testing. In the empirical section, the authors compare against SBC and show improvements in power given a reduced number of samples.

Strengths

The paper analyses different approaches for the generation of samples to be used by discriminators. They then present theoretical developments and demonstrate their results empirically. The sampling approaches and theoretical treatment are novel, of high quality and well written. The main advantage here is that even for a sub-optimal classifier, one gets improvements in data efficiency.

Weaknesses

The main weakness of the paper is that in the empirical evaluation, the authors do not explore the case where the samples present more challenging behaviors such as auto-correlations, different types of imbalances, etc. Other minor suggestions: - The colors of the thetas in the introduction are hard to see. - Describing why the bounds are tight could further help the reader in section 2.

Questions

I'm thinking of your paper as an approach to test whether the labels and the generated features are connected. This reminds me of [1] where they evaluate the mutual information via neural networks. I'm wondering both if it makes sense in your setting to estimate the divergence via the Donsker-Varadhan representation, and in their setting if your feature generation approach is beneficial. Do you have any opinions on that? The same has applications for other approaches on independency testing via classifiers where your feature generation approach also has potential. [1] Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

There are no potential negative societal impacts from this work. The authors mention limitations in a future work section.

Summary

In this paper, the authors consider conducting frequentists tests for simulation-based inference. Specifically, the authors want to test if an inference engine q(theta | y) produces samples from the true posterior p(theta | y). The key idea is to use classifiers to determine similarities between samples generated from the true and approximated posterior (similar to GAN). The foundation of the algorithm is laid in section 3, where the authors show the negative cross-entropy is associated with a generalised divergence between p and q. In particular, D4 becomes KL divergence between p and q as the number of posterior draws goes to infinity. Empirical study validate the theoretical results and shows the effectiveness of the algorithm.

Strengths

1. This paper is generally well-written and the idea is intuitive and sound. 2. This paper addresses an understudied problem (SBC) in the literature, and the expansion from a single-dimensional rank statistic to a multidimensional test statistic seems to be a significant improvement 3. The algorithm provides theoretical support (Section 3) to the proposed algorithm.

Weaknesses

1. From the methodological point of view, using a classifier to compare distributions is not a new idea (as the authors have discussed in Section 6. 2. When training a classifier, it seems authors are facing very imbalanced classes and training an imbalanced classifier itself can be difficult. I am wondering if this would cause issues in the tests performed later.

Questions

If one wants a divergence KL(p(theta|y, q(theta|y)), one can directly approximate it without using a classifier, right? KL(p(theta|y), q(theta|y) = KL(p(theta, y), q(theta|y)p(y)). Suppose you can sample from p(theta, y), q(theta|y) and p(y). Then we can directly approximate above KL using f-GAN, and perform a permutation test or compute CI using bootstrap. This approach sounds like a simpler way to obtain a divergence measure between p and q, so I wonder why the authors did not explore this idea nor use it as a benchmark method. ---------- Revision after author's response. This approximation cannot be easily applied due to the IID-ness and the sequential nature of SBI as authors mentioned.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This work generalizes the well known Simulation-Based Calibration method to a setting where the tests on the posterior approximation consistency are based on classifiers. The paper has a nice balance between theoretical discussion (Section 3) and more practical issues (Section 4) making it easy for readers to appreciate different aspects of their proposal. The paper is solid and tells a nice story from start to finish, but I would be tempted to point out the limitation of assessing only the quality of the posterior approximation **in average**. The authors mention this downside but don't give any insight regarding in which situations this might be a problem and when it could be OK to abandon local information. Furthermore, it comes somewhat as a surprise the fact that the authors use a **discriminative** approach for assessing the quality of the posterior approximation, but make absolutely no mention to the well known classifier two-samples test (C2ST). In all, I'm rather satisfied by the paper as a contribution to the SBI literature, but I'm not certain of how wide is its scope for the NeurIPS conference. I'm giving it, therefore, a score of 6 (borderline accept), and would not be surprised if in the end it was not accepted to the conference main track.

Strengths

- The authors managed to recast the simulation based calibration (SBC) method in a discriminative framework. This is very nice as it helps better understanding such method and see how it compares to other approaches. - The three theorems presented in Section 3 are of great practical utility and ensure that even though we only estimate a classifier between samples from a fixed-size dataset, we can be sure that its performance will serve as a lower bound to other theoretical quantities. In other words, the fact of actually having to abide with the "real life" setting of training a classifier from limited data is not a problem.

Weaknesses

- The mathematical notation can be sometimes a bit confusing. For instance, it is not always easy to discern what is a scalar input variable to a function with what is a random variable samples from a pdf. - There's no comment on assessing the quality of the classifier trained on calibration data and then used to build the statistical tests. - It could have been of interest to consider more examples on simulated data to illustrate the procedure. For instance, using examples from the `sbi-benchmark` which are becoming more and more used in the simulation-based inference literature.

Questions

- Why is your Figure 2 **after** the set of Figures 3-6? This is very disturbing... - How useful it is to have a diagnostics to $q(\theta \mid y)$ which is only valid in average over $y$? Can you tell a bit more about this? - The class imbalance in your Example 1-4 will probably make your classifier have a lot of trouble to discriminate one class from the other. Indeed, it will become more of an "anomaly detection" kind of thing than an actual classification. You don't make much comments on this but I think it would be very important to explain whether this is a difficulty or not. - Can you add a reference to your claim in Line 82? - Line 188: What do you mean with "classifier $c$ is good enough" ?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I've mentioned the limitations in the previous fields.

(f)-Divergence View has been Studied in SBI

Thanks for the response. > In the past of SBC, it was not clear what space to run the classifier on ... nor to interpret results as a divergence between the true and inferred posteriors. First, the second part of this claim is not true. See, Section 2.1, Variational Inference for Simulation-based Inference, ICLR 2022, where authors clearly formulated the SBI problem as a divergence minimization problem between true and inferred posterior. Second, I do not understand the first half of the sentence. The binary classifier has been studied to minimize such type of divergence (in an adversarial fashion) in the space of both theta and y, see GATSBI: Generalized Adversarial Training for SBI, ICLR 2022. thus I do not understand why "it was not clear what space to run the classifier"? Please specify. ------------------------------------------------------------------------------------------------------------------------------------------------- **Revision**: Sorry, I noticed SBC stands for SB "Calibration", not "Computation", thus authors claims **are valid**. Nonetheless, these two papers I mentioned above seem to be relevant as they both considers computing divergence measure in SBI problems.

Author-reviewer discussion

Dear all, The author-reviewer discussion period has now started. It will continue for one more week, until August 21. @authors: Please respond to the comments or questions reviewers may further have. Remain short and to the point. @reviewers: Please read the author's responses and ask any further questions you may have. To facilitate the decision by the end of the process, please also acknowledge that you have read the responses and indicate whether you want to update your evaluation. - You can update your evaluation positively (if you are satisfied with the responses) or negatively (if you are not satisfied with the responses or share other reviewers' concerns). Please note that major changes are a reason for rejection. - You can also keep your evaluation unchanged. In this case, please indicate that you have read the responses and that you do not have any further comments. Best regards, The AC

Response

Dear Authors, thank you for your detailed responses. I continue to consider this a good paper and believe it should be accepted. In terms of nuisance parameters, indeed NP is for simple hypotheses, but my point was that only likelihood ratios with simple hypotheses will result in uniformly most powerful tests. This is as opposed to Wilk's test / likelihood ratios test that deals with nuisance parameters through maximization, but is not guaranteed to be uniformly most powerful. For marginal likelihoods, it is not clear this will be a uniformly most powerful test. Nonetheless, we may be discussing an issue that is not the most relevant point for your work.