On Consistent Bayesian Inference from Synthetic Data

Summary

The authors consider the use of synthetic data $X^{sync} $ created from a model $p(X^{sync}|Z, I_S)$ on a further Bayesian analysis where the analyst has access to $p(Q|X^{sync})$ and to $p(X^{sync}| Z)$, where $Q$ is the parameter. the paper is based on equality (4) $ p(Q|Z) = \int p(Q|Z.X^*)p(X^*|Z)dx^*$ where $X^*$ essentially represents $X^{sync}$ and $Z$ is either the real data or a differentially private version of the data and the idea is to replace $\int p(Q|Z.X^*)p(X^*|Z)dx^*$ by $p_n(Q) = \int p(Q|X^*)p(X^*|Z)dx^*$ which is fully accessible by the Bayesian and then show that the latter is close to the former. To do that the authors assume that $p(Q|Z,X^*_n)$ and $p(Q|X^*_n)$ are both close in total variation to the same seq of distribution say $D_n$ in probability when $X^*_n $ follows $p(X^*_n|Z,Q_0)$ and that given $Q$ $X^*$ and $Z$ are independent. Then the authors treat 2 toy examples a Gaussian and a logistic regression example and run some simulations to illustrate. Inthe above presentation I am not mentioning the fact that the Bayesian models can be different from the generating model, which is treated but quickly pushed by assuming that this is not a problem.

Strengths

The paper is well motivated and it is an important problem. If the results are correct then the paper is relevant and interesting.

Weaknesses

I am not sure the results are correct. From the presentation I don't understand the author's eq (4) or rather their comment which says that $p(Q|X^*, Z) $ is different from $p(Q|Z)$. The reason is that the authors do not explain what is the generating model for $X^*$ (In their paper the authors sometimes use $X^{sync}$ and sometimes $X^*$ as if they were the same, so I gather that they represent the same thing , but in the DAG of fig 1, they are not the same at all. The $X^* = X^{sync}$ is distributed from $p(X^*|Z)$, which does not depend on $Q$. Hence unless the authors clarify this point then their result are not valid. The authors consider two toy examples but in neither of them do they check tht the theoretical setup considered before is valid. For instance in the Gaussian example the generating model for the synthetic data is $ X^* \sim p( \cdot | X) = \int p(\cdot | \mu) \pi(\mu| X) d\mu$, i.e. the posterior predictive density. The relation (4) writes as $\pi(\mu|X) = \int p(\mu | X, x^*)p(x^*|X)dx^*$ but the model $p(\mu | X, x^*)$ is not defined. The authors seem to consider that $x^* , X|\mu$ are iid but this is not possible because $\mu$ is unknown and it does not correspond to their Gaussian example. There are a number of other results which seem dubious to me. See below.

Questions

1. In condition 3.2 the authors write for all Q but do not mention $Z$ while the distribution depends on $Z$. Is the condition almost sure in Z? in probability ? The same is true for condition 3.6. 2. Lemma 3.3 : What does it mean " hold for the downstream analysis for all $Q_0$ " ? condition (1) says given $Q$. Are $Q$ and $Q_)$ the same? 3. eq (17) says that they have the same mean and variance but not that they have the same limiting distribution. Why don't the authors verify the assumptions for this toy example. Surely if these assumptions don't hold for that one they will never hold. 4. in the supplement equation (204) : in my version of Asymptotic statistics there is not corollary 2.3 but a continuous mapping Theorem. I imagine that the authors are referring to the Lehman - Scheffe Theorem which states that if the sequence of probability densities $f_n$ converges pointwise to a probability density $f$ then it converges in $L_1$. Hoewever here the sequence is also random and the convergence is pointwise almost surely and I don't see the argument which allows to glue all these sets (for each Q) of probabiulity 1 to apply Lehman Scheffe. In other words the sets of proba 1 may differ from one $Q$ to another. 5. Minor comments: The authors recall in the call fairly trivial probability results which they should quote and recall possibly in the supplement and free this space to better explain their setup.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

1 poor

Presentation

1 poor

Contribution

3 good

Limitations

The authors are conscious of some of the limitations of their method.

Summary

The paper works on performing consistent Bayesian inference from synthetic data under DP. The authors propose a solution that involves mixing posterior samples from multiple large synthetic datasets, proving that this technique converges to the posterior of downstream analysis under specific conditions. This was established through experimentation involving non-private Gaussian mean estimation and DP logistic regression.

Strengths

The paper offers a unique and engaging exploration of Bayesian Inference in the context of Synthetic Data, providing a fresh perspective in a field predominantly characterized by frequentist analysis.

Weaknesses

See questions.

Questions

The motivation behind the research is not explicitly stated. Could you clarify the unique benefits that Bayesian Inference offers in this context? How does it enhance the study or application beyond the capabilities of other methodologies (frequentist)? The rationale for using Synthetic Data, specifically Synthetic Data without DP, is also vague. This choice seems to offer no additional solid protection under this setting. When applying DP, why choose to release Synthetic Data instead of the DP summary directly? The paper's primary theoretical contribution is not evidently defined. The claim that the distribution of Synthetic Data can be arbitrarily close to the original distribution as the sample size 'n' approaches infinity appears to be a trivial expectation. Could you elaborate on this aspect more? There has been prior discussion on the topic of Bayesian Inference from Synthetic Data, for example, as seen in reference [1]. Could you specify what new insights or advancements your study brings to the table, beyond the contributions of these previous works? [1] Wilde, H., Jewson, J., Vollmer, S., & Holmes, C. (2021, March). Foundations of Bayesian learning from synthetic data. In International Conference on Artificial Intelligence and Statistics (pp. 541-549). PMLR.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

While the study explores the concept of Synthetic Data, its impetus is not distinctly articulated, leading to ambiguity regarding the problem the authors aim to address. The theoretical contribution appears to be weak.

Summary

Inspired by Bayesian approaches for performing multiple imputation of missing data, this paper investigates the applicability of similar strategies for the analysis of synthetic data. Namely, the paper proposes inferring the downstream posterior of a Bayesian analysis by: generating multiple synthetic datasets; inferring the analysis posterior for each synthetic dataset; and mixing the posteriors together. (Interestingly, the paper finds that, contrary to the missing data imputation context, in the synthetic data case this strategy requires the synthetic datasets to be larger than the original dataset.) The paper provides theory showing that under the regularity conditions of the Bernstein-von Mises theorem (augmented by the additional conditions presented in Lemma 3.3), and assuming the congenial conditions in Definition 3.1, then the proposed strategy will approximate the data provider posterior distribution as the number of synthetic datasets and the synthetic dataset sizes increase. (The paper also proves a convergence rate result under stronger assumptions.) The method is evaluated using two simple examples: (i) non-private univariate Gaussian mean estimation (when the variance is assumed to be known); and (ii) differentially private logistic regression.

Strengths

This is an interesting paper. It addresses an important topic with an approach that appears to be novel and sound.

Weaknesses

One limitation of the proposed approach appears to be its reliance on the congeniality assumption (which we should not expect to hold in general). While the paper uses a simple example to illustrate that the method was still able to recover the data provider’s posterior when congeniality was violated, the paper needs to provide more extensive evidence of the robustness of the proposed approach w.r.t. violations of this assumption (as, in practice, it seems that the usefulness of the proposed approach for data analysis will depend on how robust the method is to violations of congeniality). More specifically, the paper shows that for the toy problem of Gaussian mean estimation (with known variance) the mixture of posteriors converges to the data provider’s posterior even when the analyst’s variance is different from the data provider’s variance (right panel of Figure 2). However, for this example we have that the posterior distribution for the mean is already Gaussian in the finite sample setting to begin with. Providing additional examples where the posterior distribution of the quantity of interest is not Gaussian in the finite sample setting, but where the mixture of posteriors approximates the data provider’s posterior when congeniality is violated would provide more convincing illustrative examples. Perhaps, one simple example is the problem of Gaussian variance (or precision) estimation with known means. In this case, the paper could assess the robustness w.r.t. congeniality violations by choosing different means for the data provider and data analyst. The paper should provide additional examples along these lines. The paper might also want to include some discussion about some practically important settings where the Bernstein-von Mises theorem does not hold, and where the proposed approach might not be applicable (e.g., for models where the number of parameters increases with the sample size). Other minor suggestions: Line 69: change “which makes method their more” to “which makes their method more” Line 302: change “To recover the analyst’s posterior …” to “To recover the data provider’s posterior …”

Questions

See suggestions above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes, the paper addresses well the limitations of the proposed method.

Summary

The paper studies Bayesian inference based on synthetic datasets generated in a DP and non-DP setting. The paper suggests a specific sampling approach for downstream Bayesian inference using synthetic DP and non-DP dataset. It contributes theoretical results on the convergence of the inferenced posterior (from synthetic dataset) to the true posterior showing that (under certain assumptions) it converges as the the of number of synthetic datasets and the size of the datasets increases. Additionally, a convergence rate is derived Experimental results are provided for non-DP Bayesian mean inference and DP Bayesian logistic regression showing that the inference approach works, along with examples of the effect of parameters influencing the convergence (including number of observations, samples and level of congeniality)

Strengths

- Very timely and interesting topic; I enjoyed learning about the specific Bayesian+DP setting. - The theory is mostly well presented in the main paper (see suggestion/questions below). The narrative is relatively easy to follow. - The theory appears sound. I have not found any obvious issues; however I would need to rely on other reviewers (and perhaps later the community as a whole) to validate the many proofs in the supplementary.

Weaknesses

The following are question and comments; not necessarily weaknesses per se: - The balance between theory and experiments is generally very good for my taste, but I feel the experimental part (in the main paper) let the theory part down a bit. The initial experiments focus on intuition and basic insights which I string support; however once the basics have been presented it would had been helpful with an experiment which covers many more scenarios proving summaries of the performance (using TV, coverage, means/modes and variance as metrics), along the most relevant dimensions such as number of observations, level of - ... it would also have been interesting with a more realistic example (high dimensional) using a more complicated model to better motivate the paper. Have the authors validated the results on such an example? - Figure 4 (right): It is not clear to me why the non-DP posterior does not mange to center its mode closer to the true parameter, is this an effect of the prior being centered on the true parameter combined with relatively few observations (the prior is not specified in the main text as far as I can tell?) - or other things? - Figure 1: I am slightly confused by the graphical model, probably because the nature and role of $\theta$ is never really explained in detail. I hope the authors can clarify this (perhaps along with a detailed explanation the generative model in general)? For completeness, I would suggest including $I_a$ and $I_s$ in the figure as well. Overall, I am generally positive about the paper but the experimental parts misses an opportunity to convince me. I will opt for a borderline score until I get a chance to see the other reviews and the authors' response.

Questions

Included above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Included above.

Summary

This work is sloving a interesting task, which infers the downstream analysis posterior using synthetic data. The work proved that the Bernstein-von Mises theroy applies, the method can converage to the ture posterio as the number of synthetic datasets. The experimental settings are under two examples, i.e. non-private univariate Gaussian 56 mean estimation and differentially private Bayesian logistic regression.

Strengths

1, The work is trying to solve an interesting task, which is infering the downstream analysis posterior using synthetic data. 2, The paper is well-writen and presented. 3, The code is provided. So it will be helpful for the following work.

Weaknesses

1. Since synthetic data is generated by models which are trained using real data. So why synthetic data can improve the consisten bayesian inference is not clear. I think the paper needs more discussion about differences bewteen the real data and synthetic data. 2, The synthetic data is a big topic. In the work, for me, it is not clear which synthetic data methods are used and how the synthetic data method is trained using real data. 3, The applications are missing. Is it possible to extend the proposed method or therory to some kind of real application.

Questions

See weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

See weaknesses.

acknowledge and discuss the authors' detailed response

The authors have provided a **very detailed** response, which includes answers to reviewers' questions and more exposition of the technical material. **Drop a comment in this thread with how their response changed your perspective on the submission.** Remember to also update your scores/reviews to reflect how your views have changed in light of the authors' response, and respond to comments/questions the authors have left underneath individual reviews.

Summary of Changes

Summary of the changes we will make to the final version, requested by reviewer uV6S: - Additional experiments - UCI Adult data (see general rebuttal) - Plots with total variation distance (see general rebuttal) - Example of Gaussian variance estimation with known mean (see rebuttal to reviewer 4qww) - Discuss recent related work the Bayesian viewpoint to synthetic data (see general rebuttal) - Clarify why synthetic data is useful (see rebuttal to reviewer z4aQ) - Mention prior for the downstream logistic regression on toy data (see rebuttal to reviewer uV6S) - Clarify role of $\theta$ (see rebuttal to reviewer uV6S) - Clarify $I_A$ and $I_S$ in Figure 1 (see rebuttal to reviewer uV6S) - Add example settings where the Bernstein-von Mises theorem does not hold to Limitations (see rebuttal to reviewer 4qww) - Clarify the difference between $X^{Syn}$ and $X^*$ (see rebuttal to reviewer ZbGV) - Clarify statements of Condition 3.2, Condition 3.6 and Lemma 3.3 (see rebuttal to reviewer ZbGV) - Show that $\mu^*$ has a Gaussian distribution in the Gaussian mean estimation example (see rebuttal to reviewer ZbGV)