Post-processing Private Synthetic Data for Improving Utility on Selected Measures

Summary

The paper considers how to obtain improved utility from synthetic data under differential privacy. Prior work has tried to achieve utility by incorporating utility directly into the generation of the synthetic data e.g., by making use of workload data. Here, the proposal is to perform post-hoc reweighting of the synthetic data, based on measuring a difference in values between the final data under a set of queries with known answers. This measurement/reweighting step is also done under privacy. The paper shows that the optimization can be done using standard solvers, and so can be performed effectively via linear or quadratic programming (for pure or approximate DP). A specific use case based on aligning the correlation matrix is used to demonstrate that the sensitivity calculations can be quite simple. Numerical experiments demonstrate that the approach can be quite effective in terms of absolute error ratio, and F1 score.

Strengths

Synthetic data is widely promoted as a practical way to gain from private data, without running into issues around exhaustion of privacy budget. Ensuring that synthetic data is usable for the target queries is therefore an important challenge, particularly if the workload may not be known at data generation time. This approach shows a promising way to overcome these issues, provided we still have access to the data to perform the private optimization step. The approach is very clean, and shows good promise in terms of utility and performance.

Weaknesses

Evaluation is on four benchmark data sets that may be considered "easy". It would be interesting to test this out on additional data, such as that used in recent synthetic data challenges by NIST.

Questions

Apart from correlation matrix, can you generate more examples of query families to apply that have bounded sensitivity?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No need to address societal impact. Technical limitations are outlined in Section 5 in broad strokes, but it would be interesting to hear more specific suggestions about how this approach could be extended to handle more diverse data.

Summary

The paper proposes a DP post-processing method for synthetic data that weights the synthetic data to match user-selected utility measures on the real dataset. The utility measures on real data are measured with noise to make this post-processing DP. To find the synthetic data weights, the paper derives a closed-form expression, and develops an optimisation algorithm to find optimal dual variables required to evaluate the closed-form optimum. To evaluate the method, the paper conducts experiments on 5 real datasets, testing how much the method is able to improve the synthetic data produced by 5 existing methods of generating DP synthetic data.

Strengths

The paper is written well, and the main points are easy to understand. The idea behind the proposed method is interesting, and it should, at least in principle, be applicable to any kind of synthetic data and utility measure.

Weaknesses

Some important experimental details are missing. See my questions. The Private PGB method of Neunhoeffer et al. (2021) is fairly similar to the proposed method, and their differences should be discussed. Currently the paper is not even cited. Minor comments: - Slater's condition should be introduced before the proof of Theorem 1. - Seeing the raw utility and F1 scores from the experiment in Table 1 would be useful, as it would show whether the large improvement from post-processing on the GANs is just caused by the GANs generating poor synthetic data - References [App17], [Dat23], [DG17], [MOK18], [Sma23] should have URLs

Questions

Important experimental details: - Is the plain synthetic data baseline $\epsilon$ or $(\epsilon + \epsilon')$-DP? It seems to be the former, which is not a fair comparison, as the generation + post-processing is $(\epsilon + \epsilon')$-DP. - Are the categorical columns also converted to numerical features for AIM and MST? They should not be, as both methods handle categorical values natively. - What is the workload given to AIM? Does it contain the marginal queries with the variables the utility measures are looking at? Minor questions: - How is the real data split into training and test data? - Why were MST and AIM not run on the home-credit dataset? The original paper on Private-PGM (McKenna et al. 2019) that both methods are based on experiments on a dataset with a similar number of features. - Is there a difference between $\lambda^*$ in (4) and $\lambda$ in (5)? References: - Neunhoeffer et al. "Private Post-GAN Boosting" ICLR 2021 - McKenna et al. "Graphical-model based estimation and inference for differential privacy" ICML 2019

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The paper mentions that they assume the real data is normalised so each feature lies in $[0, 1]$. This is a fairly large limitation, as normalising data under DP while retaining the possibility of undoing the normalisation is not trivial, and should be discussed further..

Summary

The paper introduces a post-processing technique that enhances the utility of synthetic DP data with respect to selected measures. The proposed technique involves resampling from the synthetic data to filter out samples that do not meet the selected utility measures, using a stochastic first-order algorithm to find optimal resampling weights.

Strengths

Strengths: - The post-processing technique discussed appears to be novel and significant. - The method is model-agnostic. This makes the technique highly versatile. - The authors provide a good set of numerical experiments to validate their approach. These results are strong and show improvements across multiple benchmark datasets and synthetic data generation algorithms. - The paper is well-structured and the methodology is clearly explained.

Weaknesses

Weaknesses: - The paper could benefit from a more detailed discussion on the trade-offs involved in using the proposed post-processing technique. For instance, it would be helpful to understand the impact of the technique on the computational complexity of the data synthesis process. - The authors could provide more examples or case studies to illustrate the practical applications and benefits of their proposed technique, beside the correlation example illustrated.

Questions

- Could you elaborate on the computational complexity of the proposed post-processing technique? How does it compare to the complexity of existing synthetic data generation algorithms? - How does the proposed technique handle high-dimensional data? Are there any limitations or challenges in this regard? - Could you provide additional examples to illustrate the practical applications and benefits of the proposed technique?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

See my questions above.

Summary

The paper under review introduces a technique aimed at enhancing the quality of differentially private synthetic data. Specifically, this approach is applicable when a private synthetic data set, generated by any available privacy-preserving mechanism, does not align with the original data set on certain key measures or queries. The proposed solution involves adjusting the weights of synthetic data samples to ensure their alignment with the original data based on the specified objective. The paper specifically applies this technique to post-process synthetic data such that the resultant synthetic data aligns with the true data's correlation matrix. The proposed algorithm commences by estimating the empirical correlation matrix using the Gaussian mechanism with privacy parameters epsilon=1 and epsilon=3. This involves the addition of independent Gaussian noise to each entry of the correlation matrix. Subsequently, the algorithm resolves a convex optimization problem using a first-order method to determine the optimal sample weights that best conform to the noisy correlation matrix. The research findings demonstrate that the post-processed synthetic data offers a more accurate approximation of the correlation matrix compared to the original synthetic data (prior to the reweighting operation), thereby underscoring the effectiveness of the proposed method.

Strengths

The idea of post-processing synthetic data presented in this paper is a compelling approach that boasts potential applicability to a multitude of problems. While the primary objective here is the alignment of pair-wise correlations within the data, theoretically, the objective could be a more intricate query that existing synthetic data methods are not designed to tackle.

Weaknesses

In terms of originality, the proposed solution bears similarity to the Private Entropy Projection (PEP) mechanism referenced in [LVW21]. The PEP mechanism functions over the entirety of the data domain, assumed to be sufficiently small for its computational limitations. Nevertheless, it could, in theory, operate over any support, including samples from synthetic datasets, as suggested in this paper. Thus, it would be advantageous to explain the distinctions between these methods. Further, if they indeed differ, the paper should clarify why the PEP method is ill-suited for addressing the problem at hand. Regarding the experimental setup, it appears to have some potential flaws. The paper compares the utility of a synthetic dataset D_syn, presumably generated using an original privacy budget (termed epsilon_original), with the utility of a post-processed dataset D_post, created with an additional privacy budget of epsilon=1. Consequently, the post-processed dataset D_post possesses less privacy than the pre-processed D_syn. Hence, it remains ambiguous whether the superiority of D_post over D_syn can be attributed to the proposed resampling method or merely to the fact that a larger privacy budget was utilized to generate D_post. A more transparent comparison would involve assessing the utility of D_post against a synthetic dataset generated using a budget of (epsilon_original + 1). Lastly, while the post-processing operation enhances the synthetic data in relation to the correlation matrix, it remains uncertain whether this operation compromises other essential data properties. For instance, if the original synthetic data was trained on 3-way marginals, the post-processing step could cause a deviation from these queries, thus leading to a poorer approximation. This possibility could account for the machine learning results in Figure 1, where the post-processed dataset exhibits a lower F1 score under certain conditions.

Questions

Why was the algorithm GEM only used for the home-credit dataset ? What is the effect of the post-processes dataset on the measures that the original data was optimized for? That is, does the utility degrade? How does the quality of the original synthetic data affect the absolute performance of the post-process synthetic data ?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

1 poor

Limitations

Yes

Re: rebuttal

Thank you for your responses to my questions! Including a discussion of the computational complexity of the method would indeed be helpful.