Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training

Summary

The manuscript introduces a slicing privacy mechanism for training generative models without noisy gradients. This mechanism injects noise into random low-dimensional projections of private data, providing strong differential privacy guarantees. The study introduces the smoothed-sliced $f$-divergence and a kernel-based estimator for it, allowing the training of generative models without adversarial training. Empirical results show improved synthetic data quality compared to existing methods.

Strengths

- The introduction of the slicing privacy mechanism and smoothed-sliced $f$-divergence represents a novel approach to training privacy-preserving generative models, addressing the limitations of noisy gradient-based methods. - The paper provides strong theoretical foundations for the proposed methods, including privacy guarantees and statistical consistency of the smoothed-sliced $f$-divergence. - Extensive experiments demonstrate the method's effectiveness in generating high-quality synthetic data across various datasets, outperforming baseline methods.

Weaknesses

- This is a very solid piece of work. The proposed method is simple yet effective, and I have no particular concerns or issues with it.

Questions

None

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The paper proposes to add noise to randomly projected private data along with optimizing a newly proposed metric smoothed-sliced f-divergence to train generative models. Such paradigm can circumvent adding noise to gradient and enable more architecture choices. Experiment show the proposed method perform competitively with existing methods.

Strengths

1. The proposed approach directly add noise to data, thus enable more data processing possibilities on privatized data. 2. The proposed approach is shown to have consistency and perform competitively with baseline algorithms such as DP-SGD, PATE, and SliceWass.

Weaknesses

1.I feel thee baseline algorithms are kind of weak for the setting considered in the paper (synthetic tabular data generation). I am not an expert in this specific area but SliceWass is proposed 3 years ago, DP-SGD and PATE are classical DP algorithms which have many variants improving upon them in recent years, e.g., Adam also has DP versions. 2. The algorithm is proposed for generative model training, at least it is what claimed in the title and abstract, but it is only tested on tabular data.

Questions

1. Have the authors tested the algorithm on other type of data, such as image or text? 2. Do the authors have explanations why the proposed approach could be better than the baseline algorithms?

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

I did not find any limitation discussion specifically for the proposed approach. Adding it would be beneficial.

Summary

This paper proposes a DP generative modeling technique via f-divergence and random projection. Specifically, both real data and synthetic data are randomly projected into a lower-dimensional space, where the noise is added to the aggregation of the projected data such that the effect of individual data point is bounded. Minimizing the smoothed-slicing f-divergence between the noisy embeddings will guarantee the match of two distributions, thereby can be used to train a generative model. Compared to adding noise to gradients (e.g. DP-SGD), this method is more convenient and scalable. Experiments show that it generally outperforms prior related methods in synthesizing tabular data.

Strengths

1. Looking for better alternatives to DP-SGD is an active research area to circumvent the common issues DP-SGD has in practice, thus is well-motivated and of wide interest in the related field. 2. The method is complete, with full description and theoretical privacy analysis. 3. Empirically, multi-dimensional evaluation metrics are considered, and the proposed method outperforms other alternatives in most metrics.

Weaknesses

1. The proposed method is similar to the MMD-based methods (references on line 96) in the following way: + MMD-based methods apply MMD as the divergence measure, since MMD(P, Q)=0 iff P=Q, similar to $D_f$(P, Q)=0 iff P=Q, thus both divergence measure can be used to train a generative model. + MMD-based methods minimize the distance between kernel mean embeddings (KME) of real data and generated data, while noise is added to the KME to retain DP guarantee. In this work, both real and generated data are projected to an embedding space, and noise is injected into the embeddings to retain DP guarantee, so the idea of the overall paradigm is quite similar. + KME in theory is an infinite-dimensional embedding, which is not convenient to add Gaussian noise, so those works propose to approximate KME by projecting it into a finite-dimensional space. In this work, in theory you have infinitely many directions to slice where only a finite number ($m$) of directions are randomly chosen. So both of you apply the finite-projection idea and therefore both of you will have approximation errors. In light of the above similarity, the current discussion on the difference and association with MMD-based methods are not adequate. Also, it is more ideal to include some of those methods in the empirical comparison (Table 1), because they ran experiments on tabular dataset as well. 2. For a work in DP, I did not even find sensitivity analysis, which is crucial for proving DP guarantees. Even if it is trivial it should be included for completeness. 3. It looks to me that adding noise to synthetic data (eq. 2) is unnecessary, and will hurt the model utility. Can you please explain why?

Questions

1. Existing methods move to adversarial dual form due to "...they often suffer from scalability issues and are not friendly to gradient-based optimization..." Now you circumvent the adversarial training, then does the proposed method suffer from scalability issues or not friendly to gradient-based optimization? 2. Maybe I am out of the field, but what does "slicing and smoothing" (line 116) exactly mean? What are the differences? And you term the proposed divergence "smoothed-sliced f-divergence", what does "smoothed-sliced" mean? 3. In remark 1 the authors claim that [RL21] did something wrong in their derivation because the slicing matrix U is not included in the privacy mechanism. I am not familiar with [RL21], but is U independent of the dataset X? Why will U affect the privacy analysis? 4. Line 221, "...we can achieve a tighter privacy bound by reducing a factor of..." How this factor is derived? 4. The proposed method seems quite universal, i.e. should be scalable to other domains like image datasets as well, is it true?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

NA

Responses to the rebuttal

We thank all Reviewers for their time and thoughtful feedback! We are glad to see our paper was positively received, and the Reviewers found that: the paper is novel, rigorous, and well-written (all Reviewers); it is well-motivated and of wide interest in the related field (Reviewer NMCZ); the proposed approach is shown to have consistency and perform competitively with baseline (Reviewer JoQ4); the paper provides strong theoretical foundations (Reviewer jo82). We also recognize that the Reviewers are busy handling multiple papers, so their thoughtful feedback is even more appreciated. In response to the Reviewers’ questions, we did our best to provide comprehensive answers and included additional experimental results (please refer to the 1-page pdf in the global response). Please do follow up with us if you have additional suggestions and feedback that can further strengthen the paper. Thanks!