Gaussian Membership Inference Privacy

We propose a novel and practical privacy notion called $f$-Membership Inference Privacy ($f$-MIP), which explicitly considers the capabilities of realistic adversaries under the membership inference attack threat model. Consequently, $f$-MIP offers interpretable privacy guarantees and improved utility (e.g., better classification accuracy). In particular, we derive a parametric family of $f$-MIP guarantees that we refer to as $μ$-Gaussian Membership Inference Privacy ($μ$-GMIP) by theoretically analyzing likelihood ratio-based membership inference attacks on stochastic gradient descent (SGD). Our analysis highlights that models trained with standard SGD already offer an elementary level of MIP. Additionally, we show how $f$-MIP can be amplified by adding noise to gradient updates. Our analysis further yields an analytical membership inference attack that offers two distinct advantages over previous approaches. First, unlike existing state-of-the-art attacks that require training hundreds of shadow models, our attack does not require any shadow model. Second, our analytical attack enables straightforward auditing of our privacy notion $f$-MIP. Finally, we quantify how various hyperparameters (e.g., batch size, number of model parameters) and specific data characteristics determine an attacker's ability to accurately infer a point's membership in the training set. We demonstrate the effectiveness of our method on models trained on vision and tabular datasets.

Paper

References (48)

Scroll for more · 36 remaining

Similar papers

Peer review

Reviewer vdUQ5/10 · confidence 3/52023-07-04

Summary

The proposed f-MIP method incorporates a practical membership inference attack threat model, offering easily interpretable privacy guarantees. This approach improves utility, especially when the attacker's capabilities are realistically constrained. Through a theoretical analysis of likelihood ratio-based membership inference attacks on noisy stochastic gradient descent (SGD), μ-Gaussian Membership Inference Privacy (μ-GMIP) is introduced. μ-GMIP requires less noise compared to the corresponding Gaussian differential privacy (GDP) guarantees, resulting in higher utility.

Strengths

- The concept of applying control over type I and type II errors to MIP is interesting. - The proposed approach holds promise for enhancing privacy protection in SGD. - The numerical study provides evidence that the proposed method outperforms noisy SGD in terms of utility, highlighting its potential for practical applications.

Weaknesses

- The paper lacks a theoretical exploration of the relationship between f-MIP and f-DP, which hinders a comprehensive understanding of the proposed method. For instance, while mu-GMIP and mu-GDP are compared at the same mu in Figure 1, it remains unclear whether they actually share the same mu value. - The absence of a discussion on post-processing or the composition rule (in terms of mu) limits the practical applicability and usefulness of the proposed approach.

Questions

As mentioned in the paper, it is necessary to focus on cases where the membership is accurately determined among those classified as members, which corresponds to achieving a high true positive rate (TPR) at low false positive rate (FPR) in threshold-type tests. Therefore, it would be logical to concentrate on scenarios with low FPR. However, the trade-off function is compared across the entire range [0, 1] rather than solely focusing on these low FPR cases.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- The iid assumption is not realistic in practical scenarios, which suggests that the actual privacy guarantees would be lower than those stated in the paper. - Since the f-MIP method is specifically designed for SGD, it raises questions about its generalizability to other problem domains.

Reviewer Uu8E5/10 · confidence 4/52023-07-06

Summary

The authors introduce a new privacy notion called f-membership inference privacy (f-MIP), which relaxes strict Differential Privacy (DP) assumptions thereby promising better model utility. The paper proposes a theoretical analysis of membership inference attacks on DP-SGD based on trade-off curves (similar to f-DP) and introduces a family of f-MIP guarantees called µ-Gaussian Membership Inference Privacy (µ-GMIP)(similar to GDP). The analysis follows a similar approach to the original DP-SGD analysis: first the privacy budget is derived for single step with subsampling which is then composed over the training run. The paper then verifies the theoretical analysis by introducing gradient attacks based on likelihood ratio tests. The attack requires to know the underlying gradient distribution parameters. They present results for a single DP-SGD step with known parameters and the privacy guarantees seem tight. They also present results for estimated distribution parameters where the guarantees look loser.

Strengths

- The derivations seem sound and follow the well regarded hypothesis testing interpretation in DP - Investigating the tightness and potential relaxation of DP-SGD is an important problems and the authors make a solid contribution. - The paper is clearly structured and easy to follow.

Weaknesses

- It would be nice to see an investigation on the validity of the initial assumptions. The paper was motivated that DP is overly conservative since it also holds for pathological datasets such as empty datasets and singletons with an adversarial sample. While this is true, it has been recently shown that a simple canary insertion in an otherwise natural dataset may be sufficient to produce tight lower bounds for DP accountants [Nasr et al 2023]. - It would be great to extend the FPR and TPR ranges in the plots in figure 4 to smaller values. Ideally all the way to the first data point. These ranges capture very relevant adversary objectives where an adversary only cares about identifying one sample but that with high confidence. - Minor: - Only asymptotic guarantees for composition. Recently, there has been significant progress in numerical composition of DP guarantees. **References** Nasr, Milad, et al. "Tight Auditing of Differentially Private Machine Learning." arXiv preprint arXiv:2302.07956 (2023).

Questions

- Is it assumed that $\tau$ in the noise parameter includes the batch size? Algorithm 1 does not scale the noise by the batch size which is different to typical DP-SGD [Abadi et al 16]. - Figure 4a. Purchase and CIFAR10 seem to exceed the analytical bound for low FPRs. Is this because of the earlier mentioned assumption that the challenge points are sampled from the distribution which for low FPRs may be already distributional outliers? - Are the techniques for composing DP guarantees numerically also applicable in this setting?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

As discussed in weaknesses. I believe the discussion about the validity of the initial assumptions is limited.

Reviewer PQQ44/10 · confidence 3/52023-07-09

Summary

the submission extends the work on privacy guarantees specifically for protecting data membership inference attacks, and the extension is on using the Gaussian distribution to characterize the trade-off function. The proposed Gaussian Membership Inference Privacy has the same parametrization as the Gaussian Differential Privacy, but provides weaker privacy guarantees in the sense that an attacker only has access to the learned machine learning model and true data distribution or the public data. The submission further shows that a single step of noisy SGD is approximately Gaussian membership Inference privacy with a privacy parameter $\mu$, and empirically shows that the theoretical trade-off curve is the upper bound of the ROC curves of practical attacks with the given public knowledge.

Strengths

1. the membership inference attack captures a class of realistic attacks against machine learning models, where the attacker can't manipulate the private training data but has access to the data distribution and the learned model through an API. It indeed provides weaker privacy guarantees than Differential Privacy does, but studying it might lead us to a more practical privacy definition for protecting individuals in machine learning systems. 2. one-step noisy SGD can be captured by f-membership inference privacy, and can be captured approximately by Gaussian Membership Inference privacy. 3. a very important observation is that averaging over norm-clipped gradients satisfies GMIP with a bounded privacy parameter, which means that it already protects, to some extent, against membership inference attacks on gradients.

Weaknesses

1. Gaussian Differential Privacy exactly captures the trade-off function of the Gaussian mechanism, which is used to release noisy gradients. However, Gaussian Membership Inference Privacy only approximately captures so, and I am wondering what 'approximately' entails, and whether it is actually useful compared to f-membership inference privacy. 2. the assumption for the algorithm that an attacker would use is that the algorithm has to return binary answers, but we've seen successful attacks using ranking, for example, the watchdog experiments in [1]. Thus, I am questioning this assumption on top of an attacker's algorithm. 3. I was hoping to see experiments with real machine learning models with only API access to them, e.g. an attacker can only have access to the trained classification neural network through the final probabilistic predictions over classes. The distributional assumption over the susceptivity does not hold anymore, and it is interesting to see how the proposed privacy definition captures the success of membership inference attacks. [1] https://www.pnas.org/doi/epdf/10.1073/pnas.2218605120

Questions

please see the weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

please see the weaknesses.

Reviewer qYbt5/10 · confidence 3/52023-07-10

Summary

Authors came up with a privacy definition that is more relaxed than DP. It is called Gaussian Membership Inference Privacy (GMIP) which consists of a hypothesis testing which is supposed to decide whether a single instance is present in the training data. DP implies GMIP. The proposed privacy framework is applied with SGD and in experiments found to be much better than DP in terms of utility.

Strengths

* There is a need to come up with some privacy definition that is more practical than DP. * Research direction is promising.

Weaknesses

* I believe that standard notions from statistical hypothesis testing are reinvented, and the results seems not so surprising taken into account results already published in testing. I would suggest to reuse those results (if it is possible). Nevertheless, please address my questions!

Questions

* Definition 4.1 is basically the most powerfull test. $\mathcal{E}$ is the significance and $\beta$ is the power function, or more concretely 1 - power function? * Regarding Theorem 4.1: according to the Neyman-Pearson fundamental lemma the risk function is always convex when the null and alternative consist of a single distribution. And the testing problem presented in (3) is like that. See Lehamnn-Romano: Testing Statistical Hypotheses, Sec 3.2. * Testing are applied in many rounds and i guess only union bound is used over the individual SGD steps. Some sequential hypthesis testing might be worth to consider?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I believe that this research direction is interesting and promising. However, convexity of risk function seems not novel observation which is the key of having most uniformly powerful test. And this is what the paper relies on. I do not understand why to stick to Gaussion distribution, since any distribution from the exponential family can be used in a very similar way. Furthermore, FNR are controlled in each SGD step independently from each other however sequential testing approaches might applied here which would make this paper much more interesting. So technical contribution is somewhat limited.

Reviewer DVNo5/10 · confidence 4/52023-07-23

Summary

The paper proposes a notion of Gaussian membership inference privacy (GMIP) to capture the information leakage of a training algorithm about a data point $x$ when all the remaining training datasets are randomly drawn from a data distribution. The new GMIP definition has two main benefits compared to the prior leakage definition. 1. It captures the entire trade-off curve of membership inference as a hypothesis test and is thus more informative than prior average-case privacy definitions such as membership inference advantage. 2. It stochastically composes the trade-off curves over the randomly drawn remaining training dataset (other than the target data). It, therefore, captures leakage against a more realistic adversary that does not have control over the remaining training dataset (compared to the worst-case f-DP definition). Following this new GMIP definition, the paper analytically proves GMIP for the DP-SGD algorithm via an analytical likelihood ratio attack on an observed noisy gradient (under assumptions on the gradient distribution, model dimension, and dataset size). The proved GMIP bound has interesting dependencies on the model dimension and a data-dependent constant associated with the gradient distribution. To further illustrate the tightness of this GMIP bound, the authors evaluate the performance of various attacks for a mu-GMIP noisy SGD algorithm and show when the attack performances are close to the GMIP upper bound. Finally, the authors evaluate model accuracy under mu-GMIP and show that it improves over the model accuracy under mu-GDP, which indicates a privacy-utility trade-off gain due to relaxed adversary assumption.

Strengths

- The paper studies an important problem of analytically bounding informative information leakage of training algorithms against realistic adversaries. - The critical component is a novel analytical derivation of the likelihood ratio test, assuming that the aggregated noisy gradients follow a multivariate Gaussian distribution. - The proved GMIP bound is novel and has interesting dependencies on various factors about the model and data distribution. The discussion about the tightness of the bound is detailed and supported by empirical evaluations.

Weaknesses

[W1] The main weakness is that the proved GMIP bound is based on several approximation arguments and relies on assumptions about the gradient distribution, the model dimension, and the dataset size. The authors should clarify these approximations and assumptions in the comparison in Figure 1. Otherwise, the comparison to the mu-GDP algorithm is unfair or misleading. [W2] Specifically, one of the assumptions used in approximating the LRT (line 618 appendix D.1) is that the distribution of averaged gradient follows a Gaussian distribution, provided that the number of averaged samples is large enough. This assumption is invalid because the gradients are clipped before averaging, so the distribution of averaged gradient is bounded and is not close to a Gaussian distribution. [W3] The proved GMIP bound Theorem 4.2. grows indefinitely with regard to model dimension d. Such dimension dependency does not exist in the standard mu-GDP bound for DP-SGD (which implies GMIP). This suggests that the GMIP bound may be less tight than the standard mu-GDP bound for the high-dimensional problem, which is the case for training a deep neural network.

Questions

[Q1] What exactly are the approximations and assumptions used for the proved GMIP bound, for the comparison figure 1? How much would possible issues with assumptions (such as the one mentioned in weakness [W2] and insufficiently large dataset size n and model dimension d) break the proved GMIP bound? [Q2] Is the proved GMIP bound less tight than the standard mu-GDP bound for DP-SGD, when the model dimension is large? See weakness [W3] for more information.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The limitations regarding assumptions made for the analysis should be discussed more. See weakness [W1] and [W2] for more details.

Reviewer Uu8E2023-08-12

Thank you for the detailed clarification and including the low FPR plots. I think the table is a great addition, however, I think that some statements about the usefulness and the relevance of the threat model are overstated. > The sample $x'$ for which membership is to be inferred is drawn from the data distribution $D$. Therefore, MI is concerned with typical samples that can occur in practice I would argue that inserting a worst case sample is an easy task for an adversary in practice e.g. Census data, recommendation datasets, language modelling datasets. However, I admit that there are applications where the threat model in this paper holds and worst case samples are hard to insert e.g. medical datasets, etc. I think a clear discussion of when the proposed threat model is applicable and **also when not** would strengthen the paper. The table is a great first step.

Authorsrebuttal2023-08-14

Reply to Official Comment by Reviewer Uu8E

Thank you for your positive reply. We agree with the reviewer’s observations that the threat model is useful for some applications, but may not always be applicable as it requires an adversary with restricted dataset access: - In financial and healthcare applications, the data is often collected from actual events (e.g., past trades) or only a handful of people (i.e., trusted hospital staff) have access to the records. In such scenarios, it might be overly restrictive to protect against worst-case canary attacks as attackers cannot freely inject arbitrary records into the training datasets. - In other cases such as online surveys and census data, as mentioned by the Reviewer, an attacker may indeed be able to do sample injection. These attacks are not covered by $f$-Membership Inference Privacy and a more general notion such as Differential Privacy should be preferred. We will happily add these clarifications regarding to our paper. We will also extend the table by another line with recommended example usages: | | $f$-Differential Privacy | $f$-Membership Inference Privacy | | | | | | | | |---------------------------|---------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---|---|---|---|---|---|---| | Best used in applications | where specific attack model is unknown. Offers a form of general protection. | where dataset access (e.g. canary injection) of an attacker can be ruled out and the main attack goal lies in revealing private training data (e.g., membership inference, data reconstruction). | | | | | | | | | | | Let us know what you think. We would be happy to incorporate further suggestions!

Reviewer Uu8E2023-08-14

Thanks. I think this addition will be help the reader navigate the space. I've increased my overall score.

Reviewer vdUQ2023-08-13

I appreciate the responses provided by the authors; these have certainly improved my understanding of the paper. Nevertheless, concerning the composition rule, I am still uncertain about how to obtain the same result as proven in GDP. Could you kindly offer a step-by-step heuristic illustration?

Authorsrebuttal2023-08-14

Reply to Reviewer vdUQ

Thank you for your kind reply. Our general composition result (Lemma 4.1) follows as an application of Corollary 4 from Dong et al., which directly follows from Theorem 11 from Dong et al. Their Theorem does not only hold for f-DP in particular, but more generally as it only relies on the premise that *‘[...] $f$ is a symmetric trade-off function such that $f(0) = 1$ [...]’*. Note that our Gaussian trade-off function from Corollary 5.1 fulfills this condition as it fulfills all criteria of a trade-off function and is symmetric. Hence, we can directly apply Theorem 11 from Dong et al. to our setting. We are also happy to provide the outline of the proof of the specific composition result for Gaussian trade-offs which also highlights why their theorems are generally applicable to our trade-off functions. To this end, we consider Theorem 4 by Dong et al., which covers finite compositions (whereas Theorem 11 covers an infinite number of compositions). While this Theorem is stated in terms of $f_i$-DP, note that the $f_i$s are general (not DP specific) trade-off functions that originate from comparing two distributions against one another using hypothesis testing. The proof then proceeds in the following steps: Consider 2 sequential tests with trade-off functions $f$ and $g$. We want to compose the results from these two tests in the best possible way (i.e., what is the best trade-off the attacker can obtain that features the results from both tests $f$ and $g$). Trade-off functions can equivalently be represented by pairs of distributions that are being tested against each other, e.g., $f = \text{Test}(P, Q)$, where $P$ and $Q$ are distributions. Theorem 4 of Dong et al. states that composing tests $f=Test(P,Q)$ and $g=Test(P’,Q’)$ results in the combined trade-off $f \otimes g = \text{Test}(P \times P’, Q \times Q’)$, where $\times$ denotes the joint distribution and $\otimes$ the composed trade-off. As shown in their Lemma C.2, the choice of representation (in terms of $P, Q$) does not matter and has no effect on the result. This part of their proof implies that we can apply their result to testing distributions stemming from both f-DP and f-MIP, as long as we can characterize the tests’ trade-off curves. As an example, suppose that we are specifically composing the Gaussian trade-offs $g_{\mu_1}$ and $g_{\mu_2}$ with parameters $\mu_1$ and $\mu_2$. Recall that the $g_{\mu} = \text{Test}(N (0, 1), N (\mu, 1))$ is defined via testing two unit Gaussians of variance one at distance $\mu$ to each other. We can then do the following calculation: $g_{\mu_1} \otimes g_{\mu_2} = \text{Test}(N (0, 1), N (\mu_1, 1)) \otimes \text{Test}(N (0, 1), N (\mu_2, 1))$ $= \text{Test}(N(0, 1)\times N (0, 1), N (\mu_1, 1) \times N (\mu_2, 1))$ $= \text{Test}(N ((0,0)^\top, \mathbb{I}), N ((\mu_1,\mu_2)^\top, \mathbb{I})$ $= \text{Test}(N ((0,0)^\top, \mathbb{I}), N ((\sqrt{\mu_1^2+\mu_2^2}, 0)^\top, \mathbb{I}) = \text{Test}(N (0, 1) \times N (0, 1), N (\sqrt{\mu_1^2+\mu_2^2}, 1)\times N (0, 1))$ $= \text{Test}(N (0, 1), N (\sqrt{\mu_1^2+\mu_2^2}, 1)) \otimes \text{Test}(N (0, 1), N (0, 1)) = g_{\sqrt{\mu_1^2+\mu_2^2}} \otimes \text{Id} =g_{\sqrt{\mu_1^2+\mu_2^2}}$ We use the fact that a rotation does not change the hardness of the test. The last line follows from the fact that composition with the $\text{Id}$ test (testing two identical distributions) contains no information (formally proven in Dong et al.) and $\mathbb{I}$ denotes a 2x2 unit matrix. We hope that our clarifications regarding the applicability of Theorem 11/Corollary 4 from Dong et al. and the simple outline of the underlying proof for Gaussian compositions clear the reviewer’s uncertainty. Please let us know if you have further questions.

Reviewer vdUQ2023-08-17

I would like to thank the authors for the answer. I have adjusted my overall score based on the responses.

Reviewer DVNo2023-08-13

Thanks for the rebuttal

Thanks the authors for clarifying their CLT argument for the mean of clipped gradient distribution. It definitely clarifies my doubt about the application scope of the bound. However, the CLT argument is an approximation that is only exact at infinite $n$. The fact that the authors are analyzing the converged Gaussian distribution due to CLT argument makes their current upper bound invalid for any finite $n$ with certain probability. To this end, a correct bound would require error correction terms related to using CLT (similar to [Theorem 3.7, Dong et al.]). I'd like to request the authors to update the statement of their GMIP bound to either add explicit error correction terms due to approximations, or add precise descriptions of the assumptions required for the bound to hold (e.g. the gradient mean exactly follows the Gaussian distribution). Another remaining concern (which is direct consequence of the approximation error mentioned above) is that the proved privacy bound grows with model dimension, and is less tight than the dimension-independent Gaussian DP bound for DP-SGD under large model dimensions. (As the authors acknowledge in the rebuttal.) I still find the work interesting, and as the authors point out, it is the first-time that a trade-off function under MIA is analytically estimated. However, it also has the above two important limitations and therefore I'm keeping my score still as borderline accept.

Authorsrebuttal2023-08-14

Reply to Reviewer DVNo

Thank you for your thoughtful reply. We will be happy to make this point more clear in our manuscript. We will also discuss how Berry-Esseen’s theorem, which yields an error of order $\mathcal{O}(1/\sqrt{n})$, might be used to bound the error. It is also worthwhile to check whether the bound of Dong et al. (pointed out by the reviewer) or its proving technique are applicable. Their bound provides an even faster convergence rate of $\mathcal{O}(1/n)$. Finally, we would like to stress that our empirical studies suggest that the error seems to be negligible even for moderate $n$ in the general and in the relevant low-FPR regime. Thank you for your time and your continued support of our work.

Reviewer qYbt2023-08-17

Thanks for the rebuttal.

Authors addressed my concerns properly. I liked the topic of the paper because to better understand inference attacks is a very timely question. Therefore I recommend the paper be accepted.

Authorsrebuttal2023-08-21

Update

As the discussion period comes to a close, we would like to thank the reviewers for actively participating in the discourse and, where relevant, for their thoughtful reevaluation of the overall scores. We are fully committed to addressing the points discussed, which allow for valuable improvements to our manuscript. Let us know if you have any further questions.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC