Nearly Tight Black-Box Auditing of Differentially Private Machine Learning

Summary

This paper studies the problem of auditing DP-SGD in the black box threat model, i.e. only black-box access to the last iterate $\theta_T$ of DP-SGD. The paper's main contribution is to show experimentally that pre-training helps to get tighter auditing. The argument behind using this idea is that pre-training makes the average gradient norm of “in-distribution” points small, thus making the target point used for auditing/membership attack more “distinguishable”. The paper explores this worst-case initialisation idea for two datasets, MNIST and CIFAR10, and reports tighter privacy lower bounds.

Strengths

- Well-motivated problem. Tight black-box auditing is still an open problem compared to the white-box setting. - Extensive experimental exploration of the effect of different hyperparameters on the auditing procedure. - The paper provides confidence intervals on the empirical privacy budget $\epsilon$.

Weaknesses

- The main contribution of this work is incremental. In addition, as noted in the paper, the idea that the initialisation of $\theta_0$ affects auditing/membership inference has already been reported in the literature [10,18,26]. - The justification for using pre-training is hand-wavy and not rigorous. See the question below. - No code is available to reproduce the results. The experimental setting is well-detailed. However, some important factors for reproducing the results have not been reported. See questions below.

Questions

- Worst-case vs average-case initialisation: (a) For the worst-case, $\theta_0$ is selected by pretraining on half of the dataset, but what algorithm is used for that? Is it non-private SGD? And what is the random distribution used to generate $\theta_0$ for the average case? - In Figure 5, the average-case initialisation seems to give tighter privacy and lower bounds, especially for small epsilon. This contradicts this work's main argument, i.e., using worst-case initialization is better. How do you explain this? - What canary strategy is used for the black-box audit? Do you only use the blank example? Did you test any other black box canaries? - The justification behind the reported success of pretraining is that the average gradient norm decreases with the number of pre-training epochs, reported in Figure 2. (a) Is the average of gradients computed on all samples (training+ test) or only training samples? (b) Why does this correlate with the success of your membership attack? Specifically, the MIA attack tresholds over the loss of the canary, thus this seems to be the statistic that should be analysed over the pre-training epochs. The gradient average norm is indicative for white-box attacks. (c) Pretraining decreases the gradient average loss of “in distribution” samples, so how would you adapt your auditing procedure for datasets where the empty sample, i.e. your canary, is "in distribution"? - What are the loss thresholds used for the black-box membership attack? Are they fine-tuned to find the best privacy budget? Also, the function EstimateEps() is never explicitly detailed in the paper to reproduce the results. - Is it not already possible to use [32]’s framework to run directly your black-box auditing in one run?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

As noted by the authors in the conclusion, the auditing procedure is computationally heavy and time expensive.

Summary

The paper considers the problem of auditing DP-SGD, i.e. deriving empirical lower bounds on the DP parameter $\epsilon$. Many previous auditing works operated in the white-box setting, i.e. are allowed to choose arbitrary gradients for the canary (sensitive example) during DP-SGD, or were in the black-box setting (i.e. could determine some aspects of training, but not specific per-round gradients, and also does not get to see intermediate states) but had a sizeable gap between the empirical $\epsilon$ and the theoretical upper bound. The paper gives a black-box auditing procedure that in some cases achieves empirical $\epsilon$ very close to the theoretical upper bound on standard benchmarks of MNIST and CIFAR10. The main algorithmic insight is to use a model pre-trained on in-distribution data, which causes the gradients of in-distribution examples to be quite small even in the first round of private training, hence increasing the impact of the canary on training. In contrast, previous work used 'average-case' initializations without optimizing for their impact on the empirical $\epsilon$. The authors perform a number of experiments using this improvement, on MNIST and CIFAR, varying whether they train a full model vs the last layer, and varying the dataset size and clip norm. The authors' audit generally improves upon the baseline of an average-case initialization, sometimes as much as a 5x increase in the empirical $\epsilon$.

Strengths

* The paper provides the first audits which come close to matching the theoretical upper bounds while operating in a black-box setting. In particular the empirical improvement over past black-box work is incredibly strong, and the qualitative improvement over white-box audits is important given the threat model the authors use is much more realistic. * The empirical results are especially impressive given the number of training runs the authors use (which is much smaller than some other auditing works) and the fact that the change needed to achieve the increase in empirical $\epsilon$ is relatively lightweight (a single pre-training run). * The authors do a good job explaining intuition for why changing different aspects of the training pipeline affects the empirical $\epsilon$ which makes it more likely others can build upon this intuition in practice or in future work, and the intuition is reflected in the empirical results.

Weaknesses

* In some cases we may want to do privacy auditing not to e.g. find bugs in an existing DP-SGD pipeline, but to support the claim that a model has much better privacy guarantees than the worst-case theoretical $\epsilon$ we are reporting. In this case, taking half of the dataset and using it as public pre-training data a privacy violation, i.e. it is unclear how to extend the authors' insights to this application of auditing. This might be a limited weakness, as anyway in practice we are probably going to try to pre-train on a distribution as close to the private distribution as possible, which might retrieve the benefits the authors observe. * The improvements in Figure 5 don't seem to be very significant, and for eps = 1.0 the average empirical epsilon of average-case initialization is in fact double that of worst-case initialization. This did not affect my score as I understand the authors may not have access to enough compute budget to narrow the confidence intervals, and also I don't think they should be punished for reporting insignificant results, but I think the improvement here is somewhat overstated in the paper.

Questions

* Do the authors believe the results in Figure 5 would become significant with more trials? i.e. do the authors have some intuition for why an initialization might matter more or less for last-layer-only fine-tuning than for full-model training? If so, it might be nice to include this in the paper.

Rating

6

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper proposes a new method for auditing DP-SGD in a black-box setting, where the auditor can only see the final parameters of the model (rather than intermediate steps). The main idea is to select worst-case initial model parameters; this seems to give a substantial advantage to the black-box analysis.

Strengths

The idea of selecting worst-case initial parameters for auditing is clever. Empirically, the proposed method strongly outperforms the chosen baseline. This work shows that a (partially) black-box adversary isn't much weaker than a white-box one; this demonstrates that DP analyses (which assume the latter) may not be underestimating privacy too much. The paper is well-presented.

Weaknesses

# Motivations I think the authors should refine their motivations. They use two: 1) their method is good for detecting bugs, and 2) it provides insights into DP-SGD analyses. 1. Bugs. The ultimate purpose of _black-box_ auditing is unclear. There's two sides to this discussion. On the one hand, if our goal is to audit a DP-SGD implementation, surely we should be making all the worst-case assumptions (i.e., give as much advantage to the auditor as possible) to look for bugs. On the other hand, if our goal is to evaluate real-world attacks against DP-SGD, then it does make sense to consider a black-box assumption. The assumptions this paper makes are a bit of an hybrid: it assumes black-box for auditing, yet the adversary is allowed to specify the initial model parameters. The only semi-practical scenario I can think of is an attacker who creates a model (for subsequent fine-tuning to be carried out by the victim) and wants to augment the leakage in the fine-tuned version of the model. Aside from this scenario, I would recommend leaving out from this paper any motivation arguing that this work is more practical than other auditing methods for finding "bugs"; please refer to "Questions" for a concrete example. 2. New insights. The authors do provide one very convincing motivation: a big question is whether DP analyses of DP-SGD could be improved if they considered it as a black-box algorithm; currently, they all assume knowledge of intermediate gradients. This work offers some hope: from the analysis, it seems that the difference between black-box and the theoretically estimated "epsilon" may be closer than expected. If I may, I would strongly recommend the authors to center their motivations on this aspect. # Comparison I could not find a real comparison to previous auditing methods. Only two numbers appear (within the Datasets paragraph), reporting the performance of previous works; were these computed under the same experimental setting (and code base)? Ideally, they should be replicated under identical conditions to yours. Secondly, it'd be very interesting to see what similar white-box methods achieve, in comparison. How tight are they, on those datasets/models?

Questions

- "Although techniques to tightly audit DP-SGD exist in literature [26, 27], they do so only in active white-box threat models, where the adversary can observe and insert arbitrary gradients into the intermediate DP-SGD steps. However, real-world adversaries cannot always access the model’s inner parameters, let alone insert arbitrary gradients." As argued above, I don't think this is a valid premise. There's nothing in auditing that requires modelling a real-world adversary. Further, the assumptions that you make (e.g., worst-case initial parameters) do not correspond to a real-world adversary. I think the implications of what you claim here are incorrect, and I don't think it should be part of your motivations. - Threat model. It does look weird to see a threat model section in an auditing paper, and I would humbly recommend removing it: DP doesn't specify any particular threat model, and auditing is just about ensuring how tightly the definition is being matched empirically. If you do want to keep this section, please note that it's missing the assumption that the auditor (adversary?) can specify the initial model parameters; also, I was unclear as to what the following meant in this context: "assume that the adversary can choose a worst-case target sample as is standard for auditing DP mechanisms". - "we set B=N". Can you please clarify whether this is a constraint of your method, and if so what are the potential limitations? - Sentence "When auditing a DP mechanism [...]". Please define the symbol $R$. (It's clear what it is, but it's undefined.) - "Crafting worst-case initial parameters". This should be one of the main contributions, but unfortunately the paragraph doesn't provide much information on how exactly these parameters are created. Can you please provide more information in the text (and in rebuttal)? - Could you please double check: did your cluster only have 1 CPU core? - Typo "is is".

Rating

6

Confidence

3

Soundness

4

Presentation

3

Contribution

2

Limitations

These were adequately addressed.

Summary

The paper shows that privacy auditing is tight in the threat model where the model initialization is adversarially selected. They find that the decrease in gradient norms over the course of training helps improve the "signal to noise" ratio of the auditing example.

Strengths

The paper's main finding is useful and helps the privacy auditing community get a bit closer to understanding privacy leakage. The authors propose a reasonable hypothesis for why their approach improves on prior work.

Weaknesses

The general finding that "when other examples' contributions are smaller, auditing is tighter", has been a feature of other papers on auditing, even in the adversarial dataset experiments from Nasr et al. 2021 or the design of ClipBKD in Jagielski et al 2020. The paper's claims of "near tightness" seem somewhat exaggerated - there are several parameter settings where the proposed auditing does not even appear to be better than prior work, and all experiments show a rather large gap. Given that the batch size is an important detail for auditing, the setting of B = n should be mentioned in the main body of the paper. The experiments are unfortunately on rather small models, making it difficult to know how generalizable the strategy is to more recent advances in DP-ML training. I would appreciate experiments on other datasets - several conclusions are drawn that seem dataset specific (e.g. dataset size), and it would be useful to know if there are more general trends in these.

Questions

What is the gradient norm of the auditing example in your experiments? I wonder if at C=10, the gradient norm is not large enough to "max out" the full clipping norm. Did you retune hyperparameters when changing the values of clipping norms? Given the hypothesis that lower gradient norm of other examples leads to tighter audits, have you tried explicitly constructing an initialization where the other gradients are exactly 0? Perhaps even just nonprivate SGD on the private training data will get there.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

I think the batch size should be better highlighted, but limitations related to model architectures, datasets, auditing trials are all well addressed.