Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvature

Summary

This paper conducts a theoretical and empirical investigation of the use of input loss curvatures in MIAs. They study the train-test input loss curvature scores and use this to develop a zero order input loss curvature black-box MIA.

Strengths

1. This paper provides an interesting theoretical explanation of MIAs based on input loss curvature that have performed well empirically but lacks theoretical understanding. 2. The new MIA is simple and motivated by theory, and the evaluation of it is thorough and clearly shows that this method is more performant that existing baselines. 3. The paper is well-written and clear.

Weaknesses

1. Missing some references, e.g. Li et al. 2023 or the original DetectGPT paper (Mitchell et al. 2023). 2. I believe the paper would benefit from including some sort of proof sketch of the theorems in the main body.

Questions

1. Other authors have used the Hutchinson trace estimator to estimate information about input loss curvature. How does the performance of this method compare to the zero-order method presented in the paper?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper builds the connection between privacy and input loss curvature. Core observation is that test samples often lie in higher curvature areas than those prototypical samples, so the input loss curvature can be a good metric to distinguish between train and test sets. Utilizing this observation, authors build a stronger black-box MIA, outperforming previous MIAs, and they also reveal a previously unknown limitation of shadow model based MIAs. The paper provides both theoretical analysis and experimental results.

Strengths

S1. The introduction of input loss curvature into the area of privacy attacks are original and novel. Previous works have revealed the utility of input loss curvature on coreset selection, adversarial robustness and model memorization, this work has further broadened its application. S2. This work proposed an advanced black-box MIA, which is more effective than previous attacks. Moreover, the study reveals one limitation of shadow-model based MIAs: when the dataset size used to train shadow models is similar to dataset size used to train target model, shadow-model based MIAs perform poorly.

Weaknesses

Line 117 missing a `.`

Questions

Q1: What is $s_f$, $L_f$ and $c_f$ in Line 300? The notation seems to be missing from the context. Q2: How accurate is the zeroth-order estimation of input loss curvature? Can you provide a comparison of this estimation to the real values (assuming we have white-box access to target model)

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations of this work is properly discussed in Appendix.

Summary

This paper develop a theoretical framework to derive an upper bound on train-test distinguishability based on input loss curvature. The authors propose a new black-box membership inference attack utilizing input loss curvature, which surpasses existing methods in membership inference effectiveness. The paper also explores the potential of using subsets of training data as a defense mechanism against shadow model-based membership inference attacks.

Strengths

Membership inference attack is an important and interesting problem and this paper discussed it from a theoretical perspective. The attacking method based on theoretical guarantees makes the method more solid. In the experiment, the curvature based attack proposed by this paper also exceeds other existing methods.

Weaknesses

1. In the theoretical analysis, the authors propose the train-test distinguishability distribution upper bound for both data and curvature. However, they compare the upper bounds directly and suggest that curvature based MIA is a better method when the size of dataset is large enough. The comparison between upper bounds may not reflect the real relationship. To validate this result, one may compare between an upper bound and a lower bound. 2. The assumption that the curvature score can be modeled with Gaussian distribution has not been discussed and validated in the paper, which is a very important assumtion in theoretical analysis. Figure 3 seems that the distribution is not very similar to Gaussian.

Questions

What is the effect of the error in the zeroth-order curvature estimation? How the estimated curvature deviates Gaussian and can we still model the distribution with it?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations of their work by acknowledging that their analysis only holds when the dataset size is large enough. They emphasize the need for further exploration of the method's applicability to different models and datasets and the potential impact of computational complexity.

Summary

The paper explores the use of loss curvature, under the hypothesis that test samples lie in high curvature regions, to improve the performance of membership inference attacks. It estimated the curvature by measuring the trace of the Hessian of the loss values. It demonstrates the viability of the approach in both white-box and black-box settings. It empirically validates the success of the approach on multiple datasets.

Strengths

This paper proposes a simple but effective improvement over the canonical baseline of using loss estimation on train and test samples for membership inference attacks. It tests its effectiveness on three datasets and compares it with key baselines in the previous works. It also validates its effectiveness against differentially private models that provide provable privacy guarantees to a certain degree.

Weaknesses

While the loss value based MIA evaluation generalizes to pretty much all class of models, it’s unclear if loss curvature based MIA attack would enable a similar generalization, i.e., going beyond the small-scale classification setup to self-supervised models, contrastive models, and other foundation models [1]. As fig. 6 itself highlights, the robustness of attack against a simple defense that subsamples the training data for shadow models is very low. Just using ½ of the training data renders the attack ineffective (albeit Carlini et al. suffers from the same limitation). Since shadow models are only a proxy of the true black-box model with training data information not publicly available, in practice they will likely only be trained on a subset of original training data of the block-box model, thus significantly reducing the attack success in black-box threat model. 1. Niu, Jun, Xiaoyan Zhu, Moxuan Zeng, Ge Zhang, Qingyang Zhao, Chunhui Huang, Yangming Zhang et al. "SoK: Comparing Different Membership Inference Attacks with a Comprehensive Benchmark." arXiv preprint arXiv:2307.06123 (2023).

Questions

Why not use zero-order loss estimation in black-box method as it would be significantly more sample efficient than block-box curvature estimation. Would using the first-order statistics, gradients of the loss w.r.t input, be as helpful as curvature in MIA attacks?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper proposes using second order statistics (loss curvature) over using loss values in membership attacks. However it will fail to generalize to networks that are piecewise linear, thus having zero curvature for all samples. A simple example of it would be the resnet models with convolution layer, ReLU activation, and Batchnorm normalization. While the paper does consider the resnet models, it doesn’t clarify the architecture uses. In the piecewise linear architecture, only the cross-entropy loss on output logits would contribute to loss curvature. Thus it’s critical to ablate the choice of network architecture and demonstrate that despite the diversity in architectural designs, the approach succeeds in MIA attacks. A few simple variations would be changing the normalization layers, activation functions, and broadly the class of models (CNNs vs transformers).

Previous concerns addressed in the rebuttal

Thanks for posting a detailed rebuttal to the review. I believe that it had addressed the concerns raised in the review, thus I've increased my score to accept.