On the Importance of Difficulty Calibration in Membership Inference Attacks

The vulnerability of machine learning models to membership inference attacks\nhas received much attention in recent years. However, existing attacks mostly\nremain impractical due to having high false positive rates, where non-member\nsamples are often erroneously predicted as members. This type of error makes\nthe predicted membership signal unreliable, especially since most samples are\nnon-members in real world applications. In this work, we argue that membership\ninference attacks can benefit drastically from \\emph{difficulty calibration},\nwhere an attack's predicted membership score is adjusted to the difficulty of\ncorrectly classifying the target sample. We show that difficulty calibration\ncan significantly reduce the false positive rate of a variety of existing\nattacks without a loss in accuracy.\n