Differentially private (DP) machine learning allows us to train models on\nprivate data while limiting data leakage. DP formalizes this data leakage\nthrough a cryptographic game, where an adversary must predict if a model was\ntrained on a dataset D, or a dataset D' that differs in just one example.If\nobserving the training algorithm does not meaningfully increase the adversary's\nodds of successfully guessing which dataset the model was trained on, then the\nalgorithm is said to be differentially private. Hence, the purpose of privacy\nanalysis is to upper bound the probability that any adversary could\nsuccessfully guess which dataset the model was trained on.In our paper, we\ninstantiate this hypothetical adversary in order to establish lower bounds on\nthe probability that this distinguishing game can be won. We use this adversary\nto evaluate the importance of the adversary capabilities allowed in the privacy\nanalysis of DP training algorithms.For DP-SGD, the most common method for\ntraining neural networks with differential privacy, our lower bounds are tight\nand match the theoretical upper bound. This implies that in order to prove\nbetter upper bounds, it will be necessary to make use of additional\nassumptions. Fortunately, we find that our attacks are significantly weaker\nwhen additional (realistic)restrictions are put in place on the adversary's\ncapabilities.Thus, in the practical setting common to many real-world\ndeployments, there is a gap between our lower bounds and the upper bounds\nprovided by the analysis: differential privacy is conservative and adversaries\nmay not be able to leak as much information as suggested by the theoretical\nbound.\n