Do I Get the Privacy I Need? Benchmarking Utility in Differential Privacy Libraries

An increasing number of open-source libraries promise to bring differential\nprivacy to practice, even for non-experts. This paper studies five libraries\nthat offer differentially private analytics: Google DP, SmartNoise,\ndiffprivlib, diffpriv, and Chorus. We compare these libraries qualitatively\n(capabilities, features, and maturity) and quantitatively (utility and\nscalability) across four analytics queries (count, sum, mean, and variance)\nexecuted on synthetic and real-world datasets. We conclude that these libraries\nprovide similar utility (except in some notable scenarios). However, there are\nsignificant differences in the features provided, and we find that no single\nlibrary excels in all areas. Based on our results, we provide guidance for\npractitioners to help in choosing a suitable library, guidance for library\ndesigners to enhance their software, and guidance for researchers on open\nchallenges in differential privacy tools for non-experts.\n