Adam in Private: Secure and Fast Training of Deep Neural Networks with Adaptive Moment Estimation

Privacy-preserving machine learning (PPML) aims at enabling machine learning\n(ML) algorithms to be used on sensitive data. We contribute to this line of\nresearch by proposing a framework that allows efficient and secure evaluation\nof full-fledged state-of-the-art ML algorithms via secure multi-party\ncomputation (MPC). This is in contrast to most prior works, which substitute ML\nalgorithms with approximated "MPC-friendly" variants. A drawback of the latter\napproach is that fine-tuning of the combined ML and MPC algorithms is required,\nwhich might lead to less efficient algorithms or inferior quality ML. This is\nan issue for secure deep neural networks (DNN) training in particular, as this\ninvolves arithmetic algorithms thought to be "MPC-unfriendly", namely, integer\ndivision, exponentiation, inversion, and square root. In this work, we propose\nsecure and efficient protocols for the above seemingly MPC-unfriendly\ncomputations. Our protocols are three-party protocols in the honest-majority\nsetting, and we propose both passively secure and actively secure with abort\nvariants. A notable feature of our protocols is that they simultaneously\nprovide high accuracy and efficiency. This framework enables us to efficiently\nand securely compute modern ML algorithms such as Adam and the softmax function\n"as is", without resorting to approximations. As a result, we obtain secure DNN\ntraining that outperforms state-of-the-art three-party systems; our full\ntraining is up to 6.7 times faster than just the online phase of the recently\nproposed FALCON@PETS'21 on a standard benchmark network. We further perform\nmeasurements on real-world DNNs, AlexNet and VGG16. The performance of our\nframework is up to a factor of about 12-14 faster for AlexNet and 46-48 faster\nfor VGG16 to achieve an accuracy of 70% and 75%, respectively, when compared to\nFALCON.\n