Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets

Deep neural nets (DNNs) compression is crucial for adaptation to mobile\ndevices. Though many successful algorithms exist to compress naturally trained\nDNNs, developing efficient and stable compression algorithms for robustly\ntrained DNNs remains widely open. In this paper, we focus on a co-design of\nefficient DNN compression algorithms and sparse neural architectures for robust\nand accurate deep learning. Such a co-design enables us to advance the goal of\naccommodating both sparsity and robustness. With this objective in mind, we\nleverage the relaxed augmented Lagrangian based algorithms to prune the weights\nof adversarially trained DNNs, at both structured and unstructured levels.\nUsing a Feynman-Kac formalism principled robust and sparse DNNs, we can at\nleast double the channel sparsity of the adversarially trained ResNet20 for\nCIFAR10 classification, meanwhile, improve the natural accuracy by $8.69$\\% and\nthe robust accuracy under the benchmark $20$ iterations of IFGSM attack by\n$5.42$\\%. The code is available at\n\\url{https://github.com/BaoWangMath/rvsm-rgsm-admm}.\n