ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies

Empirical adversarial risk minimization (EARM) is a widely used mathematical\nframework to robustly train deep neural nets (DNNs) that are resistant to\nadversarial attacks. However, both natural and robust accuracies, in\nclassifying clean and adversarial images, respectively, of the trained robust\nmodels are far from satisfactory. In this work, we unify the theory of optimal\ncontrol of transport equations with the practice of training and testing of\nResNets. Based on this unified viewpoint, we propose a simple yet effective\nResNets ensemble algorithm to boost the accuracy of the robustly trained model\non both clean and adversarial images. The proposed algorithm consists of two\ncomponents: First, we modify the base ResNets by injecting a variance specified\nGaussian noise to the output of each residual mapping. Second, we average over\nthe production of multiple jointly trained modified ResNets to get the final\nprediction. These two steps give an approximation to the Feynman-Kac formula\nfor representing the solution of a transport equation with viscosity, or a\nconvection-diffusion equation. For the CIFAR10 benchmark, this simple algorithm\nleads to a robust model with a natural accuracy of {\\bf 85.62}\\% on clean\nimages and a robust accuracy of ${\\bf 57.94 \\%}$ under the 20 iterations of the\nIFGSM attack, which outperforms the current state-of-the-art in defending\nagainst IFGSM attack on the CIFAR10. Both natural and robust accuracies of the\nproposed ResNets ensemble can be improved dynamically as the building block\nResNet advances. The code is available at:\n\\url{https://github.com/BaoWangMath/EnResNet}.\n