SPLASH: Learnable Activation Functions for Improving Accuracy and Adversarial Robustness

We introduce SPLASH units, a class of learnable activation functions shown to\nsimultaneously improve the accuracy of deep neural networks while also\nimproving their robustness to adversarial attacks. SPLASH units have both a\nsimple parameterization and maintain the ability to approximate a wide range of\nnon-linear functions. SPLASH units are: 1) continuous; 2) grounded (f(0) = 0);\n3) use symmetric hinges; and 4) the locations of the hinges are derived\ndirectly from the data (i.e. no learning required). Compared to nine other\nlearned and fixed activation functions, including ReLU and its variants, SPLASH\nunits show superior performance across three datasets (MNIST, CIFAR-10, and\nCIFAR-100) and four architectures (LeNet5, All-CNN, ResNet-20, and\nNetwork-in-Network). Furthermore, we show that SPLASH units significantly\nincrease the robustness of deep neural networks to adversarial attacks. Our\nexperiments on both black-box and open-box adversarial attacks show that\ncommonly-used architectures, namely LeNet5, All-CNN, ResNet-20, and\nNetwork-in-Network, can be up to 31% more robust to adversarial attacks by\nsimply using SPLASH units instead of ReLUs.\n