SMU: smooth activation function for deep networks using smoothing maximum technique

Deep learning researchers have a keen interest in proposing two new novel\nactivation functions which can boost network performance. A good choice of\nactivation function can have significant consequences in improving network\nperformance. A handcrafted activation is the most common choice in neural\nnetwork models. ReLU is the most common choice in the deep learning community\ndue to its simplicity though ReLU has some serious drawbacks. In this paper, we\nhave proposed a new novel activation function based on approximation of known\nactivation functions like Leaky ReLU, and we call this function Smooth Maximum\nUnit (SMU). Replacing ReLU by SMU, we have got 6.22% improvement in the\nCIFAR100 dataset with the ShuffleNet V2 model.\n