Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks

Stochastic gradient descent (SGD) is the main approach for training deep\nnetworks: it moves towards the optimum of the cost function by iteratively\nupdating the parameters of a model in the direction of the gradient of the loss\nevaluated on a minibatch. Several variants of SGD have been proposed to make\nadaptive step sizes for each parameter (adaptive gradient) and take into\naccount the previous updates (momentum). Among several alternative of SGD the\nmost popular are AdaGrad, AdaDelta, RMSProp and Adam which scale coordinates of\nthe gradient by square roots of some form of averaging of the squared\ncoordinates in the past gradients and automatically adjust the learning rate on\na parameter basis. In this work, we compare Adam based variants based on the\ndifference between the present and the past gradients, the step size is\nadjusted for each parameter. We run several tests benchmarking proposed methods\nusing medical image data. The experiments are performed using ResNet50\narchitecture neural network. Moreover, we have tested ensemble of networks and\nthe fusion with ResNet50 trained with stochastic gradient descent. To combine\nthe set of ResNet50 the simple sum rule has been applied. Proposed ensemble\nobtains very high performance, it obtains accuracy comparable or better than\nactual state of the art. To improve reproducibility and research efficiency the\nMATLAB source code used for this research is available at GitHub:\nhttps://github.com/LorisNanni.\n