Artificial Neural Networks (ANNs) are known as state-of-the-art techniques in\nMachine Learning (ML) and have achieved outstanding results in data-intensive\napplications, such as recognition, classification, and segmentation. These\nnetworks mostly use deep layers of convolution or fully connected layers with\nmany filters in each layer, demanding a large amount of data and tunable\nhyperparameters to achieve competitive accuracy. As a result, storage,\ncommunication, and computational costs of training (in particular training\ntime) become limiting factors to scale them up. In this paper, we propose a new\ntraining methodology for ANNs that exploits the observation of improvement of\naccuracy shows temporal variations which allow us to skip updating weights when\nthe variation is minuscule. During such time windows, we keep updating bias\nwhich ensures the network still trains and avoids overfitting; however, we\nselectively skip updating weights (and their time-consuming computations). Such\na training approach virtually achieves the same accuracy with considerably less\ncomputational cost, thus lower training time. We propose two methods for\nupdating weights and evaluate them by analyzing four state-of-the-art models,\nAlexNet, VGG-11, VGG-16, ResNet-18 on CIFAR datasets. On average, our two\nproposed methods called WUS and WUS+LR reduced the training time (compared to\nthe baseline) by 54%, and 50%, respectively on CIFAR-10; and 43% and 35% on\nCIFAR-100, respectively.\n