Modern deep networks have millions to billions of parameters, which leads to\nhigh memory and energy requirements during training as well as during inference\non resource-constrained edge devices. Consequently, pruning techniques have\nbeen proposed that remove less significant weights in deep networks, thereby\nreducing their memory and computational requirements. Pruning is usually\nperformed after training the original network, and is followed by further\nretraining to compensate for the accuracy loss incurred during pruning. The\nprune-and-retrain procedure is repeated iteratively until an optimum tradeoff\nbetween accuracy and efficiency is reached. However, such iterative retraining\nadds to the overall training complexity of the network. In this work, we\npropose a dynamic pruning-while-training procedure, wherein we prune filters of\nthe convolutional layers of a deep network during training itself, thereby\nprecluding the need for separate retraining. We evaluate our dynamic\npruning-while-training approach with three different pre-existing pruning\nstrategies, viz. mean activation-based pruning, random pruning, and L1\nnormalization-based pruning. Our results for VGG-16 trained on CIFAR10 shows\nthat L1 normalization provides the best performance among all the techniques\nexplored in this work with less than 1% drop in accuracy after pruning 80% of\nthe filters compared to the original network. We further evaluated the L1\nnormalization based pruning mechanism on CIFAR100. Results indicate that\npruning while training yields a compressed network with almost no accuracy loss\nafter pruning 50% of the filters compared to the original network and ~5% loss\nfor high pruning rates (>80%). The proposed pruning methodology yields 41%\nreduction in the number of computations and memory accesses during training for\nCIFAR10, CIFAR100 and ImageNet compared to training with retraining for 10\nepochs .\n