As neural networks gain widespread adoption in embedded devices, there is a\nneed for model compression techniques to facilitate deployment in\nresource-constrained environments. Quantization is one of the go-to methods\nyielding state-of-the-art model compression. Most approaches take a fully\ntrained model, apply different heuristics to determine the optimal\nbit-precision for different layers of the network, and retrain the network to\nregain any drop in accuracy. Based on Activation Density (AD)-the proportion of\nnon-zero activations in a layer-we propose an in-training quantization method.\nOur method calculates bit-width for each layer during training yielding a mixed\nprecision model with competitive accuracy. Since we train lower precision\nmodels during training, our approach yields the final quantized model at lower\ntraining complexity and also eliminates the need for re-training. We run\nexperiments on benchmark datasets like CIFAR-10, CIFAR-100, TinyImagenet on\nVGG19/ResNet18 architectures and report the accuracy and energy estimates for\nthe same. We achieve ~4.5x benefit in terms of estimated\nmultiply-and-accumulate (MAC) reduction while reducing the training complexity\nby 50% in our experiments. To further evaluate the energy benefits of our\nproposed method, we develop a mixed-precision scalable Process In Memory (PIM)\nhardware accelerator platform. The hardware platform incorporates shift-add\nfunctionality for handling multi-bit precision neural network models.\nEvaluating the quantized models obtained with our proposed method on the PIM\nplatform yields ~5x energy reduction compared to 16-bit models. Additionally,\nwe find that integrating AD based quantization with AD based pruning (both\nconducted during training) yields up to ~198x and ~44x energy reductions for\nVGG19 and ResNet18 architectures respectively on PIM platform compared to\nbaseline 16-bit precision, unpruned models.\n