Investigation and Analysis of Hyper and Hypo neuron pruning to selectively update neurons during Unsupervised Adaptation

Unseen or out-of-domain data can seriously degrade the performance of a\nneural network model, indicating the model's failure to generalize to unseen\ndata. Neural net pruning can not only help to reduce a model's size but can\nimprove the model's generalization capacity as well. Pruning approaches look\nfor low-salient neurons that are less contributive to a model's decision and\nhence can be removed from the model. This work investigates if pruning\napproaches are successful in detecting neurons that are either high-salient\n(mostly active or hyper) or low-salient (barely active or hypo), and whether\nremoval of such neurons can help to improve the model's generalization\ncapacity. Traditional blind adaptation techniques update either the whole or a\nsubset of layers, but have never explored selectively updating individual\nneurons across one or more layers. Focusing on the fully connected layers of a\nconvolutional neural network (CNN), this work shows that it may be possible to\nselectively adapt certain neurons (consisting of the hyper and the hypo\nneurons) first, followed by a full-network fine tuning. Using the task of\nautomatic speech recognition, this work demonstrates how the removal of hyper\nand hypo neurons from a model can improve the model's performance on\nout-of-domain speech data and how selective neuron adaptation can ensure\nimproved performance when compared to traditional blind model adaptation.\n