Towards Enhancing Fault Tolerance in Neural Networks

Deep Learning Accelerators and Neuromorphic hardware, used in many real-time safety-critical applications, are prone to faults that manifest in the form of errors in Neural Networks. Fault Tolerance in Neural Networks is a critical attribute for applications that require reliable computation for long duration such as IoT and mobile devices. The inherent fault tolerance of Neural Networks can be improved with regularization, however, the current techniques exhibit a trade-off between generalization and classification accuracy. To this extent, in this work, a Neural Network is modelled as two distinct functional components: a Feature Extractor with an unsupervised learning objective and a Fully Connected Classifier with a supervised learning objective. Traditional approaches to train the entire network using a single supervised learning objective are insufficient to achieve the objectives of the individual functional goals optimally. In this work, a novel two phase framework with multi-criteria objective function combining unsupervised training of the Feature Extractor followed by supervised training of the Classifier Network is proposed. In the Phase I, the unsupervised training of the Feature Extractor is modeled using two games solved simultaneously in the presence of Neural Networks with conflicting objectives. The first game with a generative model, trains the Feature Extractor to generate robust features for the input image by minimizing a reconstruction loss between the input and reconstructed image. The second game with a binary classification network, updates the Feature Extractor to smoothen the feature space and match with a prior Gaussian distribution. In Phase II, the resultant Feature Extractor, which is strongly regularized, is combined with the Fully Connected Classifier for fine-tuning on the classification task. The proposed two phase training algorithm is evaluated on four architectures with varying model complexity on standard image classification datasets: FashionMNIST and CIFAR10. The proposed framework is scalable and independent of the network architecture that provides superior tolerance to stuck at “0” faults as compared to existing regularization functions without loss in classification accuracy.