Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning

Rectified linear unit (ReLU) activations can also be thought of as 'gates',\nwhich, either pass or stop their pre-activation input when they are 'on' (when\nthe pre-activation input is positive) or 'off' (when the pre-activation input\nis negative) respectively. A deep neural network (DNN) with ReLU activations\nhas many gates, and the on/off status of each gate changes across input\nexamples as well as network weights. For a given input example, only a subset\nof gates are 'active', i.e., on, and the sub-network of weights connected to\nthese active gates is responsible for producing the output. At randomised\ninitialisation, the active sub-network corresponding to a given input example\nis random. During training, as the weights are learnt, the active sub-networks\nare also learnt, and potentially hold very valuable information. In this paper,\nwe analytically characterise the role of active sub-networks in deep learning.\nTo this end, we encode the on/off state of the gates of a given input in a\nnovel 'neural path feature' (NPF), and the weights of the DNN are encoded in a\nnovel 'neural path value' (NPV). Further, we show that the output of network is\nindeed the inner product of NPF and NPV. The main result of the paper shows\nthat the 'neural path kernel' associated with the NPF is a fundamental quantity\nthat characterises the information stored in the gates of a DNN. We show via\nexperiments (on MNIST and CIFAR-10) that in standard DNNs with ReLU activations\nNPFs are learnt during training and such learning is key for generalisation.\nFurthermore, NPFs and NPVs can be learnt in two separate networks and such\nlearning also generalises well in experiments.\n