Complexity for deep neural networks and other characteristics of deep feature representations

We define a notion of complexity, which quantifies the nonlinearity of the\ncomputation of a neural network, as well as a complementary measure of the\neffective dimension of feature representations. We investigate these\nobservables both for trained networks for various datasets as well as explore\ntheir dynamics during training, uncovering in particular power law scaling.\nThese observables can be understood in a dual way as uncovering hidden internal\nstructure of the datasets themselves as a function of scale or depth. The\nentropic character of the proposed notion of complexity should allow to\ntransfer modes of analysis from neuroscience and statistical physics to the\ndomain of artificial neural networks. The introduced observables can be applied\nwithout any change to the analysis of biological neuronal systems.\n