The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry

The Fisher information matrix (FIM) is fundamental to understanding the\ntrainability of deep neural nets (DNN), since it describes the parameter\nspace's local metric. We investigate the spectral distribution of the\nconditional FIM, which is the FIM given a single sample, by focusing on\nfully-connected networks achieving dynamical isometry. Then, while dynamical\nisometry is known to keep specific backpropagated signals independent of the\ndepth, we find that the parameter space's local metric linearly depends on the\ndepth even under the dynamical isometry. More precisely, we reveal that the\nconditional FIM's spectrum concentrates around the maximum and the value grows\nlinearly as the depth increases. To examine the spectrum, considering random\ninitialization and the wide limit, we construct an algebraic methodology based\non the free probability theory. As a byproduct, we provide an analysis of the\nsolvable spectral distribution in two-hidden-layer cases. Lastly, experimental\nresults verify that the appropriate learning rate for the online training of\nDNNs is in inverse proportional to depth, which is determined by the\nconditional FIM's spectrum.\n