Network Dynamics-Based Framework for Understanding Deep Neural Networks

Advancements in artificial intelligence demand a deeper understanding of the underlying mechanisms of deep learning. This study, based on the theory of nonlinear dynamical systems, constructs a theoretical framework for analyzing deep networks at a microscopic level. The framework consists of two main components: (1) a classification of information transformation modes. We categorize the ways in which a network transforms information layer by layer into two basic types: order-preserving transformations and non-order-preserving transformations. Order-preserving transformations can be achieved by individual neurons, while non-order-preserving transformations require the cooperation of multiple neurons. (2) A dynamical method for performance evaluation. By introducing the concept of attraction basins in the sample and weight spaces, we characterize the network’s performance from the perspective of dynamical system stability. The attraction basins in the sample space reflect the generalization ability of the network, while the attraction basins in the weight space represent structural robustness. Different information transformation modes lead to different distributions of weight vectors in space. By identifying these structures, we can estimate the relative contributions of each transformation mode in different layers, thus revealing distinct learning “phases”. Based on this, we explain a key theoretical source of deep network performance advantages, provide a mechanistic explanation for the phenomenon of “grokking”, establish the theoretical basis for the existence of an optimal depth, and offer theoretical guidance for selecting hyperparameters such as learning rate and batch size.