In nature, symmetry governs regularities, while symmetry breaking brings\ntexture. In artificial neural networks, symmetry has been a central design\nprinciple to efficiently capture regularities in the world, but the role of\nsymmetry breaking is not well understood. Here, we develop a theoretical\nframework to study the "geometry of learning dynamics" in neural networks, and\nreveal a key mechanism of explicit symmetry breaking behind the efficiency and\nstability of modern neural networks. To build this understanding, we model the\ndiscrete learning dynamics of gradient descent using a continuous-time\nLagrangian formulation, in which the learning rule corresponds to the kinetic\nenergy and the loss function corresponds to the potential energy. Then, we\nidentify "kinetic symmetry breaking" (KSB), the condition when the kinetic\nenergy explicitly breaks the symmetry of the potential function. We generalize\nNoether's theorem known in physics to take into account KSB and derive the\nresulting motion of the Noether charge: "Noether's Learning Dynamics" (NLD).\nFinally, we apply NLD to neural networks with normalization layers and reveal\nhow KSB introduces a mechanism of "implicit adaptive optimization",\nestablishing an analogy between learning dynamics induced by normalization\nlayers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have\nestablished a theoretical foundation to discover geometric design principles\nfor the learning dynamics of neural networks.\n