AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs

Weight decay is a standard regularization technique for training large language models (LLMs), but using a uniform decay rate across layers ignores architectural diversity and module-specific spectral differences. We propose AlphaDecay, a simple approach that adaptively assigns weight decay strengths at the module level. Guided by Heavy-Tailed Self-Regularization (HT-SR) theory, we analyze the empirical spectral density (ESD) of weight correlation matrices to quantify heavy-tailedness. Modules with more heavy-tailed ESDs—indicative of stronger feature learning—are assigned weaker decay, while modules with lighter-tailed spectra receive stronger decay. By tailoring decay to module-wise spectral properties, AlphaDecay better balances regularization and improves training outcomes. Extensive pre-training experiments across model sizes from 60M to 1B show that AlphaDecay yields lower perplexity and better generalization than uniform decay and existing adaptive decay baselines. Code is available at https://github.com/hed-ucas/AlphaDecay .