The Role of Temporal Hierarchy in Spiking Neural Networks

Taking inspiration from the brain to design efficient computational systems remains challenging due to its complexity. This work investigates a key biological feature observed across mammals: a hierarchy of time scales in cortical areas. Experimental evidence shows that intrinsic neural dynamics slow down across the cortical hierarchy, forming a temporal hierarchy. We examine whether this property benefits artificial systems by introducing temporal hierarchy into spiking neural networks (SNNs), which inherently process information over time. We implement hierarchical time scales across neuronal, synaptic, and recurrent dynamics, and evaluate their effect under two settings: (1) as an inductive bias, and (2) as an emergent property through optimization. On temporal benchmarks such as multi-timescale-XOR and keyword spotting, hierarchical SNNs consistently outperform non-hierarchical ones, achieving 2%–6% higher accuracy and up to 5 × parameter reduction under iso-accuracy conditions. Moreover, when trained freely, temporal hierarchy emerges spontaneously through gradient descent. Finally, our theoretical analysis shows that hierarchical time constants enable processing of multi-frequency temporal signals with only log N layers—compared to N layers for non-hierarchical systems—highlighting hierarchy as a key organizational principle for efficient temporal computation.