Trimming Down Large Spiking Vision Transformers via Heterogeneous Quantization Search

Spiking neural networks (SNNs) are amenable to deployment on edge devices and neuromorphic hardware due to their lower dissipation. Recently, SNN-based transformers have garnered significant interest, incorporating attention mechanisms akin to their counterparts in artificial neural networks (ANNs) while demonstrating excellent performance. However, deploying large-scale spiking transformer models on resourceconstrained edge devices such as mobile devices still poses significant challenges resulting from the high computational demands of large uncompressed high-precision models. In this work, we introduce a novel heterogeneous quantization method to compress spiking transformers through layer-wise quantization. Our approach optimizes the quantization of each layer using one of two distinct quantization schemes, that is, uniform or power-of-two quantization, with mixed precision. Our heterogeneous quantization demonstrates the feasibility of maintaining high performance for spiking transformers while utilizing an average effective precision of 3.14-3.67 bits with less than a $\mathbf{1 \%}$ accuracy drop on neuromorphic DVS Gesture and CIFAR10-DVS datasets. It achieves $8.71 \times-10.19 \times$ model compression rate for standard floating-point spiking transformers. Furthermore, the proposed approach achieves substantial energy reductions of $5.69 \times, 8.72 \times$, and $10.2 \times$ on the N-Caltech101, DVS-Gesture, and CIFAR10DVS datasets, respectively, while maintaining high accuracy levels of $85.3 \%, 97.57 \%$, and 80.4 %.

Paper

Similar papers

© 2026 NYSGPT2525 LLC