M$^2$-ViT: Accelerating Hybrid Vision Transformers with Two-Level Mixed Quantization

Although vision transformers (ViTs) have achieved significant success, their intensive computations and substantial memory overheads challenge their deployment on edge devices. To address this, efficient ViTs have emerged, typically featuring convolution-transformer hybrid architectures to enhance both accuracy and hardware efficiency. While prior work has explored quantization for efficient ViTs to marry the hardware efficiency of efficient hybrid ViT architectures and quantization, it focuses on uniform quantization and overlooks the potential advantages of mixed quantization. Meanwhile, although several works have studied mixed quantization for standard ViTs, they are not directly applicable to hybrid ViTs due to their distinct algorithmic and hardware characteristics. To bridge this gap, we present M2-ViT to accelerate convolution-transformer hybrid efficient ViTs with two-level mixed quantization (M2Q). Specifically, we introduce a hardware-friendly M2Q strategy, characterized by both mixed quantization precision and mixed quantization schemes [uniform and power-of-two (PoT)], to exploit the architectural properties of efficient ViTs. We further build a dedicated accelerator with heterogeneous computing engines to transform algorithmic benefits into real hardware improvements. The experimental results validate our effectiveness, showcasing an average of 80% energy-delay product (EDP) saving with comparable quantization accuracy compared to the prior work. Codes are available at https://github.com/lybbill/M2ViT.