HSViT: Horizontally Scalable Vision Transformer

While the Vision Transformer (ViT) architecture gains prominence in computer vision and finds growing applications in edge computing, its lack of strong inductive biases regarding shift, scale, and rotational invariance necessitates pre-training on large-scale datasets. Moreover, the increasing depth and parameter counts in ViT models present significant challenges for training, particularly in edge environments where computational resources are constrained. To mitigate these challenges, this paper introduces a novel Horizontally Scalable Vision Transformer (HSViT) architecture. Specifically, a novel image-level feature embedding approach is introduced that incorporates convolutional layers prior to the Transformer blocks. This design helps preserve inductive biases, allowing the model to potentially eliminate the need for pre-training while achieving strong performance on small datasets. Furthermore, a novel horizontally scalable architecture is designed, facilitating collaborative model training and inference across multiple edge devices. The experimental results show that, without pre-training, HSViT achieves up to 10% higher top-1 accuracy than state-of-the-art methods on several small datasets, while improving the top-1 accuracy of existing CNN backbones by up to 3.1% on ImageNet-1k. The code is available at https://github.com/xuchenhao001/HSViT.