Vision transformers have shown tremendous success in numerous computer vision applications; however, they have not been exploited for stress assessment using physiological signals such as electrocardiogram (ECG). In order to get the maximum benefit from the vision transformer for multilevel stress assessment, in this article, we transform the raw ECG data into 2-D spectrograms using short-time Fourier transform. These spectrograms are divided into patches for feeding to the transformer encoder. We also perform experiments with 1-D convolutional neural network (CNN) and ResNet-18 (CNN model). We perform leave-one-subject-out cross validation (LOSOCV) experiments on wearable stress and affect detection (WESAD) and Ryerson Multimedia Lab (RML) dataset. One of the biggest challenges of LOSOCV-based experiments is to tackle the problem of intersubject variability. In this research, we address the issue of intersubject variability and show our success using 2-D spectrograms and the attention mechanism of transformer. Experiments show that vision transformer handles the effect of intersubject variability much better than CNN-based models and beats all previous state-of-the-art methods by a considerable margin. Moreover, our method is end-to-end, does not require handcrafted features, and can learn robust representations. The proposed method achieved 71.01% and 76.7% accuracies with RML dataset and WESAD dataset, respectively, for three class classification and 88.3% for binary classification on WESAD.