How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Vision Transformers (ViT) have been shown to attain highly competitive\nperformance for a wide range of vision applications, such as image\nclassification, object detection and semantic image segmentation. In comparison\nto convolutional neural networks, the Vision Transformer's weaker inductive\nbias is generally found to cause an increased reliance on model regularization\nor data augmentation ("AugReg" for short) when training on smaller training\ndatasets. We conduct a systematic empirical study in order to better understand\nthe interplay between the amount of training data, AugReg, model size and\ncompute budget. As one result of this study we find that the combination of\nincreased compute and AugReg can yield models with the same performance as\nmodels trained on an order of magnitude more training data: we train ViT models\nof various sizes on the public ImageNet-21k dataset which either match or\noutperform their counterparts trained on the larger, but not publicly available\nJFT-300M dataset.\n