Task-agnostic forms of data augmentation have proven widely effective in\ncomputer vision, even on pretrained models. In NLP similar results are reported\nmost commonly for low data regimes, non-pretrained models, or situationally for\npretrained models. In this paper we ask how effective these techniques really\nare when applied to pretrained transformers. Using two popular varieties of\ntask-agnostic data augmentation (not tailored to any particular task), Easy\nData Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrichet al.,\n2015), we conduct a systematic examination of their effects across 5\nclassification tasks, 6 datasets, and 3 variants of modern pretrained\ntransformers, including BERT, XLNet, and RoBERTa. We observe a negative result,\nfinding that techniques which previously reported strong improvements for\nnon-pretrained models fail to consistently improve performance for pretrained\ntransformers, even when training data is limited. We hope this empirical\nanalysis helps inform practitioners where data augmentation techniques may\nconfer improvements.\n