Recent advances in the field of artificial intelligence have been made\npossible by deep neural networks. In applications where data are scarce,\ntransfer learning and data augmentation techniques are commonly used to improve\nthe generalization of deep learning models. However, fine-tuning a transfer\nmodel with data augmentation in the raw input space has a high computational\ncost to run the full network for every augmented input. This is particularly\ncritical when large models are implemented on embedded devices with limited\ncomputational and energy resources. In this work, we propose a method that\nreplaces the augmentation in the raw input space with an approximate one that\nacts purely in the embedding space. Our experimental results show that the\nproposed method drastically reduces the computation, while the accuracy of\nmodels is negligibly compromised.\n