Several data augmentation methods deploy unlabeled-in-distribution (UID) data\nto bridge the gap between the training and inference of neural networks.\nHowever, these methods have clear limitations in terms of availability of UID\ndata and dependence of algorithms on pseudo-labels. Herein, we propose a data\naugmentation method to improve generalization in both adversarial and standard\nlearning by using out-of-distribution (OOD) data that are devoid of the\nabovementioned issues. We show how to improve generalization theoretically\nusing OOD data in each learning scenario and complement our theoretical\nanalysis with experiments on CIFAR-10, CIFAR-100, and a subset of ImageNet. The\nresults indicate that undesirable features are shared even among image data\nthat seem to have little correlation from a human point of view. We also\npresent the advantages of the proposed method through comparison with other\ndata augmentation methods, which can be used in the absence of UID data.\nFurthermore, we demonstrate that the proposed method can further improve the\nexisting state-of-the-art adversarial training.\n