Deep Learning models are highly susceptible to adversarial manipulations that\ncan lead to catastrophic consequences. One of the most effective methods to\ndefend against such disturbances is adversarial training but at the cost of\ngeneralization of unseen attacks and transferability across models. In this\npaper, we propose a robust defense against adversarial attacks, which is model\nagnostic and generalizable to unseen adversaries. Initially, with a baseline\nmodel, we extract the latent representations for each class and adaptively\ncluster the latent representations that share a semantic similarity. We obtain\nthe distributions for the clustered latent representations and from their\noriginating images, we learn semantic reconstruction dictionaries (SRD). We\nadversarially train a new model constraining the latent space representation to\nminimize the distance between the adversarial latent representation and the\ntrue cluster distribution. To purify the image, we decompose the input into low\nand high-frequency components. The high-frequency component is reconstructed\nbased on the most adequate SRD from the clean dataset. In order to evaluate the\nmost adequate SRD, we rely on the distance between robust latent\nrepresentations and semantic cluster distributions. The output is a purified\nimage with no perturbation. Image purification on CIFAR-10 and ImageNet-10\nusing our proposed method improved the accuracy by more than 10% compared to\nstate-of-the-art results.\n