When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

Contrastive learning (CL) can learn generalizable feature representations and\nachieve the state-of-the-art performance of downstream tasks by finetuning a\nlinear classifier on top of it. However, as adversarial robustness becomes\nvital in image classification, it remains unclear whether or not CL is able to\npreserve robustness to downstream tasks. The main challenge is that in the\nself-supervised pretraining + supervised finetuning paradigm, adversarial\nrobustness is easily forgotten due to a learning task mismatch from pretraining\nto finetuning. We call such a challenge 'cross-task robustness\ntransferability'. To address the above problem, in this paper we revisit and\nadvance CL principles through the lens of robustness enhancement. We show that\n(1) the design of contrastive views matters: High-frequency components of\nimages are beneficial to improving model robustness; (2) Augmenting CL with\npseudo-supervision stimulus (e.g., resorting to feature clustering) helps\npreserve robustness without forgetting. Equipped with our new designs, we\npropose AdvCL, a novel adversarial contrastive pretraining framework. We show\nthat AdvCL is able to enhance cross-task robustness transferability without\nloss of model accuracy and finetuning efficiency. With a thorough experimental\nstudy, we demonstrate that AdvCL outperforms the state-of-the-art\nself-supervised robust learning methods across multiple datasets (CIFAR-10,\nCIFAR-100, and STL-10) and finetuning schemes (linear evaluation and full model\nfinetuning).\n