Pre-trained Language Models (PLMs) have achieved great success in various\nNatural Language Processing (NLP) tasks under the pre-training and fine-tuning\nparadigm. With large quantities of parameters, PLMs are computation-intensive\nand resource-hungry. Hence, model pruning has been introduced to compress\nlarge-scale PLMs. However, most prior approaches only consider task-specific\nknowledge towards downstream tasks, but ignore the essential task-agnostic\nknowledge during pruning, which may cause catastrophic forgetting problem and\nlead to poor generalization ability. To maintain both task-agnostic and\ntask-specific knowledge in our pruned model, we propose ContrAstive Pruning\n(CAP) under the paradigm of pre-training and fine-tuning. It is designed as a\ngeneral framework, compatible with both structured and unstructured pruning.\nUnified in contrastive learning, CAP enables the pruned model to learn from the\npre-trained model for task-agnostic knowledge, and fine-tuned model for\ntask-specific knowledge. Besides, to better retain the performance of the\npruned model, the snapshots (i.e., the intermediate models at each pruning\niteration) also serve as effective supervisions for pruning. Our extensive\nexperiments show that adopting CAP consistently yields significant\nimprovements, especially in extremely high sparsity scenarios. With only 3%\nmodel parameters reserved (i.e., 97% sparsity), CAP successfully achieves 99.2%\nand 96.3% of the original BERT performance in QQP and MNLI tasks. In addition,\nour probing experiments demonstrate that the model pruned by CAP tends to\nachieve better generalization ability.\n