Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models

Continual (or "incremental") learning approaches are employed when additional\nknowledge or tasks need to be learned from subsequent batches or from streaming\ndata. However these approaches are typically adversary agnostic, i.e., they do\nnot consider the possibility of a malicious attack. In our prior work, we\nexplored the vulnerabilities of Elastic Weight Consolidation (EWC) to the\nperceptible misinformation. We now explore the vulnerabilities of other\nregularization-based as well as generative replay-based continual learning\nalgorithms, and also extend the attack to imperceptible misinformation. We show\nthat an intelligent adversary can take advantage of a continual learning\nalgorithm's capabilities of retaining existing knowledge over time, and force\nit to learn and retain deliberately introduced misinformation. To demonstrate\nthis vulnerability, we inject backdoor attack samples into the training data.\nThese attack samples constitute the misinformation, allowing the attacker to\ncapture control of the model at test time. We evaluate the extent of this\nvulnerability on both rotated and split benchmark variants of the MNIST dataset\nunder two important domain and class incremental learning scenarios. We show\nthat the adversary can create a "false memory" about any task by inserting\ncarefully-designed backdoor samples to the test instances of that task thereby\ncontrolling the amount of forgetting of any task of its choosing. Perhaps most\nimportantly, we show this vulnerability to be very acute and damaging: the\nmodel memory can be easily compromised with the addition of backdoor samples\ninto as little as 1\\% of the training data, even when the misinformation is\nimperceptible to human eye.\n