Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks

Artificial neural networks are well-known to be susceptible to catastrophic\nforgetting when continually learning from sequences of tasks. Various continual\n(or "incremental") learning approaches have been proposed to avoid catastrophic\nforgetting, but they are typically adversary agnostic, i.e., they do not\nconsider the possibility of a malicious attack. In this effort, we explore the\nvulnerability of Elastic Weight Consolidation (EWC), a popular continual\nlearning algorithm for avoiding catastrophic forgetting. We show that an\nintelligent adversary can bypass the EWC's defenses, and instead cause gradual\nand deliberate forgetting by introducing small amounts of misinformation to the\nmodel during training. We demonstrate such an adversary's ability to assume\ncontrol of the model via injection of "backdoor" attack samples on both\npermuted and split benchmark variants of the MNIST dataset. Importantly, once\nthe model has learned the adversarial misinformation, the adversary can then\ncontrol the amount of forgetting of any task. Equivalently, the malicious actor\ncan create a "false memory" about any task by inserting carefully-designed\nbackdoor samples to any fraction of the test instances of that task. Perhaps\nmost damaging, we show this vulnerability to be very acute; neural network\nmemory can be easily compromised with the addition of backdoor samples into as\nlittle as 1% of the training data of even a single task.\n