Continual Learning in the Teacher-Student Setup: Impact of Task Similarity

Continual learning-the ability to learn many tasks in sequence-is critical\nfor artificial learning systems. Yet standard training methods for deep\nnetworks often suffer from catastrophic forgetting, where learning new tasks\nerases knowledge of earlier tasks. While catastrophic forgetting labels the\nproblem, the theoretical reasons for interference between tasks remain unclear.\nHere, we attempt to narrow this gap between theory and practice by studying\ncontinual learning in the teacher-student setup. We extend previous analytical\nwork on two-layer networks in the teacher-student setup to multiple teachers.\nUsing each teacher to represent a different task, we investigate how the\nrelationship between teachers affects the amount of forgetting and transfer\nexhibited by the student when the task switches. In line with recent work, we\nfind that when tasks depend on similar features, intermediate task similarity\nleads to greatest forgetting. However, feature similarity is only one way in\nwhich tasks may be related. The teacher-student approach allows us to\ndisentangle task similarity at the level of readouts (hidden-to-output weights)\nand features (input-to-hidden weights). We find a complex interplay between\nboth types of similarity, initial transfer/forgetting rates, maximum\ntransfer/forgetting, and long-term transfer/forgetting. Together, these results\nhelp illuminate the diverse factors contributing to catastrophic forgetting.\n