SPACE: Structured Compression and Sharing of Representational Space for Continual Learning

Humans learn adaptively and efficiently throughout their lives. However,\nincrementally learning tasks causes artificial neural networks to overwrite\nrelevant information learned about older tasks, resulting in 'Catastrophic\nForgetting'. Efforts to overcome this phenomenon often utilize resources\npoorly, for instance, by growing the network architecture or needing to save\nparametric importance scores, or violate data privacy between tasks. To tackle\nthis, we propose SPACE, an algorithm that enables a network to learn\ncontinually and efficiently by partitioning the learnt space into a Core space,\nthat serves as the condensed knowledge base over previously learned tasks, and\na Residual space, which is akin to a scratch space for learning the current\ntask. After learning each task, the Residual is analyzed for redundancy, both\nwithin itself and with the learnt Core space. A minimal number of extra\ndimensions required to explain the current task are added to the Core space and\nthe remaining Residual is freed up for learning the next task. We evaluate our\nalgorithm on P-MNIST, CIFAR and a sequence of 8 different datasets, and achieve\ncomparable accuracy to the state-of-the-art methods while overcoming\ncatastrophic forgetting. Additionally, our algorithm is well suited for\npractical use. The partitioning algorithm analyzes all layers in one shot,\nensuring scalability to deeper networks. Moreover, the analysis of dimensions\ntranslates to filter-level sparsity, and the structured nature of the resulting\narchitecture gives us up to 5x improvement in energy efficiency during task\ninference over the current state-of-the-art.\n