A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks

The ability of artificial agents to increment their capabilities when\nconfronted with new data is an open challenge in artificial intelligence. The\nmain challenge faced in such cases is catastrophic forgetting, i.e., the\ntendency of neural networks to underfit past data when new ones are ingested. A\nfirst group of approaches tackles forgetting by increasing deep model capacity\nto accommodate new knowledge. A second type of approaches fix the deep model\nsize and introduce a mechanism whose objective is to ensure a good compromise\nbetween stability and plasticity of the model. While the first type of\nalgorithms were compared thoroughly, this is not the case for methods which\nexploit a fixed size model. Here, we focus on the latter, place them in a\ncommon conceptual and experimental framework and propose the following\ncontributions: (1) define six desirable properties of incremental learning\nalgorithms and analyze them according to these properties, (2) introduce a\nunified formalization of the class-incremental learning problem, (3) propose a\ncommon evaluation framework which is more thorough than existing ones in terms\nof number of datasets, size of datasets, size of bounded memory and number of\nincremental states, (4) investigate the usefulness of herding for past\nexemplars selection, (5) provide experimental evidence that it is possible to\nobtain competitive performance without the use of knowledge distillation to\ntackle catastrophic forgetting and (6) facilitate reproducibility by\nintegrating all tested methods in a common open-source repository. The main\nexperimental finding is that none of the existing algorithms achieves the best\nresults in all evaluated settings. Important differences arise notably if a\nbounded memory of past classes is allowed or not.\n