Dark Experience for General Continual Learning: a Strong, Simple Baseline

Continual Learning has inspired a plethora of approaches and evaluation\nsettings; however, the majority of them overlooks the properties of a practical\nscenario, where the data stream cannot be shaped as a sequence of tasks and\noffline training is not viable. We work towards General Continual Learning\n(GCL), where task boundaries blur and the domain and class distributions shift\neither gradually or suddenly. We address it through mixing rehearsal with\nknowledge distillation and regularization; our simple baseline, Dark Experience\nReplay, matches the network's logits sampled throughout the optimization\ntrajectory, thus promoting consistency with its past. By conducting an\nextensive analysis on both standard benchmarks and a novel GCL evaluation\nsetting (MNIST-360), we show that such a seemingly simple baseline outperforms\nconsolidated approaches and leverages limited resources. We further explore the\ngeneralization capabilities of our objective, showing its regularization being\nbeneficial beyond mere performance.\n