Summary
This paper argues, and attempts to empirically demonstrate, that cumulatively averaging the parameters of models trained on individual tasks is a promising approach for continual learning. First, using task-incremental versions of Split MNIST and Split CIFAR-100, the paper shows that by averaging the parameters of task-specific models trained on each task individually, only a moderate amount of performance is lost relative to using the single-task models directly. Motivated by this, the paper then combines cumulative parameter averaging with self-supervised learning to engineer a multi-module approach for continual learning with strong empirical performance on both task- and class-incremental benchmarks.
Strengths
I think it is an intriguing claim of the paper that a promising approach for continual learning is to cumulatively average the parameters of models trained on individual tasks. As the paper discusses, there is some other recent work (e.g., CLS-ER and DualNet) that builds on this approach, but I think that a principled demonstration of the benefits of the parameter averaging approach is still missing.
I think it is also a strength that the authors demonstrate that their proposed approach can work with different self-supervised learning techniques.
Weaknesses
Unfortunately, I do not think that this paper provides a clear or principled demonstration of the benefit of parameter averaging.
Regarding the experiments of Table 1, I am not convinced that these results are indicative that (cumulative) parameter averaging is a promising method. On Split MNIST, STL-me drops to less than 97%, and even after fine-tuning the classifier the performance is still below 99%. This is on two-way classification tasks of MNIST digits. What would be the performance of a linear classifier in this situation? That is, what would be obtained if you do “classifier fine-tunining” directly on the raw pixels? Another baseline that I think would be good to include in Table 1 is a “small version of STL”, in which there is a separate model for each task (i.e., like in STL), but each task-specific model is relatively small, such that the total parameter count of the combined models is the same as STL-me. Would STL-me outperform this baseline?
Perhaps the authors would argue that the strong performance of their engineered multi-module method in Tables 2 and 3 demonstrates the effectiveness of cumulative parameter averaging for continual learning, but I do not think that is the case. For example, an aspect present in DLCPA but not in the compared baselines (at least not in most of them) is self-supervised learning. Indeed, from Table 4 it seems that if the self-supervised learning aspect is left out, the superior performance of DLPCA disappears.
An intriguing property of DLCPA is that it is able to distinguish between classes from different tasks (e.g. Appendix E2). This is surprising to me, at least at first, because my understanding is that DLCPA is actually never trained to distinguish between classes from different tasks. Could the authors attempt to explain why it is nevertheless able to do so?
Although code is provided, I could not find instructions on how to run or reproduce the experiments reported in the paper.
Questions
I would like to encourage the authors to include a clear / principled demonstration of the benefit of cumulative parameter averaging for continual learning.
I would be happy to actively engage in the discussion period.
Rating
3: reject, not good enough
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.