Summary
The authors address the problem of online class incremental learning (OCIL), where new tasks arrive periodically in a data stream and the trainer seeks to learn these new tasks without catastrophic forgetting of past performance. The paper presents two modes of OCIL; replay-based methods and exemplar-free methods. Replay-based methods offer strong performance but require storing some amount of replay data from the stream to include in incremental training. Exemplar-free methods lift this limitation, but have not thus far achieved comparable performance to replay-based methods. The authors method (forward-only online analytic learning, or F-OAL) uses projections of blocks from a frozen, pre-trained ViT encoder as input to a trainable linear classifier, and then perform recursive least-squares updates on the linear classifier to avoid the catastrophic forgetting that backpropagation would cause in the same setting. The authors extensively compare their method by accuracy, training time, and memory usage to other recently-SOTA methods for OCIL, both replay-based and exemplar-free. The comparisons show that F-OAL significantly improves over existing exemplar-free methods and achieves comparable performance with replay-based methods, while being significantly more efficient in terms of both training time and memory usage. Lastly, the authors ablate their solution and find that keeping the ViT encoder frozen and using their analytically-learned classifier (instead of learning it through backpropogation) are both key ingredients in F-OAL's accuracy.
Strengths
The paper is generally quite strong. The F-OAL method is natural and intuitive; overall, it seems like a significant improvement to the Pareto frontier of accurate and efficient OCIL. The comparisons to past SOTA in both replay-based and exemplar-free methods are extensive and compelling. The authors' exposition is clear and logical, and the experiments are largely informative and useful for the reader. The F-OAL method seems pragmatic, and a natural baseline to which all future work in this area should be compared.
Weaknesses
This paper (and past papers in this vein) suggest that exemplar-free methods are "good for data privacy", but there is very little justification for this claim. I understand the basic premise as this: if replay-based methods force you to store some subset of the data stream, which is worse for users' data privacy than exemplar-free methods that don't require such storage. While it's worth noting as a design consideration/feature, I do not agree with this framing as relevant for "data privacy". There is no legitimate security model of privacy in the literature that would recognize this as "higher privacy". Once the model provider has seen/processed the data by running it through the model, any legitimate security model would view this as data that's been made public. Methods that enhance security (e.g. SMPC, HE, TEEs) and methods that provably reduce statistical privacy leakage (i.e. differential privacy) are orthogonal; these methods can be used interchangeably with both replay-based and exemplar-free methods! In a realistic setting, I can see how exemplar-free methods might help assuage compliance concerns or company-specific rules, but I know of no data privacy regulation or cryptographic threat model where this would be a relevant factor. I would suggest that the authors attempt to correct this misconception in the literature by simply stating the feature in terms of its utility: data need not be stored for replay in production. The reader can implicitly understand that this can have several benefits depending on the circumstances of the deployment.
Otherwise, the only criticism I'd have for the paper is its ablation study. While it's a useful sanity check to see their result of ablating AC and FCC, the results with Frozen are obvious and unnecessary. The ViT-B model was developed for ImageNet-sized datasets, of course it will overfit CIFAR-100! I think there are more useful ablations that could be performed (more on that below).
Questions
It seems clear to me that the authors method of fusing the ViT blocks $B_i(.)$ with random linear projection is valuable, but also not entirely necessary. The goal of this approach appears two-fold; (1) capture information from different levels of abstraction in the representation that their analytic classifier uses, and (2) be able to control the dimensionality of that representation, which will surely need to be tune-able at training time for different datasets (e.g. to avoid overfitting). The ablation in Appendix B reassures the reader that (2) is necessary, but none of the ablations suggest that (1) is necessary. A simple ablation that could've helped would be to compare their block-averaging + smoothed projection approach with a simpler method that applies the smoothed projection to the last ViT block. I'd be curious to know why the authors chose this particular approach. The paper itself states that this feature fusion was implemented to "further enhance the representativeness of the features", but is there any work or experiment they can point to that suggests this?
In any case, this seems to be the only weak point of the paper, and a clarification or improvement would be nice.
Limitations
The only concern I would have is that the paper (and its cited works) abuse the notion of "data privacy", which I have previously argued against in this review. Works that claim to "improve data privacy" without treatment of the staggering amount of literature that has gone into defining and proving what is and is not "private" in an information-theoretic, statistical, or engineering-focused sense are likely to muddy the waters for those fields. The paper's setting and solution are already worth publishing; in my opinion, the unsubstantiated data privacy claim is hurting more than it's helping.