Thank you for the clarification and positive recommendation. Please find the comparisons of our work with your references [2-6] below (we have omitted [1] as it is a survey article).
Reference [2] (Lee et al., ICML 2021) is indeed closely related to our work, but we believe our study introduces two significant advancements. First, while their analysis of readout similarity and its comparison to input feature similarity was conducted numerically, we have derived analytical expressions for knowledge transfer and retention as functions of both feature and readout similarity. This approach uncovered non-trivial interactions between these two aspects of similarities. Secondly, their study focused on vanilla neural network training, whereas our work investigates the interaction between task similarity and widely used continual learning algorithms, such as activity gating and weight regularization, offering insights into their robustness against task similarity. Please note that we have provided a detailed comparison with reference [2] in the related work section (L88-L91) of the manuscript.
The reference [3] (Lin et al., ICML 2023) is also related to our work. This work analyzed continual learning in a linear regression setting, derived the generalization error and the total forgetting in the presence of arbitrary number of tasks. In terms of the impact of task similarity, this work found that low task similarity may reduce forgetting, as observed in other theoretical works with different model settings (Lee et al., ICML 2021; Evron et al., CoLT 2022). In contrast, our work introduces a latent structure that allows us to disambiguate these two aspects. Additionally, their study did not explore the effects of continual learning algorithms such as activity gating and weight regularization.
Reference [4] (Lin et al., ICLR 2022) presents a Trust-region gradient projection algorithm for continual learning. Unlike traditional approaches, this algorithm promotes forward knowledge transfer by projecting the gradient onto the weight space of previously learned, related tasks. Our work is motivated by a similar challenge—balancing the tradeoff between knowledge transfer and retention, a problem that remains not fully understood. In our study, we provide theoretical insights into how input and output similarity affect this tradeoff and how activity gating and weight regularization influence these interactions. We believe our findings will advance the development of algorithms in this area.
References [5] (Peng et al., ICML 2023) and [6] (Kim et al., NeurIPS 2022) explore model-agnostic theories of continual learning. Peng et al. [5] investigated the necessary and sufficient conditions for achieving continual learning without forgetting. They demonstrated that, in a linear regression framework, an ideal continual learner could be constructed with respect to training error. However, their work does not address the effects of task similarity or specific learning algorithms. Similarly, Kim et al. [6] analyzed model-agnostic bounds in continual learning, with a focus on detecting task boundaries in a class incremental setting. Notably, like in many studies, our approach assumes that the task boundaries are known to the model, which makes their theoretical bounds less directly applicable to our scenario.
We will discuss these works, particularly [3] and [4], in the related work section of the revised manuscript.