Privacy-Aware Lifelong Learning

Response to Reviewer iYfp (part 1/2)

>_**(Q1)** I noticed that the maximum number of tasks in this manuscript is 40 in S-TinyImage and the maximum unlearning requests $N_u$ is 9 for the S-TinyImageNet 20-split setting. I wonder if the proposed method is scalable for larger numbers of tasks and more unlearning requests, e.g., on the Omniglot-Rotation dataset which has 100 tasks in total._ >_**(W2)** The scalability of the proposed method needs to be further demonstrated with more experiments on other datasets._ We now performed additional experiments with S-TinyImageNet (100 tasks) using ViT-T/8, also for increased number of unlearning requests $N_u$. Results are provided in the table below, for the metrics $\mathcal{A}_l / \mathcal{F}_u$ in each experimental setting. We could demonstrate that there is no particular limitation of implementing PALL in larger scale settings. In the extreme experimental setting, for a user request sequence with 100 task learning and $N_u=50$ task unlearning requests, PALL ($\alpha=0.01$) achieves $\mathcal{A}_l=81.36\%$ by using at most 54K parameters per task. We included these results as **Table A6** in the new Appendix A.3.3. We also partially discussed these results under Section 5.3 of the main text. | S-TinyImageNet ($T=100$) | $N_u=3$ | $N_u=5$ | $N_u=10$ | $N_u=20$ | $N_u=50$ | |:-------------------------:|:----------:|:----------:|:----------:|:----------:|:----------:| | **PALL** ($\alpha=0.01$) | 85.89/0.53 | 85.45/0.41 | 85.18/0.27 | 84.21/0.21 | 81.36/0.17 | | **PALL** ($\alpha=0.025$) | 86.11/0.43 | 85.84/0.31 | 85.52/0.22 | 84.29/0.19 | 79.77/0.20 | | **PALL** ($\alpha=0.05$) | 85.80/0.35 | 85.60/0.29 | 85.04/0.23 | 84.00/0.20 | 77.96/0.24 | >_**(Q2)** Can the proposed PALL be adapted to the class-incremental setting? If so, what changes should be introduced to make it compatible with the scenario if we don't know the task identity?_ >_**(W3)** The current version only considered the task-incremental setting where the task id is known during the inference. I wonder the applicability on other settings, such as class-incremental setting or domain incremental setting._ We now discuss a future direction for PALL to be applicable in class- and domain-incremental learning settings in the last paragraph of the Discussion section of our paper as follows: *''In this work, we focus on task-incremental learning settings, where the task ID is available to the learner. Our approach is not yet readily applicable to a class- or domain-incremental scenario where all previously learned tasks' label spaces are unified, since PALL disentangles the choice of the task-specific subnetwork based on the task IDs to ensure exact unlearning. To be applicable in these settings, PALL can be extended with a privacy-aware auxiliary algorithm to first identify the task, and subsequently utilize task-specific subnetwork via gating (Aljundi et al., 2017; Von Oswald et al., 2019).''* Aljundi et al., ''Expert Gate: Lifelong Learning with a Network of Experts'', CVPR 2017. Von Oswald et al., ''Continual learning with hypernetworks'', ICLR 2020.

Response to Reviewer iYfp (part 2/2)

>_**(Q3)** About the training efficiency. In section 5, the authors only discussed the memory that needs to store the additional parameters along the learning dynamic. The comparisons of the training time among different methods were missed. The proposed method is mainly based on WSN [1]. However, according to my experience in training WSN, I found that WSN could be a little slow since it requires the parameter-level gradient masking within each backpropagation step and I believe this operation also exists in your proposed PALL. Thus, could the authors provide a comparison w.r.t. the training efficiency among different methods?_ We now added training time comparisons and discussions in the new subsection Appendix A.3.4 with **Table A8**. This table is also provided below, where we indicate the "task learning time per epoch / total task unlearning time" for each dataset. For total task unlearning times which were less than $10^{-4}$ seconds, we only write "$-$". Indeed, during task learning, PALL incurs an additional training time overhead to optimize task-specific subnetworks with knowledge transfer. This is also observed in WSN (e.g., for S-TinyImageNet, WSN: 2.96 sec/epoch, PALL: 3.13 sec/epoch). Both algorithms have a higher training time than the other baselines, since there is a parameter-level gradient masking within each gradient update step. This difference is larger in ResNets with convolutional layer masking, as opposed to the ViT-T/8 where only dense layers throughout the model are being masked. During unlearning, the total retraining time of PALL has a negligible difference to other methods. | | S-CIFAR10 ($T=5$) | S-CIFAR100 ($T=10$) | S-TinyImageNet ($T=20$) | |-------------|---------------------------|---------------------------|---------------------------| | Sequential | 3.23 sec/epoch / $-$ | 2.91 sec/epoch / $-$ | 2.71 sec/epoch / $-$ | | EWC | 6.42 sec/epoch / $-$ | 5.59 sec/epoch / $-$ | 3.59 sec/epoch / $-$ | | LwF | 4.87 sec/epoch / $-$ | 4.23 sec/epoch / $-$ | 3.37 sec/epoch / $-$ | | LSF | 9.02 sec/epoch / 0.51 sec | 8.21 sec/epoch / 0.90 sec | 5.28 sec/epoch / 1.84 sec | | GEM | 23.8 sec/epoch / 0.93 sec | 12.4 sec/epoch / 1.64 sec | 3.19 sec/epoch / 2.10 sec | | ER | 6.65 sec/epoch / 0.94 sec | 5.70 sec/epoch / 1.73 sec | 2.95 sec/epoch / 2.13 sec | | DER++ | 9.09 sec/epoch / 1.38 sec | 8.44 sec/epoch / 2.44 sec | 3.11 sec/epoch / 3.06 sec | | PackNet | 3.40 sec/epoch / $-$ | 3.07 sec/epoch / $-$ | 2.65 sec/epoch / $-$ | | WSN | 32.6 sec/epoch / $-$ | 30.2 sec/epoch / $-$ | 2.96 sec/epoch / $-$ | | **PALL ($\alpha=1/T$)** | 33.1 sec/epoch / 1.09 sec | 30.8 sec/epoch / 1.88 sec | 3.13 sec/epoch / 2.59 sec | | Independent | 3.20 sec/epoch / $-$ | 2.89 sec/epoch / $-$ | 2.66 sec/epoch / $-$ | >_**(W1)** Most parts of this manuscript came from a previous study WSN [1]. Although the experience replay (in section 3.3) was tailored for knowledge recovery after an unlearning step, the new message conveyed from this manuscript is limited._ We introduce a novel problem configuration (i.e., task-incremental lifelong learning with exact unlearning), and present a new solution in this setting. We adapt importance score optimization based sparse structure learning algorithms to solve this problem (Ramanujan et al. 2020), which were also explored in traditional lifelong learning by (Kang et al., 2022). We acknowledged both of these works in our submission, and clearly discussed the distinction and innovation of our work in the Discussion section as follows: *''Our method is partially based on architecture-based lifelong learning methods that allow selective knowledge transfer (Kang et al., 2022; Ramanujan et al. 2020), which we innovatively extended into a hybrid learning strategy that is also equipped with a rehearsal-based lifelong learning method (Buzzega et al., 2020). This enables our algorithm to handle exact task unlearning requests in the presence of knowledge transfer, which was not addressed to date. Additionally, we also performed critical modifications to the existing subnetwork optimization methods, such as reinitializing the scores and unused weights after each learning request. These were required to satisfy overall exact unlearning guarantees, and resulted in effective learning and unlearning abilities simultaneously.’'*

Comments after Author's Rebuttal

I appreciate the effort from the authors in addressing my concerns. I have some further questions: - Thanks for providing the time consumption statistics. It seems that PALL indeed suffers from the same efficiency issue as WSN. I wonder if this issue deteriorates for the longer the task length. In other words, it seems that most of the time consumption comes from the binary mask consolidation and gradient masking process. I am not sure if this problem is more severe at the later stages within a long task sequence (e.g., 100 tasks). It would be better if the authors could plot a figure to show the time consumption along the task number increases. - Thanks for providing the experimental results under the 100-task setting. I have two further questions regarding the experiments: (1) how about the performance compared to other baselines? (2) how about the parameter size increment along the task number increases during this process? It would be better if this could be quantitatively depicted (maybe in the figure), and that is my concrete concern about the scalability compared to simply discussing the performances. I understand the additional experiments can take some time. The authors can answer my questions according to your time and effort.

Further Comments from Reviewer iYfp

Thanks for providing additional explanations to my questions. After carefully reading the author's rebuttal, my concerns (i.e., scalability, training efficiency, and model size increment) have been addressed. Furthermore, I also read the discussion between the authors and other reviewers, and it seems that the questions from my colleagues were also addressed during the peaceful discussions. Thus, I decided to increase my rating to 8 and continue to support the acceptance of this manuscript.

Response to Reviewer KbAJ (part 1/2)

>_**(Q1)** How do you guarantee that a random sparse subset would not have significant overlap with a previous subset?_ We do not constrain the algorithm in terms of the overlap between task-specific subnetwork masks. Our update rule for the importance scores $s_t$ in Eq. (5) is applied to all weights without masking, and any pretrained/frozen weight from previous tasks can be assigned high importance scores if they are found useful to minimize the loss at the current task. If any task $t$ and task $t+1$ to be learned are very similar, the algorithm will ideally optimize a subnetwork mask for task $t+1$ that highly overlaps with the mask for task $t$. If this is not desired, one can disable knowledge transfer and enforce the subnetworks to be independent, which would require masking of the importance score updates, similar to the parameter updates, in Eq. (5). >_**(Q2)** Seems like memory buffers are an integral part of this lifelong learning algorithm. Do the authors have an idea how it could be done without them?_ The memory buffer is only required to recover performance after unlearning, if knowledge transfer is performed during learning. Currently, no method can jointly satisfy: (1) no catastrophic forgetting, (2) allowing forward knowledge transfer, (3) exact unlearning capabilities, (4) memory-efficiency in terms of the number of model weights. PALL is the first solution at this intersection, and using a memory buffer was a reasonable remedy to balance the desired (2) *knowledge transfer* and (3) *exact unlearning* capabilities, simultaneously. The alternative that excludes memory buffers would be to disable ''(2) *knowledge transfer*'' across tasks, and optimize sparse _independent_ subnetworks (as presented in our Section 5.3), such that any unlearning request could be performed by only resetting the corresponding subnetwork weights. We now clarified this distinction in our Section 5.3 revisions. >_**(W1)** The algorithm is not necessarily always memory-efficient since the memory complexity still grows linearly in the number of tasks. In other words, the improvement is a constant, albeit a very small one given that it's stored in 1-bit format. [...]_ Our memory-efficiency claims are only addressed in comparison to training *Independent* models for each task. This was the only existing solution to the problem of *lifelong learning with exact unlearning*. Now, to achieve this, one can use PALL in a more memory efficient way as the number of tasks grow, even in the case of lower weight quantization. This is because of the required total memory usage of training *Independent* models. We re-stated this again in our revisions. >_**(W2)** The tasks are given explicitly. While this might be the case sometimes in practice (e.g., tasks are user data), it still a disadvantage in terms of privacy (e.g., the data of some task are known to belong to some user)._ This is not a weakness of our method, but instead the nature of the task-incremental lifelong learning problem. We focus on traditional task-incremental lifelong learning scenarios, and in this case the task ID is always provided to the model. Particularly in this context, we are proposing a solution that can ensure exact task unlearning. >_**(W3)** Eq. 6. is not novel. It is introduced as a generalized case of DER++ (Buzzega et al., 2020) [...] I suggest the author rephrase this section [...]_ We now rephrased our narrative to be more explicit. Indeed, we did not need to re-introduce a novel experience replay objective since existing methods performed well for such a sparse and fast retraining phase. Eq. (6) is generic to apply any experience replay objective by changing the weighting factor. Our formulation of PALL uses the state-of-the-art approach with DER++ ($\beta=0.5$) in the main results, although our further analyses in Table A4 of the Appendix show that DER++ is _not required_ for PALL to perform well. >_**(W4)** I would further argue that Eq. 6 is not an elegant remedy for such a privacy-aware algorithm since it requires access to some data from previous tasks._ We propose a previously non-existing solution to the problem of ensuring exact unlearning, when lifelong learning *with knowledge transfer* is performed. Therefore, Eq. (6) or a similar experience replay mechanism, is only necessary to recover performance degradations caused by unlearning, *if knowledge transfer is desired*. If forward knowledge transfer is not of concern, a memory replay mechanism or Eq. (6) is also not necessary. Thus, our solution is a generalized formulation that can satisfy all pillars of lifelong learning and exact unlearning.

Response to Reviewer KbAJ (part 2/2)

>_**(W5)** The authors mentions ''strong privacy considerations, where exact unlearning is possible.'' The authors should mention that the unlearning is found to be exact empirically because it could be the case that the proposed algorithm would not be able to unlearn exactly on real-world data (e.g., foundational models might not be able to unlearn exactly even when using PALL)._ The algorithm does not yield exact unlearning only empirically, but guarantees this by design of the algorithm. Our method resets the subnetwork that observes any task-specific data, and that the new model becomes identical to one that has never observed any data from the task that is unlearned. Considering the reviewer's example, if a foundational model was trained in a task-incremental setting with our proposed subnetwork structures, then PALL could also exactly unlearn a specific task dataset. >_**(W6)** There are no investigations done on language data. This is especially important since the right to be forgotten is very applicable in this case._ This is not a limitation of our method. Our solution is agnostic to the machine learning task and architecture, and we only focused our experiments to the vision domain due to our interest. We acknowledge the potential in extending our work to language processing tasks, and leave these explorations for future work. We added this in the Discussion section. >_**(W7)** Dynamic sparsity does not seem to be very important. Perhaps using static sparsity and increasing model size can match the same performance with the same memory cost._ In a true lifelong learning scenario where the number of tasks are unknown and can be very large, dynamically exploring both the sparse connectivity structure and the selective weight sharing can be useful, as opposed to using independent models with static sparsity without knowledge transfer. Exploitation of forward knowledge transfer across related tasks is one of the main pillars in lifelong learning algorithms, and dynamic sparsity helps us to achieve this. >_**(W8)** Some experiments are not directly related to PALL. For example, Table 2. [...]_ We moved this table to Appendix A.3.1 as Table A1, and edited the text around it by referring to Appendix A.3.1. >_**(W-Mi1)** Eq. 1, uses inconsistent notation for $\mathcal{L}$ in terms of arguments. [...]_ We corrected the notation in Eq. (1) by fixing the $\theta^0$. The learning algorithm $\mathcal{L}$ is indeed stochastic, and we actually perform lifelong learning sequentially via $\theta^t\sim\mathcal{L}(\theta^{t-1},\mathcal{D}^t)$. Due to this notation we defined early on in Section 3.1, we keep $\theta^t\sim$ in Eq. (1). >_**(W-Mi2)** Eq. 2., $\mathbf{U}$ and $\mathbf{L}$ not introduced. I believe they are just flag variable?_ Yes, we now defined these terms explicitly. >_**(W-Mi3)** Eq. 3., the first constraint is an expression without equality/inequality (min ERM). Perhaps "s.t." should be replaced with an inequality and "min" should be replaced with "argmin"._ The current expression is also correct. The first constraint of this expression indicates the overarching lifelong learning goal, that the learning algorithm $\mathcal{L}$ updates the parameters such that the *test set* error is minimized on all previously observed tasks. >_**(W-Mi4)** I believe experiments and experimental results should traditionally be in the same section._ We renamed Section 3 as ''Experimental Setup'' to avoid the confusion. >_**(W-Mi5)** $\alpha$ is not clearly introduced, only briefly after Eq. 4. I suggest to reintroduce $\alpha$ somewhere in the experimental results section for clarity as it is used extensively there._ We now reintroduced $\alpha$ again in the caption of Table 1, while discussing the results. >_**(W-Mi6)** The choice $\alpha=1/T$ provides a fixed memory budget, but this might make comparisons between CIFAR-10 and CIFAR-100 trickier since they have a different number of tasks._ Correct. We tried not to make comparisons of results between these datasets. This is only an intuitive choice to simplify further hyperparameter decisions. We allocate a subnetwork with $1/T$ portion of the complete architecture for each task. >_**(W-Mi7)** The experiments are being compared with methods that are not designed for the problem in this work. While this is not necessarily bad, it makes PALL more likely to outperform them since PALL is designed for this problem, so its better peformance, particularly in unlearning, is expected and not surprising._ This is not a weakness of PALL, but the nature of the state-of-the-art. We tackle a problem without any existing solution. We needed to create a set of benchmark comparisons from the lifelong learning literature to simply demonstrate why existing lifelong learning methods would not work well in this problem. The only baseline method that we truly compare against is training *Independent* models for each task, which can ensure exact unlearning.

Response to Reviewer ggGQ (part 1/3)

>_**(Q1)** For the ''independent subnetworks without knowledge transfer'' ablation, could you explain specifically in what way(s) ''Dynamic Sparse Ind.'' differs from PALL? [...] The authors claim that PALL additionally supports knowledge transfer but it’s not obvious to me what the mechanism is for that, given that each new task only modifies (a chosen subset of) previously-unused weights?_ The only difference in PALL is the presence of knowledge transfer, by allowing to *selectively* re-use pretrained/frozen weights from previous task-specific subnetworks. This is achieved by optimizing the importance scores $s_t$ over *all* connections via Eq. (5), *without masking*. This means that if any pretrained weight from previous tasks is found useful to solve the current task $t$, its importance score can get higher and this pretrained weight can be selected within $m_t$, resulting in selective re-using of weights. Therefore, in PALL, the dynamically optimized sparse subnetworks can be partially overlapping as a result of the optimization process. In ''Dynamic Sparse Ind.'', this knowledge transfer via weight-sharing is disabled, thus the dynamically optimized sparse subnetworks are kept *independent*. This is implemented by keeping the importance scores corresponding to any pretrained/frozen weight as 0, such that they are not optimized for selection and cannot be re-used in other tasks. We restated the main difference in these methods and revised Section 5.3. >_**(Q2)** What is the worst-case sequence of interleaved learning and unlearning requests, w.r.t each metric considered here? Are there cases where the proposed method is no better than naive baselines? It would be great to discuss this in the paper._ We retrieved these results and added as **Table A7** in the new Appendix A.3.5 subsection. Results are presented below, in the form of $\mathcal{A}_l^{\text{min}}\uparrow/\mathcal{F}_u^{\text{max}}\downarrow$, where $\mathcal{A}_l^{\text{min}}$ denotes the worst-case task learning performance across randomized repetitions of different user request sequences, and $\mathcal{F}_u^{\text{max}}$ denotes the maximum negative impact of any unlearning request. We do not discuss any other evaluation metrics, since PALL is always superior without catastrophic forgetting ($\mathcal{F}_l=0$) and exact unlearning ($\mathcal{A}_u$ is irrelevant). We mainly observed that the worst-case evaluations are consistent with the averaged metrics presented in the Table 1 of the main text, and discussed these in Appendix A.3.5. | | S-CIFAR10 | S-CIFAR100 | S-TinyImageNet | |-------------------------|:----------:|:-----------:|:--------------:| | Sequential | 36.78/0.0 | 21.03/0.0 | 13.93/0.0 | | EWC | 28.60/0.0 | 51.79/0.0 | 52.18/0.0 | | LwF | 83.43/0.0 | 48.83/0.0 | 41.04/0.0 | | LSF | 81.50/3.13 | 51.53/8.87 | 35.42/11.60 | | GEM | 79.85/5.71 | 52.10/10.28 | 39.52/12.94 | | ER | 80.63/5.23 | 49.87/15.07 | 39.79/12.12 | | DER++ | 84.45/2.22 | 61.57/4.58 | 44.46/7.11 | | PackNet | 89.85/0.0 | 72.99/0.0 | 58.35/0.0 | | WSN | 86.75/0.0 | 69.93/0.0 | 61.34/0.0 | | **PALL ($\alpha=1/T$)** | 89.95/1.94 | 69.83/2.17 | 59.91/4.42 | | Independent | 90.85/0.0 | 69.46/0.0 | 60.29/0.0 | >_**(W1)** It seems that the ''independent subnetworks without knowledge transfer'' ablation actually performs very similarly to the proposed approach [...] Are there perhaps other sequences of tasks / datasets where the ''knowledge transfer'' would be more ''needed'' [...] ?_ We revised these ablation results with additional experiments on **S-TinyImageNet (100 tasks)**. Results are presented below as $\mathcal{A}_l\uparrow/\mathcal{F}_u\downarrow$, which are also presented in our revised **Section 5.3**. In these new experiments, we test the utility of knowledge transfer across a larger number of tasks, where the model is split into 100 subnetworks. As a result, we were able to observe a larger relative increase in performance as opposed to using independent subnetworks. Here, our main claim is to show that PALL is favorable in lifelong learning and unlearning scenarios where the maximum number of task can be even higher, and unknown a priori. | | $T=40$ | $T=100$ | |:-------------------------:|------------|------------| | Static Sparse (Ind.) | 70.30/0.0 | 80.70/0.0 | | Dynamic Sparse (Ind.) | 71.19/0.0 | 83.75/0.0 | | **PALL ($\alpha=0.01$)** | 71.00/0.56 | 85.89/0.53 | | **PALL ($\alpha=0.025$)** | 72.07/0.36 | 86.11/0.43 | | **PALL ($\alpha=0.05$)** | 72.03/0.32 | 85.80/0.35 | | Independent | 71.76/0.0 | 86.80/0.0 |

Response to Reviewer ggGQ (part 2/3)

>_**(W2-A)** I don’t understand the difference between $m_t$ and $\overline{m_t}$._ $\overline{m}_t$ is the binary mask that indicates whether a parameter was updated based on gradients computed with samples from $\mathcal{D}_t$. This $\overline{m}_t$ mask is not necessarily identical to the task-specific subnetwork mask $m_t$, because $m_t$ can additionally indicate weights trained on previous tasks due to the use of knowledge transfer via weight-sharing across tasks. >_**(W2-B)** When describing the metric $A_t$, it’s unclear whether the final model checkpoint (after addressing the whole sequence) is used to compute accuracy on all tasks, or is a separate checkpoint? [...]_ We compute $\mathcal{A}_l$ and $\mathcal{A}_u$ by using the final checkpoint at the end of the lifelong learning and unlearning sequence. These metrics are defined by using $a_{r,t}$ in Eq. (7), where $r$ is the length of the whole request sequence, and in the text we define: ''$a_{i,t}$ denotes the test set accuracy after request $i$ was completed''. >_**(W2-C)** ''Regularization-based methods [...] were indifferent to task unlearning instructions, since the original methods are not adapted to unlearning'' – how were these applied to unlearning tasks, then?_ >_**(W3)** Insufficient description (in the main paper) for how various other methods were adapted from the standard lifelong setting to the privacy-aware lifelong setting. [...]_ We clarified these in our revisions. Due to space limitations, we could only include further details of how each method was adapted under ''Unlearning Implementations'' subsection of Appendix A.2. We revised Section 4.2 of the main text to make our pointers to the Appendix more explicit. For EWC, LwF and Sequential training, upon receiving a task unlearning request, we did not perform any specific parameter update step (i.e., they were indifferent to the task unlearning instruction). In Sequential, we simply ignore the unlearning request. For EWC and LwF, we delete the previously stored model weights/checkpoint from the task to be unlearned. For EWC, we also discard the previously computed and stored Fisher information matrix from the task to be unlearned. We performed *inexact unlearning* adaptations on lifelong learning algorithms, since it was a seamless integration. Existing approximate unlearning algorithms mostly require access to previously observed data, thus it is harder to adapt these inexact methods to lifelong learning. Nevertheless, these baselines were not critical for the main outcome of our work since our focus was to achieve exact unlearning, which is a stricter notion of privacy. >_**(W4)** It is unclear how one should interpret the $A_u$ metric. [...]_ We included this metric only to compare the approximate ''unlearning'' of the baseline methods, towards achieving poor classification on unlearned tasks. Indeed, this does not guarantee privacy. We do not make any strong claims based on $\mathcal{A}_u$, since PALL can achieve exact unlearning regardless of $\mathcal{A}_u$. We now explicitly state this in our revisions under Section 4.2: *''[...] it is important to note that a lower $\mathcal{A}_u$ does not necessarily correspond to an exact unlearning guarantee. We include $\mathcal{A}_u$ only to evaluate inexact unlearning baselines through a weak measure. We leave detailed investigation of inexact unlearning methods with better metrics, e.g., via empirical privacy auditing (Steinke et al., 2024), for future work.''* >_**(W5)** Missing an important baseline: training sequentially with Differential Privacy (DP). [...]_ >_**(W9)** A limitation of this framework is that “user privacy” can only be addressed (via exact unlearning) if a user’s data is cleanly separated in terms of “tasks”. Other scenarios, where all users participated in all “tasks”, would make this framework unable to support users to request their data to be deleted. It would be great to discuss this._ We considered differential privacy (DP), but excluded it from the current submission to be concise. DP baselines would help maintaining deletion guarantees *for each data sample of each task* as an extreme case with stricter privacy. We perform exact task unlearning altogether. However, related to the other comment **(W9)**, we could potentially utilize differentially private optimization while learning each task, to preserve privacy of individual data points within tasks. We now included a brief limitation discussion on DP and selective data unlearning, at the end of our revised paper, in the context of potential future work: *''Our work also does not yet consider selective data unlearning, but instead performs complete task unlearning. To achieve stricter privacy with deletion guarantees for each data sample, our approach can be combined with differentially private optimization methods (Lai et al., 2022), in future work.''*

Response to Reviewer ggGQ (part 3/3)

>_**(W6)** Missing baselines from the unlearning literature. A plethora of approximate unlearning methods have been proposed (see e.g. [Golatkar et at.; Kurmanji et al.; Fan et al.]) [...]. It feels like these would be readily combinable with the “Sequential” approach, for instance._ These baselines are not applicable to the *Sequential* approach, since they require access to previously observed ''retain'' or ''forget'' set samples in their objectives. An important factor to consider here is the lack of access to previous data in lifelong learning. One *can* combine Sequential learning with another inexact unlearning algorithm that does not require such data. However, this would only be another *inexact* baseline. We prefer to keep our main focus on *exact* unlearning (i.e., adaptation of SISA as Independent model training for each task). >_**(W7)** [...] it would be very useful to build a comparison between the proposed method (and other exact solutions) with inexact unlearning. While the authors have considered some inexact methods, there aren’t proper metrics considered to empirically estimate the privacy of such methods. [...] I fully understand that adding empirical privacy auditing is a substantial amount of additional work and potentially difficult to do all of this within a single paper [...]._ We thank the reviewer for this comment and suggestion for future work. We now mention this missing piece under Section 4.2, where we define our evaluation metrics. We include $\mathcal{A}_u$ only to evaluate inexact unlearning baselines through a weak measure. We leave the detailed investigation of inexact methods with better metrics, e.g., via empirical privacy auditing, for future work. >_**(W8)** Another baseline can be training a model sequentially (without subnetworks), and when unlearning is needed, simply retrain the entire model from scratch (using the data in the buffers). [...] One may additionally consider an ``oracle'' version of this, that assumes access to all past data as another reference point [...]_ We implemented this on S-CIFAR, annotated as ''ER (Retraining)'', in the new **Table A5** provided below. Since the suggested setting requires a memory buffer, we perform experience replay (ER) based task-incremental learning as well. For unlearning, we retrain the entire model from scratch using the memory buffer. We exclude the ''oracle'' setting with access to all past data, since it would conflict with our storyline from a traditional continual learning perspective. ''ER (Retraining)'' results showed that the performance was highly restricted by the memory buffer following the first unlearning request. Although retraining the model yields exact unlearning, re-learning previous tasks from scratch using ER did not scale well even to the slightly more complex task of S-CIFAR100. | | | | | | | | | | | | | |-------------------------|-------------------------|---------------------------|---------------------------|---------------------------|---|---|-------------------------|-------------------------|---------------------------|---------------------------|---------------------------| | **S-CIFAR10 ($T=5$)** | $\mathcal{A}_l\uparrow$ | $\mathcal{A}_u\downarrow$ | $\mathcal{F}_l\downarrow$ | $\mathcal{F}_u\downarrow$ | | | **S-CIFAR100 ($T=10$)** | $\mathcal{A}_l\uparrow$ | $\mathcal{A}_u\downarrow$ | $\mathcal{F}_l\downarrow$ | $\mathcal{F}_u\downarrow$ | | Independent | 95.19 | **Exact** | **0.0** | 0.0 | | | Independent | 73.22 | **Exact** | **0.0** | 0.0 | | **PALL ($\alpha=1/T$)** | 94.34 | **Exact** | **0.0** | 0.60 | | | **PALL ($\alpha=1/T$)** | 72.35 | **Exact** | **0.0** | 0.40 | | ER (Retraining) | 78.80 | **Exact** | 2.05 | 7.53 | | | ER (Retraining) | 34.50 | **Exact** | 4.07 | 20.40 | | ER | 87.88 | 58.48 | 3.45 | 1.61 | | | ER | 57.63 | 42.69 | 4.67 | 7.91 |

Response to Reviewer QDPb (part 1/2)

>_**(Q1)** The cited references for Regularization-based methods mainly cover foundational works, but the discussion could benefit from including more recent studies that reflect the latest advancements in this area._ >_**(W5)** The cited references of Regularization-based methods are mostly foundational works, but some are relatively dated, potentially overlooking recent improvements or newer approaches in the field._ Our work focuses on a completely novel problem setting, which was not explored via regularization-based continual learning methods. We are not aware of any particular regularization-based lifelong learning work that is specifically related to our scope in the context of exact machine unlearning. Perhaps the reviewer could elaborate more on this comment on which particular works could be of interest? >_**(Q2)** The experience settings of other baselines are different from those in this paper. The Baseline Methods section should describe more detailed setups._ >_**(W6)** The details of the Baseline you compared are not clearly described, e.g., how you tailor these baseline methods to fit the proposed problem setting?_ All implementation details of the baseline methods are discussed in Appendix A.2 under subsection ''Unlearning Implementations''. We now revised Section 4.2 of the main text to make our pointers to the Appendix more explicit. _**Unlearning Implementations:** For Sequential, EWC and LwF, we do not perform any changes to the model parameters for task unlearning. For EWC and LwF, we only discard the previously stored model weights (which are used for continual learning in these methods), corresponding to the unlearned task. For EWC, we also discard the previously stored Fisher information matrix corresponding to the unlearned task. We perform algorithm-specific unlearning updates for LSF, gradient episodic memory (GEM), experience replay (ER), dark experience replay (DER++), when task unlearning is instructed. For task unlearning with GEM, ER and DER++, we implement an experience replay finetuning objective on the remaining tasks' episodic memories, and also predict uniform distributions using the unlearned task's episodic memories to accelerate forgetting. Task unlearning with LSF exploits its unique notion of manipulating the input space by superposing distinct auxiliary mnemonic codes on task-specific data. For unlearning, the mnemonic code of the unlearned task is directly provided to the network to be predicted incorrectly via a uniform label distribution, and the remaining task codes are directly provided to the network to be predicted accurately._ >_**(Q3)** Including SOTA models would strengthen the study’s impact by showcasing its performance relative to the best-performing approaches in the field._ >_**(W7)** In the experimental section, the results should be compared with the latest state-of-the-art (SOTA) models in Continual Learning. A more detailed comparison with existing SOTA methods would provide a clearer context for evaluating the proposed approach and strengthen the overall analysis._ Since our work focuses on a novel lifelong learning problem that involves exact unlearning guarantees, the state-of-the-art (SOTA) in this setting corresponds to training *Independent* models for each task. This was already included in all our analyses as an upper performance baseline as the best-performing approach in the proposed problem of task-incremental lifelong learning with exact unlearning. Additionally, we implemented several continual learning methods with inexact unlearning approximations, as weaker baselines as well. >_**(Q4)** It would be helpful to clarify the initialization and optimization processes more explicitly and outline each configuration's practical implications. Adding these clarifications would aid in understanding the differences and rationale behind these two configurations, improving the overall readability and transparency of this section._ We have included all initialization and optimization hyperparameter specifications for each method explicitly in Appendix A.2 - ''Training Configurations and Implementations'', under subsections ''Model Training'' and ''Baseline Methods''.

Response to Reviewer QDPb (part 2/2)

>_**(W1a)** This paper presents a new problem setting, but this problem lack significant innovation compared with existing problems. What is the most different points that distinguish this problem from others? [...]_ >_**(W2)** The methods presented for the problem setting proposed in this paper appear to be well-established and lack significant innovation. The authors should either highlight any novel contributions or improvements to existing methods or explore more advanced techniques that could offer better solutions to the unique aspects of the problem._ In this paper, both the introduced problem formulation and the proposed solution are novel. There are no well-established methods that can tackle the problem of efficient lifelong learning with exact unlearning guarantees. We adapted existing lifelong learning methods in the context of the new problem, and demonstrated how these methods do not generalize to a setting with exact unlearning guarantees. >_**(W1b)** [...] It seems the objective of this problem in section 3.2 also holds for existing problem such as domain incremental learning and unlearning. Is there any challenges specific to this problem so that we must formulate it as a new problem?_ >_**(W4)** The paper should clarify why the task-incremental setting was chosen instead of class- or domain-incremental learning. Providing a rationale for this choice would help to better understand the appropriateness of the selected setting for the problem at hand._ We are not aware of existing works that studies ''domain incremental learning with exact unlearning''. Our work studies the problem of task-incremental learning with *exact* unlearning, which was not previously explored. Our formulation can, however, naturally extend to domain-incremental learning with *exact* unlearning as well. We have discussed a future direction for PALL to be applicable in class- or domain-incremental learning settings in our manuscript (see last paragraph of the Discussion section). Indeed, PALL is not readily applicable to a class- or domain-incremental scenario, since model disentanglement with task-specific subnetworks based on task IDs are necessary to ensure exact unlearning. To be applicable in such settings, one can use an auxiliary algorithm to first identify the context/task, and subsequently utilize task-specific subnetwork components via gating as previously suggested by various lifelong learning methods [1-2]. Aljundi et al., ''Expert Gate: Lifelong Learning with a Network of Experts'', CVPR 2017. Oswald et al., ''Continual learning with hypernetworks'' ICLR 2020. >_**(W3)** This paper introduces an interesting approach, but it is unclear how the proposed method has been specifically tailored to address the new problem setting. The authors should provide a more detailed explanation of how their approach was designed to handle the new settings._ This comment is not very specific, and we hope that the reviewer can elaborate which parts are not clear. Currently, the complete Section 3 describes how PALL is tailored to the problem of achieving exact unlearning in the context of lifelong learning without catastrophic forgetting and allowing forward knowledge transfer, with memory-efficiency. >_**(W8)** In the section on Comparisons to Independent Subnetworks without Knowledge Transfer, the descriptions of the two configurations—static sparsity and dynamic sparsity—are not sufficiently clear. Although static sparsity is introduced as a random initialization of T independent sparse subnetworks, and dynamic sparsity as an optimization of sparse connectivity via score optimization, the explanation lacks details on how each configuration functions within the broader experimental setup._ We define ''independent static sparsity'' as random initialization of $T$ independent sparse subnetworks with $1/T$ connectivity within a model, where the connectivity structure is kept frozen during training. We define ''independent dynamic sparsity'' to be the same as PALL, without the capability to perform knowledge transfer across tasks such that subnetworks are not overlapping. These models then process the user request sequence similar to the previous *Independent* baseline, following the detailed optimization procedures outlined under Appendix A.2 ''Training Configurations and Implementations''. We revised Section 5.3 to clarify our descriptions better.