Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning

We thank the reviewer for their feedback and positive comments about technical content. **Novelty of FOT** Our paper indeed uses a similar orthogonality idea as in GPM. However, the straightforward adaptation of GPM into federated learning is to make each client apply GPM locally. As shown in Table 1 (see GPM+FL), this does not work well in terms of average accuracy and forgetting. This is because GPM+FL only could extract local subspaces and the updates that are orthogonal to these local subspaces are not necessarily orthogonal to the global subspace after aggregation (See Figure 2). Besides, it adds additional computation cost and storage to the client. When trying to adopt that orthogonality idea to FL successfully, we need to solve 2 major problems which is our technical novelty in this paper: 1 - How to make global model updates orthogonal to the previous task layer subspace? 2 - More importantly, how to get the global principal subspace of layers in a privacy-preserving, communication, and computation cost-friendly way. In our work we propose 2 methods FedProject and GPSE, addressing problems 1 and 2 respectively. FedProject is a novel aggregation method ensuring that the global model converges to the low-loss regions for all considered tasks and GPSE is proposed to yield the global principal subspace. *Novelty of GPSE:* Utilization of randomized linear algebra is the crucial part in this algorithm while Secure Aggregation is used as an additional privacy layer. Without randomization, clients would need to directly send the activations to the server. Then, the server would need to concatenate the activations rather than addition/averaging before computing the principal subspace. Recall that the computation of the principal subspace (without the additional random transform in Eq (9) of the manuscript) is a function of the matrix $X$ with each column representing activations for different data points not an aggregated sum of the activations across points. Hence, direct secure aggregation would not be applicable. Furthermore, the server would have direct access to the activations (data points itself for the first layer) which violates privacy considerations in FL, and the overall communication overhead for computing the subspace would scale with the number of data points in the FL system. GPSE solves these issues by utilizing methods in randomized linear algebra in a novel way. By applying random Gaussian transforms to the activations, we ensure that the size of the communicated matrices only grows with the model size rather than the data amount. Also, the required operation to do at the server side becomes linear addition, enabling the use of Secure Aggregation. Lastly, the resulting matrix $A^\ell$ on the server side protects the principal column subspace information and does not leak information to the server about the clients, adding a layer of privacy to the system.

We thank the reviewer for their feedback and positive comments about technical content. We address your questions below point by point. **Weakness 1** *Comparison with Other Baselines:* LwF [1] is a centralized continual learning algorithm proposed in 2017. While determining baselines from centralized CL, our approach involved a thorough examination of both the SOTA methodologies and well-established earlier works. Notably, FedLwF is compared with FedCIL in [A] and shown to perform much worse. Given the inclusion of FedCIL in our baseline, and show that FOT has superior performance in comparison, we have chosen to exclude FedLwF from our analysis. While categorized under Continual Federated Learning, it's crucial to recognize that FedWeIT [2] addresses a fundamentally distinct problem. It is primarily interested in learning individual local tasks, not learning a global model but utilizing the information of other clients. It employs federated learning not to train a global model but to enhance the continual learning process for individual local tasks by leveraging information from other clients. Due to the distinct objective of this paper in comparison to ours, we haven’t included it in our baseline. We mention TARGET [3] in our related work section (Section 2 and Appendix B). Unfortunately, due to its publication date being in close proximity to the submission deadline, we were unable to incorporate it into our baselines. Following the reviewer’s advice, we did an initial comparison of TARGET with FOT on mini-Imagenet dataset. At the end of each task, the generator in TARGET is trained on the server side in a data-free way. Then using that generative model, the server generates synthetic samples (750 samples per task in our experiments) and sends them to the clients. Clients perform knowledge distillation between the current model and the stored old-task model with the synthetic samples to prevent forgetting. | | IID | | nonIID | | |---|---|---|---|---| | | ACC(%) | FGT(%) | ACC(%) | FGT(%) | | FedAvg | 50.43 | 33.08 | 41.00 | 32.92 | | TARGET | 58.45 | 25.12 | 53.67 | 26.31 | |FedCIL | 55.98 | 28.90 | 51.12 | 25.66 | | FOT | 69.07 | 0.19 | 62.06 | 0.17 | Similar to FedCIL, TARGET achieves a better forgetting and accuracy compared to vanilla FedAvg, but is outperformed by our FOT approach. We would like to highlight that these only represent initial comparisons with TARGET and we plan to extend to a more thorough comparison on other datasets in the revised manuscript in Appendix G.7. Given your suggestions, we have also extended our related work discussion in the manuscript to highlight [1] and [2]. **Weakness 2 & Question 2** In the pursuit of fairness, we empirically identified the optimal number of rounds for the base case, namely FedAvg, and employed this determined round number consistently across all other methods. *Convergence of FOT:* Thank you for asking this question. FOT does not exhibit slower convergence; However, it is crucial to highlight that in subsequent tasks, convergence is directed toward the optimal point within a restricted convex set through a subspace projection. While this projection doesn't slow down the convergence speed, it does lead to a slight decrease in that task’s accuracy due to regularization by the target convex set. In the updated version of the manuscript (Appendix G.5), we discuss the convergence behaviors of FOT and share the accuracy-round plots. As we can see from these plots, the convergence behavior of FOT is similar to FedAVG. While FOT may exhibit lower accuracy than FedAvg in a specific task, FOT maintains this accuracy in later tasks. On the other hand, the accuracy of other methods tends to decline below FOT's performance as new tasks are introduced. That’s why, at the end of training a sequence of different tasks, the average accuracy of FOT is higher and the forgetting is less than other methods. **Question 1** In the server, after aggregating all local $A_{t,i}^\ell$, the server gets global $A_{t}^\ell$ (see equation 10). This new matrix is equal to the multiplication of global input matrix $X^\ell$ and random Gaussian matrix $G$. Therefore, dimension n from the theorem corresponds to the second dimension of $X^\ell$ which is equal to the total number of samples distributed among clients. The value of n remains constant for the task whether we simulate a task with 100 clients or 50 clients. For the specific quantity of data for each task and dataset, refer to Table 8 in the manuscript, specifically examining the ' # training/task' rows. [A] Daiqing Qi, Handong Zhao, and Sheng Li. Better generative replay for continual federated learning, ICLR, 2023.

We thank the reviewer for their feedback and positive comments about technical content. We address your questions below point by point. **Communication Overhead:** We would like to emphasize that the additional communication cost incurred by FOT is negligible. First, it is important to note that this additional cost is associated only with the GPSE step, and this step consists of just one round, conducted once at the end of each task. In this round, the server transmits $\\{ O_\ell \\}^L_{\ell=1}$ to the clients, and the clients send only $\\{ A_\ell \\}^L_{\ell=1}$ to the server. Importantly, the communication cost of GPSE remains independent of the amount of data and is scaled only with the model size. Furthermore, while the communication complexity aligns with the model size, the actual size is observed to be smaller than the model itself as shown in Table 5. As a numerical example from our experiments, if we train Alex-net on one Cifar100 task for 1000 federated learning rounds by communicating the model, then the FOT cost and FedAVG incur exactly the same cost (2840 MB in total) over the 1000 model training rounds because FOT only communicates the model between clients and the server (as in FedAvg). At the end of 1000 rounds, GPSE uses a communication load less than or equal to only a single round which is much smaller than rounds used for model aggregation because GPSE communicates $\\{ O_\ell \\}^L_{\ell=1}$ and $\\{ A_\ell \\}^L_{\ell=1}$ and their total size (0.29 mb - See table 5) is less than model size (1.42 mb - See table 5 ). As a result, the additional communication cost is less than 0.1% of the total training cost for a single task. That’s why we claim the additional communication overhead is minimal. In contrast, recent federated continual learning work - FedCIL [1] has a huge overhead in terms of communication. FedCIL communicates a generative model with the original classifier in each training round (compared to FOT which only does additional communication only once per task). Given the generative model is bigger (1.60 mb) than the classifier, it increases the communication cost by more than 100%. We provide the total communication cost (in MB) of one client at the end of each Cifar100 task in the below table. We give a more extensive description of the additional communication cost in Appendix G.4 in the updated version of the paper. | Task | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |---|---|---|---|---|---|---|---|---|---|---| | FedAvg | 2840.0 | 5680.0 | 8520.0 | 11360.0 | 14200.0 | 17040.0 | 19880.0 | 22720.0 | 25560.0 | 28400.0 | | FOT | 2840.0 | 5680.3 | 8520.6 | 11360.9 | 14201.2 | 17041.4 | 19881.7 | 22722.0 | 25562.3 | 28402.6 | | FedCIL | 2840.0 | 8880.0 | 14920.0 | 20960.0 | 27000.0 | 33040.0 | 39080.0 | 45120.0 | 51160.0 | 57200.0 | **Convergence Analysis:** The paper presents an approach for enabling continual learning in a federated learning setting. We rely on an orthogonalization approach, however, we take careful consideration in allowing this without violating the privacy requirements and the convergence guarantees in FL. Our proposed approach allows us to achieve protection against forgetting using projection into a convex set which allows us to rely on convergence guarantees in the literature for ProximalSGD. We would argue that this simplicity is a strength of the proposed method as it allows us to guarantee theoretical convergence with privacy guarantees and little communication overhead in addition to observed empirical benefits. We would also like to point out that the central theoretical challenge in the considered problem and our proposed approach is enabling the learning of principal subspaces and the application of orthogonal projection in a distributed privacy-preserving setting. [1] Daiqing Qi, Handong Zhao, and Sheng Li. Better generative replay for continual federated learning, ICLR, 2023.