Rebuttal by Authors
**Response to [W1].** Thanks for your question. While we acknowledge the limitations of unstructured pruning in the *Conclusion*, we also highlight significant industry progress in Appendix A, supported by notable evidence [1,2]. For instance, Nvidia GPUs support unstructured pruning through the NVIDIA Ampere Architecture and NVIDIA TensorRT.
The rest of the response demonstrates the advantages of FIARSE in terms of computation and communication overhead. We train ResNet-18 using CIFAR-10, which is partitioned into 100 clients using Dirichlet distribution with a parameter 0.3. In each communication round, we sample 10 clients, and these clients locally train the model for 4 epochs with a batch size of 20.
According to the numerical results presented in Table 1 and Figure 3 in the manuscript, all baselines can achieve a global accuracy of 65% or higher, and various submodels can realize a global accuracy of 60%. Therefore, the table below presents the number of communication rounds, overall computation, and communication costs when **the average global accuracy reaches 65% and all submodels should achieve an accuracy higher than 60%**:
|Method|Rounds|Total computation costs (TFLOPs)|Total communication costs (GB)|
|-|-|-|-|
|HeteroFL|560|4089.59|309.86|
|FedRolex|NA(640)|4673.82|354.13|
|ScaleFL|460|5144.41|274.57|
|FIARSE|**380**|**3620.36**|**210.08**|
Notes to the table: (i) we evaluate the result every 20 rounds, so the value of rounds can be divided by 20. (ii) Model 1/64 of FedRolex never achieves an accuracy above 60%. So we present its result of the 640th round, where FedRolex achieves an average global accuracy of 65% for the first time.
The table demonstrates the superiority of the proposed FIARSE: it requires the fewest communication rounds, and the least communication and computation overhead among the selected methods.
**References:**
[1] How Well Do Sparse ImageNet Models Transfer?, CVPR'22
[2] Sparse Fine-tuning for Inference Acceleration of Large Language Models, 2023
***
**Response to [W2].** Thanks for your suggestion. We conduct empirical studies to train a ResNet with CIFAR-10 using 100 clients, while only 10 clients are selected to train the model in each communication round. The tables below make a comparison between NeFL, FjORD, and the proposed FIARSE.
|Method|Model (1/64)|Model (1/16)|Model (1/4)|Model (1.0)|
|--|--|--|--|--|
|NeFL|65.88|72.08|76.52|79.44|
|FjORD|71.36|71.48|70.04|77.40|
|FIARSE|73.12|77.20|77.24|82.04|
Among these three methods, the proposed FIARSE significantly outperforms another two baselines.
***
**Response to [W3/Q2].** Thanks for your suggestion and for bringing interesting work. Due to the space limit, we moved our related work to Appendix A and discussed the similarities and differences compared to the previous works. For example, this work achieves model sparsification in federated learning, but different from the previous studies [1,2], we consider each client to possess heterogeneous computation capacities. We understand that paper organization is not a common practice, and we will insert a brief summarization of related work between *Introduction* and *Preliminary*.
IST [3] focuses on model parallelism in federated learning, where each node independently trains a submodel, and the server assembles all submodels into a global network. In contrast, our proposed FIARSE explores submodel extractions for different model sizes, which can be efficiently trained to achieve remarkable performance. These two works address different problems. From a theoretical perspective, IST analyzes the convergence while training with various submodels. Similar to the conclusion drawn from Theorem 2 of IST, the proposed FIARSE converges to a neighborhood of a stationary point. IST confirms that the bias cannot be eliminated, indicating that the model keeps fluctuating within a constant area. In contrast, our theoretical analysis demonstrates the possibility that FIARSE can minimize this neighborhood area to zero.
**References:**
[1] Sparse Random Networks for Communication-Efficient Federated Learning, ICLR'23
[2] DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training, ICML'22
[3] Towards a Better Theoretical Understanding of Independent Subnetwork Training, ICML'24
***
**Response to [Q1].** In our opinion, a static model never explores other parameters, while a dynamic model investigates new parts of the model. According to FjORD, the model architecture is fixed at the beginning of training, meaning the submodel never actively explores new parts of the model. Therefore, we consider FjORD to be a static strategy. We will clarify this classification in the Introduction.