Additional Experiments on Local Epochs=15 and Rebuttal
Dear reviewer g4w5, thanks for your quick reply. We finish the experiments on the local epochs=15 as shown in the following.
### About the question of "It is plausible that the performance of baseline methods could be further enhanced by extending the number of local epochs (perhaps the author could perform experiments with SCAFFOLD alone for efficiency). If this hypothesis holds true, particularly for the outcomes showcased in Table 1, the claimed superiority of FedInit may not be as compelling."
**This hypothesis is not correct.** If we extend the local epochs to a very large number, the local client will be optimized well and almost achieve the local minimal. This is fatal for federated learning. All local models will overfit the local dataset and are far away from the global optimal. This is the "client-drift" problem proposed in [1], which could be formulated as $\frac{1}{m}\sum_i x_i^\star \neq x^\star$. This is also one of the major issues in FL. Therefore, the local epochs can not be extended too much. As shown in the following table, under large local epochs, SCAFFOLD will also be affected.
| Method | E=1 | E=5 | E=10 | E=15 |
| :----: | :----:| :----:| :----: | :----: |
| SCAFFOLD | 71.51% | 75.06%| **75.74%** | 75.25%|
Actually, many previous works [1,2,3] have been devoted to preventing local clients from fully converging. [1] uses the variance reduction to correct the local gradient as the global gradient estimation to avoid overfitting. [2] changes the local objective from $F_i(w_i)$ to $F_i(w_i)+\lambda\Vert w_i-w\Vert^2$ to force the local model to be close to the last global state. [3] uses the ADMM method to change the local objective. However, even with these advanced methods, the local interval still cannot be set too large. Training with large intervals will eventually degrade performance and fall into the average of the local minimal $\frac{1}{m}\sum_i x_i^\star$ instead of the global minimal $x^\star$. Our proposed FedInit also could work as a light plug-in and help them to further improve the performance without conflicts.
[1] SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
[2] Federated Optimization in Heterogeneous Networks
[3] Federated Learning Based on Dynamic Regularization
### About the fair comparison:
As we mentioned above, due to the "client-drift" problem in FL, increasing the local interval does not always improve the results of the algorithm. So about your question about SCAFFOLD outperforming FedInit on local epochs=10, that is not fair. Here we make a comprehensive table comparing the performance of SCAFFOLD and FedInit and their training costs.
Under the setup in our paper, 1 epochs=10 iterations. After 500 communication rounds, the total iterations is 5000 if local epoch=1.
| | E=1 | E=5 | E=10 | E=15 |
| :----: | :----:| :----:| :----: | :----: |
| SCAFFOLD | 71.51% | 75.06%| **75.74%** | 75.25% |
| FedInit | 73.11% | **75.95%** | 75.48% | 73.92% |
| Total Iterations | 5000 | 25000| 50000 | 75000 |
As stated above, **to achieve the best results, FedInit requires fewer iterations and less training time**. Furthermore, SCAFFOLD requires to communicate 2 vectors per communication round and FedInit only requires to communicate 1 vector. Another main issue in FL is the communication bottleneck in the practical training. Therefore, the proposed FedInit is better for its fewer communication costs and running wall-clock time costs. SCAFFOLD applies a variance reduction technique, which does not conflict with our relaxed initialization. What we would like to claim is that SCAFFOLD adopts the variance reduction and double communication costs to improve performance, while FedInit can achieve comparable performance with fewer training costs.