Thank the Reviewer for the Constructive Comments
Dear reviewer,
We do sincerely appreciate your constructive comments and questions. With all the respect, we provide our replies and hope they would address your concerns appropriately.
`>>> Cross-device setting V.S. cross-silo setting`
`>>> Answer:` Thanks for the insightful comments. The current model design is indeed not optimally suitable to the large-scale cross-device federated learning. Your invaluable input has enabled us to accurately define the scope of our contributions within the realm of cross-silo federated learning. In our final version, we will provide a comprehensive clarification of the cross-silo federated learning setting and expound upon the limitations of our work concerning the cross-device scenario. Your feedback greatly assists in enhancing the clarity and context of our research.
`>>> Computational overhead`
`>>> Answer:` Thanks for pointing out. In our current experimental setting, we set B as 10 when the number of clients is 100, aligning with the setting of existing work [1,2]. We can increase the value of B in our experiments, but it could lead to an increase of computational overhead on the server side, which is a limitation of our current model design. In our future work, we will explore new strategies to tackle this limitation and make the technique of heterogeneous model reassembly more efficient. Your insights serve as valuable guidance in refining our model.
In response to your query about the parameter M, we would like to clarify that it is indeed a learned value in our model. In our experiments on the SVHN dataset with 100 clients and 0.1 active client ratio, we observed a dynamic range for M across multiple runs, spanning from 7 to 12. Regarding the size of stitching parts, we appreciate your interest in this aspect. As exemplified in Figure 4 of our original paper, we presented two generated candidate models. These models featured 4 stitching parts at time t and 5 stitching parts at time t', each corresponding to different depths. For each stitching part between two consecutive layers in the candidate model, we have a nonlinear activation function on top of a linear layer (sec 3.1.3). Notably, each stitching part involves only two parameters: $\mathbf{W}$ and $\mathbf{b}$, which dynamically adjust in dimensions based on the specific layers being stitched. Importantly, when compared to the parameter quantities present in the original models, the size of these parameters remains relatively modest. Furthermore, we only tune the stitching parts with fixing all the other parameters in the candidate models at the server side, which allows us to effectively enhance model performance while maintaining computational efficiency.
[1] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.
[2] T Dinh, Canh, Nguyen Tran, and Josh Nguyen. "Personalized federated learning with moreau envelopes." Advances in Neural Information Processing Systems 33 (2020): 21394-21405.
`>>> Improvement of state-of-the-art`
`>>> Answer:`We appreciate your comments. It's worth noting that in comparison to the CIFAR-10 dataset, the MNIST and SVHN datasets are relatively less complex. While the performance enhancements on these latter two datasets may not always yield statistically significant results, it's still possible to discern discernible improvements.
We would like to emphasize that these outcomes are based on scenarios where both client data and public data are derived from the same dataset.
Furthermore, we have conducted another experiment to specifically investigate the impact of public data (Figure 1 in the original paper). Interestingly, when the public dataset and client data originate from different datasets, the performance gap between the proposed model and the baselines is further magnified. This underscores the significance of our approach and its potential for even greater differentiation in such scenarios.