Summary
The authors propose a nonparametric version of FedMD, a distillation-based collaborative learning methodology. They also propose a neural network variant of the nonparametric approach as an extension for heterogeneous local neural network models. They provide both theoretical results on the nonparametric approach and experimental results demonstrating efficacy of the neural network extension. The theoretical results prove the nearly minimax optimality of this nonparametric collaborative learning algorithm in massively distributed statistically heterogeneous environments.
Strengths
(1) The authors prove the (nearly) minimax optimality of a privacy-preserving collaborative learning algorithm using kernel regression in massively distributed, statistically heterogeneous environments. (2) Theorem 3.4 addresses a more general setting than prior works [48, 58]. Su et al. [58] only cover r = \frac{1}{2} of Assumption 3.3. Park et al. [48] do not consider Assumption 3.2, which provides a finer result. Furthermore, compared to [48], the required size of public inputs is reduced and the statistical homogeneity condition is dropped. (3) The rate n^{-\frac{r}{2r+s}} is the minimax lower rate under Assumptions 3.1, 3.2, and 3.3, making DCL-KR nearly optimal in a minimax sense. (4) On the difference between public and local distributions: Theorem 3.4 allows the public input distribution \tilde{\rho}_x to differ from the local input distribution \rho_x. (5) Finally, The difference between \rho_x and \tilde{\rho}_x impacts the expected risk upper bound as a multiplication of B_r, which can be removed by increasing public inputs.
Weaknesses
(1) The need to perform learning rate scaling to ensure the impact of local iterations is consistent across models introduces additional complexity. The exact method for scaling may vary and might not be trivial to determine or implement. (2) The computation of Gram matrices for CKA involves O(p^2) operations, where p is the number of public data points. This quadratic complexity can become prohibitive as the size of the public dataset increases. (3) The method assumes that the ensemble kernel k = \sum_{i=1}^{m} \frac{n_i}{n} k_{f_i} will perform better than individual feature kernels. This assumption might not hold in cases where local models are highly heterogeneous, as the averaged kernel could lose important characteristics of individual models, leading to suboptimal performance. The authors empirically verify the superiority of the ensemble in the appendix.
Questions
(1) How do you handle the computational burden associated with the calculation of Gram matrices for large datasets? (2) How do you handle cases where these representations before the last layers of the neural networks are suitable for kernel matching?