Summary
The paper introduces a method for fine-tuning a client's model based on a pretrained model using its local data. The proposed personalization approach involves fine-tuning the learnable weights and biases (assume, there are $M$ such layers) as well as the means and standard deviations (SDs) of batch normalization (BN) layers (assume, there are $B$ BN layers).
This fine-tuning process is facilitated by two meta neural networks. The first meta neural network predicts $2M$ learning rates, which correspond to each weight and bias layer, enabling the local model to perform several gradient steps from the pretrained model. The input for this first meta neural network consists of the mean and standard deviation values for each of the $M$ layers.
On the other hand, the second meta neural network directly predicts the means and SDs. Its input comprises the distance between the pretrained means and the running means of the model with the pretrained weights on local data, as well as a similar distance measure between the SDs.
Both meta neural networks are trained using FedAvg on hyper gradients.
--------
Post-rebuttal update: I did not have significant concerns about this paper. I acknowledge the authors' rebuttal and extend my gratitude to them, particularly for adding the new experiment. I have opted not to modify my favorable rating of 6.
Strengths
1. The paper demonstrates sufficient originality by introducing a novel approach using meta neural networks for computing personalized hyperparameters, as outlined in the summary above. Notably, the concept of trainable meta nets holds promise and appears to have practical applicability.
2. The authors effectively elaborate on the algorithm's intricacies, providing comprehensive descriptions of various components of the main method. Additionally, Figure 1 illustrates a schematic depiction of the main approach, enhancing the clarity of their explanations.
3. Extensive computational experiments are incorporated within the paper to substantiate the efficacy of the proposed method.
4. The paper is a valuable contribution to the fields of personalized Federated Learning and Meta Learning.
Weaknesses
1. Despite the authors' efforts in providing extensive explanations for their ideas, regrettably, the text still contains several notable inconsistencies, as detailed in the "Questions" section.
2. A comparative analysis with the FOMAML method [1] is notably absent in the work. It is worth mentioning that FOMAML shares significant similarities with the authors' proposed method, as both utilize the MAML approach for fine-tuning the model using the training dataset and making optimization steps for the meta parameter based on the validation dataset, as observed in the current study. Additionally, in another related work [2], which builds upon MAML, the hyper gradient is explicitly addressed (see lines 14 and 16 of Algorithm 1 in [2]).
[1] "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks", Finn et al.
[2] "Unsupervised Meta-Learning for Few-Shot Image Classification", Khodadadeh et al.
Questions
1. Line 225: If I understand correctly, the text implies that $\partial_{{\omega}_{lr}} {\cal L}_V = 0$. If my interpretation is correct, could you please explain why this is the case?
2. Equation 3 & Line 199: What is the range of the index $i$?
Minor comments & suggestions:
1. Line 6: Depending -> depending
2. Line 35: Even -> even
4. Inaccuracy on lines 87-91: Although it is evident what the authors intended to convey, the wording is imprecise. I would recommend specifying the subset of chosen clients as $\tilde{C}$ on line 87, as done in Algorithm 1, and then using $i \in \tilde{C}$ instead of $i \in C$ on line 88. The statement $P_i = \cal{X} \times \cal{Y}$ on line 90 is also misleading: $\cal{X} \times \cal{Y}$ represents the domain of the distribution and does not define the distribution itself. Accordingly, the summation on line 91 should be taken over $i \in \tilde{C}$.
5. Line 93: The space before the word "for" is missing.
6. The first part of the statement on lines 93-95, in my opinion, is incorrect. From a statistical perspective, the aim of Federated Learning is to minimize the generalized local loss function $\mathbb{E}_{(x, y) \in P_i} \cal{L}_i (\theta, x, y)$. However, many Federated Learning frameworks minimize the finite sum because they make an implicit assumption that the distributions are similar. Otherwise, one could construct two distributions $P_1$ and $P_2$ such that the global solution for $1/2(P_1 + P_2)$ is ineffective for both clients.
7. The abbreviation SD is used on line 109 but only defined later on line 180.
8. Line 121: "are" -> "is"
10. Line 191: For clarity, I would suggest spelling out the abbreviation MLP.
11. Line 190: Please provide explicit definitions and domains for $\eta_i$ and $\beta_j$ as they are not scalars.
12. Equation 5 is misleading. The loss function on both lines is taken over the data distribution $P_i$, but there are also indices $T$ and $V$, which denote train and test datasets. $theta_i(\lambda)$ is not a minimizer of the generalized loss function but rather a fine-tuning of the global solution $\theta_g$. It takes some time to understand the authors' intended meaning here.
13. Line 226: There is a redundant full stop after the first word "Algorithm."
14. Algorithm 1, the input section: The phrases "Total no. of clients C. Num of update iterations K" use different notation to express "number," leading to inconsistency within the text.
15. Lines 9-10 in Algorithm 1 do not clearly depict the intended FedAvg iterates.
16. The document concludes on page 8 with Section "4.4 Personalizing to Different Domains."
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Authors adequately address the limitations of the work.