Response to Reviewer hdps
Thanks for your valuable comments. We hope our response below can adequately address your concerns.
----
**Q1**: The novelty is quite limited, where the proposed method is to use an existing reparametrization trick (Brock et al. (2021a)) in the FL setting.
A1: We would like to highlight that we are the first work that considers FL without normalization. Normalizations have been an important topic in FL to handle statistical heterogeneity. Our new perspective offers important insights for the community. We believe our extensive empirical studies provide useful lessons for the community on the intriguing properties of FL without normalization.
----
**Q2**: Why the proposed method is only applied to the convolution layer? From Eq. 1, it seems that it can be applied to any kinds of layers represented by a weight matrix W.
A2: The equation outlines a general methodology applicable to various layers represented by weight matrices. We initially focus on convolution layers as they are commonly followed by a BN layer (e.g., ResNet and model architectures in Table 5 and 6 in the Supplementary). It has the potential to extend to other layers such as fully connected layers in scenarios where it comes with BNs.
----
**Q3**: In Eq. 1, how is γ chosen? In particular, how robust is the training result to the choice of γ?
A3: Thank you for raising this question. Eq. 1 is the standard BN formulation, so we suppose the reviewer is interested in our method's γ in Eq. 3. The value of γ is associated with the activation function used. For ReLUs, $γ = \sqrt{\frac{2}{1-(1-\pi)}}$.
----
**Q4**: Since the proposed method does not depend of batch statistics, I'm curious why FedWon B=1 better than B=2 (e.g., Figure 4)?
A4: Thank you for this interesting question. The results shown in the manuscript use the same learning rate of 0.01 for both batch sizes B = 1 and B = 2 of FedWon. To further analyze this result, we further tune the learning rate of B = 2 to 0.02 and they achieve similar performance, as shown in the table below. We have updated the paper to reflect these results.
| B | MNIST | SVHN | USPS | SynthDigits | MNIST-M | Average |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 98.7 | 85.8 | 98.3 | 93.7 | 90.6 | 93.4 |
| 2 | 98.6 | 85.6 | 98.2 | 93.7 | 90.8 | 93.4 |
----
**Q5**: Also, in figure 4, I'm curious to see how would FedAvg (B=1) perform.
A5: We also intend to report results for FedAvg with B = 1. However, methods such as FedAvg contains BN layers and cannot run with B = 1. This is because the statistics computed in BN (such as mean and variance) rely on batch-wise computations, and with a batch size of 1, there's no variance among samples, making normalization difficult.
----
**Q6**: It is claimed in this paper that FedBN can't do cross-device FL. However, there is not enough evidence, as far I can tell, from the paper that supporting this argument. Can the authors elaborate more on why FedBN can't do cross-device FL?
A6: FedBN stores BN statistics locally in clients and reloads these BN statistics after receiving the global model aggregated in the server. It assumes that the clients are stateful -- each client may participate in each round of the computation, carrying these BN statistics from round to round.
However, cross-device FL described in [1] involves a vast number of clients, with only a small subset participating in FL training per round. In cross-device FL, clients are mostly stateless -- each client will likely participate only once in the whole training.
When applying FedBN to cross-device FL, the randomly selected clients have a high possibility of not carrying the BN statistics from the previous round. Thus, FedBN is largely unsuitable for cross-device FL.
[1] Kairouz, Peter, et al. "Advances and open problems in federated learning." Foundations and Trends® in Machine Learning 14.1–2 (2021): 1-210.