Summary
This paper comprehensively evaluated the effect of different components of DNNs on heterogeneous data federated learning. Based on their empirical, experimental results, the paper further proposes the neural architecture design principle guidelines for FL. Leveraging the proposed guideline, pure CNNs can achieve competitive performance with ViTs in data heterogeneity FL.
Weaknesses
**Proposed principle – removing all normalization layers and normalization-free setup**:
The paper conducts an experiment to show the effectiveness of removing all normalization layers only on the neural architecture view and lacks consideration for other factors.
It's important to note that the aim of normalization technique is to create faster and more stable training, while improving generalization (i.e generalization performance of the network on unseen data.)
The normalization layer is highly "model size and dataset" aware; for instance, the larger the dataset, the superior results with the normalization you will get. In your experiments, for instance, the normalization technique in ResNet is targeted to a larger dataset i.e ImageNet. So instead of simply keeping and removing normalization, you should also consider the dataset and mode size. Since FL is a complex system, some clients may run large-size models, and simple removal of normalization from a large model might easily go to over-fittings.
In short, more comprehensive factors need to be discussed.
**Proposed principle – Reducing activation layers**:
The effect of reducing activation is similar to adjusting the size of Deep Neural Networks ( reducing the `depth` of DNNs and increasing the `width` of the DNN).
For instance, let's consider 2 hidden layers in DNNs: layer 1 with weights W1, and layer 2 with weights W2. The calcuation with activation $\sigma$: $$Y =\sigma(W_2 \sigma(W_1 X))$$
if we remove the activation between layer 1 and 2, the computation between layer 1 and 2 will become:
$$Y = \sigma(W_2 W_1 X)$$
According to the associative property of matrix multiplication, we can have $Y = \sigma((W_2 W_1) X)$, and let $W_3 =W_2 W_1 $, then we have $Y = \sigma(W_3 X)$. Hence, it's similar to replacing layer 1 and layer 2 with a new layer with learnable weights $W_3$
Again, in this paper's case, the effect of removing activation layers is essentially we modify the DNNs' size (the depth and width), and the size of the DNNs is a complex hyper-parameter tuning and AutoML topic, it depends on many factors and no universal answers.
**Proposed principle – Kernel Size**:
The paper [1] also has a similar conclusion, they use large kernel design 31*31, but not in FL settings, and achieve competitive performance with ViT. However, selecting the right kernel size raises trade-offs among model size, computational complexity, memory usage, etc. It's not always universal for the larger kernel size the better, it depends on specific questions and tasks.
**The interpretability of neural architecture in FL shouldn't separate from existing research**:
It's very hard to separate the interpretability of the neural architecture in FL from existing research, again, FL systems are complex and dynamic, and it's challenging to find a universal guideline for all cases. Even the authors in this paper tried to limit their domain to heterogeneous data (Non-IID) FL, the experiment settings are primarily focused on neural architecture view. and lack the ablations on different data-heterogeneous settings. I list some of the references for investigating the effect and interpretability of neural architecture components for general model optimization guidelines. [1, 2, 3, 4]
**Unfair design of experiments**:
Since FL challenges traditional centralized training with its heterogeneous data, simply active and removing some components of DNNs is unfair for investigating the effect in FL. Different non-IID settings need to be discussed. For instance, the normalization designed for overfittings and generalizability, and the overfitting and generalizability should also be discussed, even in an FL setup.
**Reference**
[1]Ding, Xiaohan, et al. "Scaling up your kernels to 31x31: Revisiting large kernel design in cnns." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
[2] Ş. Öztürk, U. Özkaya, B. Akdemir and L. Seyfi, "Convolution Kernel Size Effect on Convolutional Neural Network in Histopathological Image Processing Applications," 2018 International Symposium on Fundamentals of Electrical Engineering (ISFEE), Bucharest, Romania, 2018, pp. 1-5, doi: 10.1109/ISFEE.2018.8742484.
[3] Yu, Jiahui, and Konstantinos Spiliopoulos. "Normalization effects on deep neural networks." arXiv preprint arXiv:2209.01018 (2022).
[4]Liu, Sheng, et al. "Convolutional normalization: Improving deep convolutional network robustness and training." Advances in neural information processing systems 34 (2021): 28919-28928.
Questions
Please address my concern in the Weakness section, besides that:
I like the idea of this paper since model model compression and architecture design are growing important to edge-FL scenarios.
However, addressing the interoperability issue of neural networks shouldn't be separate from the existing research. Readers are more expected to see the heterogeneous data (Non-IID)'s effect for mode architecture components. Hence, experiments with various Non-IID levels of settings are essential to improve the fairness of the experiments.