Summary
The authors propose an algorithm for decentralized learning, where training datasets with heterogenous distributions are collected across different devices. They propose a global update tracking (GUT)-based approach where the IID assumption for the data is removed, and aim to generalize their method to non-IID data by tracking the consensus model while reducing the communication overhead by their GUT algorithm, at the expense of additional memory for storing the parameters for the neighboring model. They validate their method by experiments on various datasets.
Strengths
- Proposed method is applicable to more realistic settings for decentralized learning, where non-IID heterogeneous distribution is more common for the real world data.
- Their GUT-based approach adds minimal computational overhead on the overall agents, and they show the performance of their algorithm on multiple datasets in the experimental results.
- They provide ablation studies for further analysis on the hyperparameter sensitivity, different levels of heterogeneity, and various implementations of momentum.
- Authors provide the source code for their implementation in the supplementary materials.
Weaknesses
- Since one of the main claims of this approach is its generalizability to non-IID heterogeneous data distributions for each agent, what aspect in the proposed formulation corrects the gap between the individual distributions?
- Compared to previous decentralized learning approaches where the agent tracks gradients, momentum of gradients, or gloabl gradient obtained from the adjacent agents, proposed GUT tracks model updates. What statistical characteristic/effect makes the model update ($x^{t+1}_i - x^{t}_i$) beneficial for tracking? A simple toy (e.g. Gaussian) example experiment with update with/without model update tracking would suffice.
- Are there any visualizations for non-IID characteristic on the datasets under varying degree of $\alpha$? Also, are there experimetal results of additional memory consumption and computational overhead due to the additional update terms?
Questions
Please refer to the questions in the weaknesses section. Since I am not an expert in this field, my positive and negative opinions are mixed.
Additionally, repeating the full paper in the supplementary materials seems unnecessary.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
Limitations
The authors included a separate discussions and limitations section in their paper, with adequate explanations and descriptions on the weaknesses and possible future directions.