FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

Federated Learning (FL) has been gaining significant traction across\ndifferent ML tasks, ranging from vision to keyboard predictions. In large-scale\ndeployments, client heterogeneity is a fact and constitutes a primary problem\nfor fairness, training performance and accuracy. Although significant efforts\nhave been made into tackling statistical data heterogeneity, the diversity in\nthe processing capabilities and network bandwidth of clients, termed as system\nheterogeneity, has remained largely unexplored. Current solutions either\ndisregard a large portion of available devices or set a uniform limit on the\nmodel's capacity, restricted by the least capable participants. In this work,\nwe introduce Ordered Dropout, a mechanism that achieves an ordered, nested\nrepresentation of knowledge in deep neural networks (DNNs) and enables the\nextraction of lower footprint submodels without the need of retraining. We\nfurther show that for linear maps our Ordered Dropout is equivalent to SVD. We\nemploy this technique, along with a self-distillation methodology, in the realm\nof FL in a framework called FjORD. FjORD alleviates the problem of client\nsystem heterogeneity by tailoring the model width to the client's capabilities.\nExtensive evaluation on both CNNs and RNNs across diverse modalities shows that\nFjORD consistently leads to significant performance gains over state-of-the-art\nbaselines, while maintaining its nested structure.\n