QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning

Traditionally, federated learning (FL) aims to train a single global model\nwhile collaboratively using multiple clients and a server. Two natural\nchallenges that FL algorithms face are heterogeneity in data across clients and\ncollaboration of clients with {\\em diverse resources}. In this work, we\nintroduce a \\textit{quantized} and \\textit{personalized} FL algorithm QuPeD\nthat facilitates collective (personalized model compression) training via\n\\textit{knowledge distillation} (KD) among clients who have access to\nheterogeneous data and resources. For personalization, we allow clients to\nlearn \\textit{compressed personalized models} with different quantization\nparameters and model dimensions/structures. Towards this, first we propose an\nalgorithm for learning quantized models through a relaxed optimization problem,\nwhere quantization values are also optimized over. When each client\nparticipating in the (federated) learning process has different requirements\nfor the compressed model (both in model dimension and precision), we formulate\na compressed personalization framework by introducing knowledge distillation\nloss for local client objectives collaborating through a global model. We\ndevelop an alternating proximal gradient update for solving this compressed\npersonalization problem, and analyze its convergence properties. Numerically,\nwe validate that QuPeD outperforms competing personalized FL methods, FedAvg,\nand local training of clients in various heterogeneous settings.\n