New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning

In Federated Learning (FL), the clients learn a single global model (FedAvg)\nthrough a central aggregator. In this setting, the non-IID distribution of the\ndata across clients restricts the global FL model from delivering good\nperformance on the local data of each client. Personalized FL aims to address\nthis problem by finding a personalized model for each client. Recent works\nwidely report the average personalized model accuracy on a particular data\nsplit of a dataset to evaluate the effectiveness of their methods. However,\nconsidering the multitude of personalization approaches proposed, it is\ncritical to study the per-user personalized accuracy and the accuracy\nimprovements among users with an equitable notion of fairness. To address these\nissues, we present a set of performance and fairness metrics intending to\nassess the quality of personalized FL methods. We apply these metrics to four\nrecently proposed personalized FL methods, PersFL, FedPer, pFedMe, and\nPer-FedAvg, on three different data splits of the CIFAR-10 dataset. Our\nevaluations show that the personalized model with the highest average accuracy\nacross users may not necessarily be the fairest. Our code is available at\nhttps://tinyurl.com/1hp9ywfa for public use.\n