Unsupervised Model Personalization while Preserving Privacy and Scalability: An Open Problem

This work investigates the task of unsupervised model personalization,\nadapted to continually evolving, unlabeled local user images. We consider the\npractical scenario where a high capacity server interacts with a myriad of\nresource-limited edge devices, imposing strong requirements on scalability and\nlocal data privacy. We aim to address this challenge within the continual\nlearning paradigm and provide a novel Dual User-Adaptation framework (DUA) to\nexplore the problem. This framework flexibly disentangles user-adaptation into\nmodel personalization on the server and local data regularization on the user\ndevice, with desirable properties regarding scalability and privacy\nconstraints. First, on the server, we introduce incremental learning of\ntask-specific expert models, subsequently aggregated using a concealed\nunsupervised user prior. Aggregation avoids retraining, whereas the user prior\nconceals sensitive raw user data, and grants unsupervised adaptation. Second,\nlocal user-adaptation incorporates a domain adaptation point of view, adapting\nregularizing batch normalization parameters to the user data. We explore\nvarious empirical user configurations with different priors in categories and a\ntenfold of transforms for MIT Indoor Scene recognition, and classify numbers in\na combined MNIST and SVHN setup. Extensive experiments yield promising results\nfor data-driven local adaptation and elicit user priors for server adaptation\nto depend on the model rather than user data. Hence, although user-adaptation\nremains a challenging open problem, the DUA framework formalizes a principled\nfoundation for personalizing both on server and user device, while maintaining\nprivacy and scalability.\n