Federated learning (FL) provides a distributed learning framework for\nmultiple participants to collaborate learning without sharing raw data. In many\npractical FL scenarios, participants have heterogeneous resources due to\ndisparities in hardware and inference dynamics that require quickly loading\nmodels of different sizes and levels of robustness. The heterogeneity and\ndynamics together impose significant challenges to existing FL approaches and\nthus greatly limit FL's applicability. In this paper, we propose a novel\nSplit-Mix FL strategy for heterogeneous participants that, once training is\ndone, provides in-situ customization of model sizes and robustness.\nSpecifically, we achieve customization by learning a set of base sub-networks\nof different sizes and robustness levels, which are later aggregated on-demand\naccording to inference requirements. This split-mix strategy achieves\ncustomization with high efficiency in communication, storage, and inference.\nExtensive experiments demonstrate that our method provides better in-situ\ncustomization than the existing heterogeneous-architecture FL methods. Codes\nand pre-trained models are available: https://github.com/illidanlab/SplitMix.\n