In Federated Learning (FL), clients may have weak devices that cannot train the full model or even hold it in their memory space. To implement large-scale FL applications, thus, it is crucial to develop a distributed learning method that enables the participation of such weak clients. We propose <inline-formula><tex-math notation="LaTeX">$\mathtt{EmbracingFL}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">EmbracingFL</mml:mi></mml:math><inline-graphic xlink:href="lee-ieq1-3392212.gif"/></alternatives></inline-formula>, a general FL framework that allows all available clients to join the distributed training regardless of their system resource capacity. The framework is built upon a novel form of partial model training method in which each client trains as many consecutive output-side layers as its system resources allow. Our study demonstrates that <inline-formula><tex-math notation="LaTeX">$\mathtt{EmbracingFL}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">EmbracingFL</mml:mi></mml:math><inline-graphic xlink:href="lee-ieq2-3392212.gif"/></alternatives></inline-formula> encourages each layer to have similar data representations across clients, improving FL efficiency. The proposed partial model training method guarantees convergence to a neighbor of stationary points for non-convex and smooth problems. We evaluate the efficacy of <inline-formula><tex-math notation="LaTeX">$\mathtt{EmbracingFL}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">EmbracingFL</mml:mi></mml:math><inline-graphic xlink:href="lee-ieq3-3392212.gif"/></alternatives></inline-formula> under a variety of settings with a mixed number of strong, moderate (<inline-formula><tex-math notation="LaTeX">$\sim\! 40\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mspace width="-0.166667em"/><mml:mn>40</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="lee-ieq4-3392212.gif"/></alternatives></inline-formula> memory), and weak (<inline-formula><tex-math notation="LaTeX">$\sim\! 15\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mspace width="-0.166667em"/><mml:mn>15</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="lee-ieq5-3392212.gif"/></alternatives></inline-formula> memory) clients, datasets (CIFAR-10, FEMNIST, and IMDB), and models (ResNet20, CNN, and LSTM). Our empirical study shows that <inline-formula><tex-math notation="LaTeX">$\mathtt{EmbracingFL}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">EmbracingFL</mml:mi></mml:math><inline-graphic xlink:href="lee-ieq6-3392212.gif"/></alternatives></inline-formula> consistently achieves high accuracy as like all clients are strong, outperforming the state-of-the-art width reduction methods (i.e., HeteroFL and FjORD).