Retargeting 3D garment meshes over digital characters and avatars involves non‐rigid deformation of garments to plausibly fit the target body in arbitrary poses. Existing learning‐based methods for garment retargeting require the garments to be canonicalized and can only be retargeted onto parametric human body models. In this work, we present a novel framework for retargeting garments in arbitrary poses to any given human mesh. We adopt a robust Isomap‐based representation to first estimate correspondences between garment and body mesh to achieve an initial coarse retargeting. We further adapt a fast and efficient neural optimization step, governed by Physics‐based constraints to obtain realistic draping of the garment. We show generalization to biped cartoon characters and non‐parametric human meshes in arbitrary poses. We perform extensive experiments on publicly available datasets and our proposed dataset of 3D clothing, demonstrating the effectiveness of our method.