Deep Metric Learning (DML) aims to find representations suitable for\nzero-shot transfer to a priori unknown test distributions. However, common\nevaluation protocols only test a single, fixed data split in which train and\ntest classes are assigned randomly. More realistic evaluations should consider\na broad spectrum of distribution shifts with potentially varying degree and\ndifficulty. In this work, we systematically construct train-test splits of\nincreasing difficulty and present the ooDML benchmark to characterize\ngeneralization under out-of-distribution shifts in DML. ooDML is designed to\nprobe the generalization performance on much more challenging, diverse\ntrain-to-test distribution shifts. Based on our new benchmark, we conduct a\nthorough empirical analysis of state-of-the-art DML methods. We find that while\ngeneralization tends to consistently degrade with difficulty, some methods are\nbetter at retaining performance as the distribution shift increases. Finally,\nwe propose few-shot DML as an efficient way to consistently improve\ngeneralization in response to unknown test shifts presented in ooDML. Code\navailable here:\nhttps://github.com/CompVis/Characterizing_Generalization_in_DML.\n