To achieve peak predictive performance, hyperparameter optimization (HPO) is\na crucial component of machine learning and its applications. Over the last\nyears, the number of efficient algorithms and tools for HPO grew substantially.\nAt the same time, the community is still lacking realistic, diverse,\ncomputationally cheap, and standardized benchmarks. This is especially the case\nfor multi-fidelity HPO methods. To close this gap, we propose HPOBench, which\nincludes 7 existing and 5 new benchmark families, with a total of more than 100\nmulti-fidelity benchmark problems. HPOBench allows to run this extendable set\nof multi-fidelity HPO benchmarks in a reproducible way by isolating and\npackaging the individual benchmarks in containers. It also provides surrogate\nand tabular benchmarks for computationally affordable yet statistically sound\nevaluations. To demonstrate HPOBench's broad compatibility with various\noptimization tools, as well as its usefulness, we conduct an exemplary\nlarge-scale study evaluating 13 optimizers from 6 optimization tools. We\nprovide HPOBench here: https://github.com/automl/HPOBench.\n