Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

While early AutoML frameworks focused on optimizing traditional ML pipelines\nand their hyperparameters, a recent trend in AutoML is to focus on neural\narchitecture search. In this paper, we introduce Auto-PyTorch, which brings the\nbest of these two worlds together by jointly and robustly optimizing the\narchitecture of networks and the training hyperparameters to enable fully\nautomated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art\nperformance on several tabular benchmarks by combining multi-fidelity\noptimization with portfolio construction for warmstarting and ensembling of\ndeep neural networks (DNNs) and common baselines for tabular data. To\nthoroughly study our assumptions on how to design such an AutoDL system, we\nadditionally introduce a new benchmark on learning curves for DNNs, dubbed\nLCBench, and run extensive ablation studies of the full Auto-PyTorch on typical\nAutoML benchmarks, eventually showing that Auto-PyTorch performs better than\nseveral state-of-the-art competitors on average.\n