NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

One-shot neural architecture search (NAS) has played a crucial role in making\nNAS methods computationally feasible in practice. Nevertheless, there is still\na lack of understanding on how these weight-sharing algorithms exactly work due\nto the many factors controlling the dynamics of the process. In order to allow\na scientific study of these components, we introduce a general framework for\none-shot NAS that can be instantiated to many recently-introduced variants and\nintroduce a general benchmarking framework that draws on the recent large-scale\ntabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS\nmethods. To showcase the framework, we compare several state-of-the-art\none-shot NAS methods, examine how sensitive they are to their hyperparameters\nand how they can be improved by tuning their hyperparameters, and compare their\nperformance to that of blackbox optimizers for NAS-Bench-101.\n