NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

Neural architecture search (NAS) has achieved breakthrough success in a great\nnumber of applications in the past few years. It could be time to take a step\nback and analyze the good and bad aspects in the field of NAS. A variety of\nalgorithms search architectures under different search space. These searched\narchitectures are trained using different setups, e.g., hyper-parameters, data\naugmentation, regularization. This raises a comparability problem when\ncomparing the performance of various NAS algorithms. NAS-Bench-101 has shown\nsuccess to alleviate this problem. In this work, we propose an extension to\nNAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple\ndatasets, and more diagnostic information. NAS-Bench-201 has a fixed search\nspace and provides a unified benchmark for almost any up-to-date NAS\nalgorithms. The design of our search space is inspired from the one used in the\nmost popular cell-based searching algorithms, where a cell is represented as a\nDAG. Each edge here is associated with an operation selected from a predefined\noperation set. For it to be applicable for all NAS algorithms, the search space\ndefined in NAS-Bench-201 includes all possible architectures generated by 4\nnodes and 5 associated operation options, which results in 15,625 candidates in\ntotal. The training log and the performance for each architecture candidate are\nprovided for three datasets. This allows researchers to avoid unnecessary\nrepetitive training for selected candidate and focus solely on the search\nalgorithm itself. The training time saved for every candidate also largely\nimproves the efficiency of many methods. We provide additional diagnostic\ninformation such as fine-grained loss and accuracy, which can give inspirations\nto new designs of NAS algorithms. In further support, we have analyzed it from\nmany aspects and benchmarked 10 recent NAS algorithms.\n