Local Search is a Remarkably Strong Baseline for Neural Architecture Search

Neural Architecture Search (NAS), i.e., the automation of neural network\ndesign, has gained much popularity in recent years with increasingly complex\nsearch algorithms being proposed. Yet, solid comparisons with simple baselines\nare often missing. At the same time, recent retrospective studies have found\nmany new algorithms to be no better than random search (RS). In this work we\nconsider, for the first time, a simple Local Search (LS) algorithm for NAS. We\nparticularly consider a multi-objective NAS formulation, with network accuracy\nand network complexity as two objectives, as understanding the trade-off\nbetween these two objectives is arguably the most interesting aspect of NAS.\nThe proposed LS algorithm is compared with RS and two evolutionary algorithms\n(EAs), as these are often heralded as being ideal for multi-objective\noptimization. To promote reproducibility, we create and release two benchmark\ndatasets, named MacroNAS-C10 and MacroNAS-C100, containing 200K saved network\nevaluations for two established image classification tasks, CIFAR-10 and\nCIFAR-100. Our benchmarks are designed to be complementary to existing\nbenchmarks, especially in that they are better suited for multi-objective\nsearch. We additionally consider a version of the problem with a much larger\narchitecture space. While we find and show that the considered algorithms\nexplore the search space in fundamentally different ways, we also find that LS\nsubstantially outperforms RS and even performs nearly as good as\nstate-of-the-art EAs. We believe that this provides strong evidence that LS is\ntruly a competitive baseline for NAS against which new NAS algorithms should be\nbenchmarked.\n