Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators

While maximizing deep neural networks' (DNNs') acceleration efficiency\nrequires a joint search/design of three different yet highly coupled aspects,\nincluding the networks, bitwidths, and accelerators, the challenges associated\nwith such a joint search have not yet been fully understood and addressed. The\nkey challenges include (1) the dilemma of whether to explode the memory\nconsumption due to the huge joint space or achieve sub-optimal designs, (2) the\ndiscrete nature of the accelerator design space that is coupled yet different\nfrom that of the networks and bitwidths, and (3) the chicken and egg problem\nassociated with network-accelerator co-search, i.e., co-search requires\noperation-wise hardware cost, which is lacking during search as the optimal\naccelerator depending on the whole network is still unknown during search. To\ntackle these daunting challenges towards optimal and fast development of DNN\naccelerators, we propose a framework dubbed Auto-NBA to enable jointly\nsearching for the Networks, Bitwidths, and Accelerators, by efficiently\nlocalizing the optimal design within the huge joint design space for each\ntarget dataset and acceleration specification. Our Auto-NBA integrates a\nheterogeneous sampling strategy to achieve unbiased search with constant memory\nconsumption, and a novel joint-search pipeline equipped with a generic\ndifferentiable accelerator search engine. Extensive experiments and ablation\nstudies validate that both Auto-NBA generated networks and accelerators\nconsistently outperform state-of-the-art designs (including\nco-search/exploration techniques, hardware-aware NAS methods, and DNN\naccelerators), in terms of search time, task accuracy, and accelerator\nefficiency. Our codes are available at: https://github.com/RICE-EIC/Auto-NBA.\n