Understanding the wiring evolution in differentiable neural architecture search

Controversy exists on whether differentiable neural architecture search\nmethods discover wiring topology effectively. To understand how wiring topology\nevolves, we study the underlying mechanism of several existing differentiable\nNAS frameworks. Our investigation is motivated by three observed searching\npatterns of differentiable NAS: 1) they search by growing instead of pruning;\n2) wider networks are more preferred than deeper ones; 3) no edges are selected\nin bi-level optimization. To anatomize these phenomena, we propose a unified\nview on searching algorithms of existing frameworks, transferring the global\noptimization to local cost minimization. Based on this reformulation, we\nconduct empirical and theoretical analyses, revealing implicit inductive biases\nin the cost's assignment mechanism and evolution dynamics that cause the\nobserved phenomena. These biases indicate strong discrimination towards certain\ntopologies. To this end, we pose questions that future differentiable methods\nfor neural wiring discovery need to confront, hoping to evoke a discussion and\nrethinking on how much bias has been enforced implicitly in existing NAS\nmethods.\n