Robustifying and Boosting Training-Free Neural Architecture Search

Response to Reviewer 81X4

Dear Reviewer 81X4, We thank you for taking the time to review our paper and for your valuable feedback. We would like to address your concerns below. ## About the Efficiency of RoBoT > the queried architecture should be trained from scratch to obtain the objective evaluation during the BO stage, which seems to require large amounts of search costs since a standard BO procedure usually requires tens of queries. Thank you for pointing out the concern regarding the search efficiency of each query in our method. In practice, we usually apply the validation performance of each candidate architecture that is trained from scratch for only a limited number of epochs (e.g., 12 epochs in NAS-Bench-201) to approximate the true performance of this architecture as stated in our Appendix B.2 and C.2, which in fact is quite computationally efficient (e.g., about 110 GPU-seconds for the model training of each candidate in NAS-Bench-201). Of note, such an approximation, which has been applied in the literature [1], is already reasonably good to help find well-performing architectures, as supported by the results in both Table 2 and Table 4. We would like to add this discussion to the main body of our revised paper to make it clearer. > RoBoT only requires 0.6 GPU-day to search, does it only count the search cost of the greedy search procedure? I wonder what is the cost of the BO stage. We thank you for pointing out this question. We would like to clarify that the search cost in our paper includes the computational costs incurred not only in the greedy search stage but also in the BO stage. In fact, these two stages have the same type of training cost that is used to evaluate the validation performance of each queried candidate architecture. Specifically, such a performance evaluation procedure requires about 0.04 GPU-day for the model training of each queried architecture in the DARTS search space for ImageNet where we have queried approximately 3 architectures in the BO stage and 7 architectures in the greedy search procedure. We would like to add this discussion to the main body of our revised paper to make it clearer. ## About the Effectiveness of RoBoT and Greedy Search > According to Fig. 2, it seems that the optimal architecture has been found in less than 10 queries during the BO procedure. It shows that there is no need to conduct the greedy search through $M^*$, and BO is enough to get the optimal architecture. We would like to clarify that the search costs in Figure 2 include the queries from both the BO stage (e.g., the first 5 queries in Figure 2a that are automatically determined by our RoBoT) and the greedy search procedure (e.g., the last 15 queries in Figure 2a). The results show that greedy search is indeed essential and necessary to help our method achieve competitive search performances, which is further supported by our ablation study in Figure 6. To enhance clarity, we will revise Figure 2 to highlight the search costs incurred by the BO stage and the greedy search procedure separately. ## About Notations > What does $R_f(A)$ denote in Eq. 1? Does it represent the objective evaluation of an architecture? Since Alg.1 directly uses $f$ to denote the objective evaluation metric, I suggest the authors utilize the same notation. Thank you for addressing the concern regarding our use of the notations $R_f(A)$ and $f$. As presented in the first paragraph of Section 4, $f(A)$ denotes the objective evaluation metric for $A$, while $R_f(A)$ denotes the **ranking** of $A$ in the entire search space based on the objective evaluation metric $f$. Particularly, $R_f(A)=1$ indicates that architecture $A$ is an optimal architecture in the entire search space based on the evaluation metric $f$. Of note, we introduce $R_f$ mainly to facilitate our theoretical analysis detailed in Section 4.3 and Section 5, given that our theoretical analysis mainly focuses on the ranking performance. But in fact, finding the minimum of $R_f(A(w))$ w.r.t $A(w)$ is essentially equivalent to find the maximum of $f(A(w))$ w.r.t $A(w)$, according to our definition of $R_f(A)=1$. We agree with you that using both $R_f(A)$ and $f$ in our formulation and algorithm could be confusing. Therefore, we would like to follow your suggestion to revise Equation 1 using only $f$ and reserve $R_f(A)$ only for the theoretical sections. This change aims to make our presentation more coherent and accessible to our readers. --- Thank you again for your detailed and valuable feedback. We hope that our additional clarifications could help improve your opinion of our paper. [1] Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search, NeurIPS 2022

Comment on authors' rebuttal?

@Reviewer 81X4: Does the reply by the authors address the issues raised by you? Do you have any follow-up questions or comments?

Thanks to Reviewer 81X4

Dear Reviewer 81X4, We would like to thank you again for appreciating the motivation and effectiveness of our paper and for the constructive comments to further improve our paper. Kindly let us know if you have any further comments on our paper, and we would like to do our best to address them in the remaining time.

Follow-up

@Reviewer GJPv: Thanks for your earlier review. Apart from the issues you raised with writing, did you have more specific issues with the method or experiments? If yes, has the revision by the authors improved on any of those issues? If no, could you please list a few pressing issues that still persist, in your opinion? Thanks.

Response to Reviewer 3buk

Dear Reviewer 3buk, Thank you for recognizing the strengths of our paper and providing the valuable feedbacks. We address your concerns as follows: ## About Robust Metric > What are the base training-free metrics considered in the search? As reported in our Appendix B.2 and C.2, we borrowed the six widely known training-free metrics (i.e., *grad_norm, snip, grasp, fisher, synflow,* and *jacob_cov*) from [1] for most of the experiments in our paper except for the tasks of *Segment.*, *Normal*, and *Autoenco* in TransNAS-Bench-101 where *synflow* was excluded due to its incompatibility with the tanh activation applied in each candidate architecture within the search space. > What are the optimized linear combination weights for them? Do they significantly differ on different datasets/tasks? Thank you for raising this interesting question regarding the optimized linear combination weights in various tasks. We present the optimized weights, along with their similarities and correlations, across four tasks in TransNAS-Bench-101-micro as follows: Varying Optimized Weights | | grad_norm | snip | grasp | fisher | synflow | jacob_cov | |---|---|---|---|---|---|---| | Scene | -1.00 | -0.08 | -0.97 | 1.00 | 1.00 | 1.00 | | Object | 0.03 | -0.21 | -0.76 | 0.51 | 0.95 | 0.16 | | Jigsaw | -0.74 | 0.18 | 0.04 | -1.00 | -1.00 | 1.00 | | Layout | -0.65 | -0.27 | 0.57 | -0.48 | 1.00 | 0.67 | Cosine Similarity | | Scene | Object | Jigsaw | Layout | |---|---|---|---|---| | Scene | 1.00 | 0.78 | -0.07 | 0.37 | | Object | 0.78 | 1.00 | -0.55 | 0.19 | | Jigsaw | -0.07 | -0.55 | 1.00 | 0.2 | | Layout | 0.37 | 0.19 | 0.2 | 1.00 | Pearson Correlation | | Scene | Object | Jigsaw | Layout | |---|---|---|---|---| | Scene | 1.00 | 0.77 | -0.02 | 0.35 | | Object | 0.77 | 1.00 | -0.52 | 0.16 | | Jigsaw | -0.02 | -0.52 | 1.00 | 0.3 | | Layout | 0.35 | 0.16 | 0.3 | 1.00 | The results show that the optimized weights typically vary for different tasks, which aligns with the observations and motivations in Section 3 and further highlights the necessity of developing robust metrics that can perform consistently well on diverse tasks such as our RoBoT. In addition, for tasks with similar characteristics, e.g., the *Scene* and *Object* tasks, both of which are classification tasks, the optimized weights share a relatively high similarity/correlation, indicating the potential transferability of the optimized linear combination within similar tasks. We would like to add these results and discussions to our revised paper. ## About Including More Baselines We appreciate your suggestion on more recent baselines. We show our comparison on NAS-Bench-201 as below. Note that we have applied smaller search budgets for Shaply-NAS and $\beta$-DARTS than the ones presented in their original papers in order to maintain comparable search budgets for different training-based NAS algorithms (e.g., DARTS (2nd), GDAS, DrNAS). The results show that our RoBoT can still achieve better search performances than these two recent baselines. We will add these baselines and comparisons in our revised paper. | | C10 (Test Acc %) | C100 (Test Acc %) | IN-16 (Test Acc %) | Cost (GPU Sec.) | |---|---|---|---|---| | Shaply-NAS | 94.05±0.19 | 73.15±0.26 | 46.25±0.25 | 14762 | | $\beta$-DARTS | 94.00±0.22 | 72.91±0.43 | 46.20±0.38 | 3280 | | RoBoT | **94.36**±0.00 | **73.51**±0.00 | **46.34**±0.00 | 3051 | ## About the Presentation of Table 3 > Why are the results on TransNAS-Bench-101 presented as the validation ranking? We would like to clarify that since there is a noticeable gap between the validation and the test performances in TransNAS-Bench-101, we follow the common practice [2] to present the validation ranking, as detailed in Appendix B.2. > the search costs are not listed in Table 3 We would like to clarify that, in fact, we have included the search costs (measured by the number of queries, i.e., 100) in the caption of Table 3. --- We sincerely hope our clarifications above have addressed your concerns and can improve your opinion of our work. [1] Zero-Cost Proxies for Lightweight NAS, ICLR 2021 [2] NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies, NeurIPS Datasets and Benchmarks Track 2022

Comment on authors' rebuttal?

@Reviewer 3buk: Does the reply by the authors address the issues raised by you? Do you have any follow-up questions or comments?