EvoPrompting: Language Models for Code-Level Neural Architecture Search

Summary

This paper proposed an interesting idea for neural architecture search. Specifically, the idea is based on evolutionary computation and an LLM, where the LLM plays the role of genetic search, i.e., crossover operation and mutation operation. The experiments are conducted on two benchmarks: MNIST-1D and CLRS, which are for convolutional network networks and graph neural networks, respectively. =================================== Thanks very much for the authors' effort in explaining. As can be seen, the necessary ablation study is still not provided for the justification. However, considering the idea is interesting in this work, I will adjust my score to be 'Borderline reject'.

Strengths

The LLM is explored for the neural architecture search.

Weaknesses

The experiments can only serve the purpose of demonstration, while the usefulness of the idea is not clear. The main problem is the experiments are not common ones within the community of neural architecture search, and the compared algorithms are not the state of the arts in the community. The use of LLM plays the role of crossover and mutation, while the ablation studies are missing for the verification of LLM better than the original crossover operator and mutation operator. In addition, the fitness function is quite odd because it is obtained through the test from the authors, while the motivation is not clear.

Questions

This paper mentioned multiple times that existing evolutionary approaches for neural architecture search have required careful design and specification of a discrete search space, while the proposed method includes any neural network architecture. I am not sure of the particular meaning of: "any" in this claim. However, I would like to say that the references mentioned for the evolution approaches just used fixed-length encoding, so they may have that limitation. In addition, there is also some work [1] using evolutionary approaches for neural architecture search with variable-length encoding, the search space also includes any neural network architectures. [1] Sun et. al, “Evolving deep convolutional neural networks for image classification,” IEEE Transactions on Evolutionary Computation, vol. 24, no. 2, pp. 394-407, 2020.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Limitations

See Weaknesses.

Summary

The paper presented an evolutional neural architecture search method that utilizing LLM as mutator. During the evolutional search, the LLM is updated according to the evaluation result. Meanwhile the evaluation results are fed back to the LLM during the mutation process in the format of in-context learning.

Strengths

The work explores a novel direction of using LLMs in NAS. Many efforts have been made to design this approach to enable the LLM in NAS as it is not particularly trained on this task. The experimental results also proved the naive way of using LLM is not as effective as the proposed one.

Weaknesses

One missing part of the paper is why we will be interested in using LLM for NAS. Also the experiment mainly demonstrates the proposed method is better than naive prompting methods. How does the NAS with LLM compare to the existing NAS method? The experiments are better to be conducted with large scale realistic datasets.

Questions

1. How does the proposed method perform comparing to the existing non-LLM method? Given the same number of search? And what’s the computation resource used for LLM search method and nonLLM methods? 2. The author mentioned the CNN architecture search might be in the training set of the LLM. Does author train the LLM? Or is it a publicly accessible model?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors have not discussed the limitation and border impact. However, the reviewer has not seen any significant limitation of the method, and potential negative border impact.

Summary

This is an example of LLMs being used in evolutionary algorithms. The authors use an LLM to crossover code snippets that execute to define a graph or neural architecture. It does this by generative means, not syntactic manipulation as typically in EAs. They use code in the prompt for context and they improve the crossover by including code snippets that resulted in better results. The prompt also has ranges for the desired size of the model and accuracy what are set relative to a parent model, allowing them to guide an incremental improvement in accuracy while keeping model size in check. They empirically evaluate on MNIST-1D and on CLRS. On MNIST-1D they get variants in terms of accuracy and model size that are comparable to human designs or naive few shot prompting. On CLRS they get novel architectures that are better than a rather modest benchmark on 21/30 tasks (related to algorithm design). The added bonus is a method that improves prompts through evolutionary search.

Strengths

EvoPrompting is based on in-context prompting and the variation of the "candidate solution" occurs on a code representation. This changes the search space and makes problem solving more flexible and less reliant on adhoc choices on a different representation level. In general, the approach is applicable to LM tasks that rely on in-context learning (ICL) or prompt-tuning. These approaches don't need gradients. The results are promising on the CLRS Algorithmic Reasoning Benchmark but not compared to other NAS techniques. Novel architectures were discovered and they were smaller and had better accuracy than the benchmark.

Weaknesses

The choice of what's in the appendix and in the paper must be hard but I feel at least one code snippet should be shown. The paper could do with one entire example of the prompt and response. The search space variation in the MNIST-1D benchmark was modest. For the comparisons with CLRS, one architecture was the compared benchmark for all programs, while EvoPrompting was judged when it was allowed to evolve a a solution for each problem independently. Authors mention how one evolved model was evaluated with 3 different problems but no others. There is no comparison of the effort expanded to evolve and test vs NAS by another means. If the generality of the approach as stated by the authors is to be taken seriously, evidence beyond the narrow scope of code variation and NAS is necessary. There are many many more problems than MNIST-1D NAS or CLRS NAS.

Questions

How would evoprompting do on the problems in the GP symbolic regression benchmarks? How do they differ from CLRS and why or why are they not applicable here? Can you compare to NAS? How would you select one model from the 21 or 31 options and train them to be tested on all the other problems? What if you changed the seed architectures? They seem pretty good in both problems.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

NA

Summary

This paper introduces a method that uses LLMs as mutation and crossover operators in an evolutionary search process that generates diverse and high performing neural architectures. They evaluate their method on two datasets, MNIST-1D and CLRS, a benchmark measuring algorithmic reasoning. The evoprompting process generated SOTA models on CLRS and performed well on MNIST-1D. Importantly it generated nontrivial architectures.

Strengths

Originality: The concept of using a sequence model to generate neural network architectures in NAS is not new (Neural Architecture Search with Reinforcement Learning by Zoph), but was previously very constrained because the parameters the sequence model output were simple, things like width and kernel size for a CNN. This model is much more expressive, can describe a much broader set of architectures, because it generates code. Combining LLMs and evolution is also not strictly novel, see Language Model Crossover: Variation through Few-Shot Prompting, but I believe this is quite different. Quality: I think the quality of the writing and the research is high. They show state of the art on an admittedly new benchmark. Nobody has done this before, they show that the sample complexity of the search is good. Significance: It demonstrates a practical way to leverage the capabilities of language models for complex tasks which has wide ranging implications. I could imagine this being extended to program synthesis for example. I like that the authors included some sample architectures in the supplemental materials. It looks like the search discovered nontrivial architectures.

Weaknesses

The paper could be improved by: - Comparing to other NAS approaches - Evaluating on larger tasks. I don't fault the authors at all for this, NAS is expensive and I like that they used MNIST-1D, but more domains would strengthen the claims.

Questions

How do you think this would fare with larger models? Are there any specific domains where you think this approach would struggle?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

The paper explores prompt tuning for neural architecture search. While language models can produce code snippets with prompting, it is often quite difficult for language models to succeed at this task. The authors propose EvoPrompting, a prompt tuning method that produces neural architectures. The authors experiment with MNIST-1D where they find that EvoPrompting produces better performing neural networks with fewer parameters than the baselines. They then test their method on producing graph neural networks for algorithmic reasoning and find that they achieve the state-of-the-art on 21 out of 30 tasks.

Strengths

This is a high quality work with strong results on the CLRS algorithmic benchmark. The authors first carefully validate their method on a simple task MNIST-1D and find that their method produces smaller but effective neural networks compared to existing baselines. The results on the CLRS benchmark are quite impressive, sometimes improving higher than 20% (QuickSort) over the baseline while still having the same number of parameters.

Weaknesses

Analysis of the language model: the paper uses a large language model as a blackbox but fails to provide analysis about the language model itself. The paper trains a new 62B parameter decoder-only language model trained on conversational, web, and code documents. It would be useful to know if there are equivalent open-source models such as StarCoder that can be used to replicate the results? Furthermore, it is unclear how important the model size will affect the capabilities. It would be interesting to the readers to know the performance on CLRS across model sizes if possible. Unclear use of “soft prompt-tuning”: The work uses soft prompting in each round to find better in-context learning prompts. However, it is not clear if soft prompting is the best method to find the relevant prompts. It would be helpful to compare soft prompting with related techniques such as adapters [a], LoRA [b], and T-Few [c]. Nit: the work could provide more background regarding genetic programming similar to [d]. [a] Parameter-Efficient Transfer Learning for NLP. ICML 2020. [b] LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. [c] Improving In-Context Few-Shot Learning via Self-Supervised Training. NeurIPS 2022. [d] Evolution through Large Models. 2022.

Questions

See the questions for weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Acknowledgement

Thank you for your response. The authors have clarified my questions. I will keep my score at 7.

Re: ablations

Thanks for reviewing our rebuttal! We appreciate it. Concerning the ablations, we wanted to re-emphasize that: (1) Our paper includes **ablation studies of simpler crossover/mutation operators** that are natural with an LM setup, such as using naive few-shot prompting or selecting random parents to crossover from. (2) There is unfortunately no "original" crossover and mutation operator - these operators vary vastly in design across NAS works because they are search-space specific. One of the benefits of EvoPrompting is that it does not require a hand designed search space and so comparing to such a search space and its corresponding mutator operators would introduce too many confounders to be useful. (3) We have also implicitly compared against the crossover/mutation operators used in the methods published on the NATS-Bench benchmark. **An important question**: Maybe it'd be helpful if you could specify what you mean by the "original crossover operator and mutation operator"? It's hard for us to conduct these experiments or respond without understanding clearly what this experiment would be. Thank you!

Thanks for the response, my score is unchanged.

Thanks authors for the responses. I generally agree with most the intuition and potential benefits of LLMs that authors mentioned in the response: flexible space, knowledge transferred from other training tasks that could lead to high probability good results, etc. However, I don't think the experimental results solidly supports those benefits. Especially, A.9 is emphasized but its result can not conclude any of the baselines or the EvoPrompt is better than others.