Summary
This work proposes LLMatic which is a Neural Architecture Search approach using LLMs and quality diversity optimization. This work combines the domain knowledge of code-generating LLMs with a robust quality-diverse search mechanism.The application of LLMs trained for generating architecture code NAS is novel and interesting way to exploit the capabilities of code-gen LLMs. The method is evaluated on CIFAR-10 dataset and the NAS-bench-201 benchmark.
Strengths
- The application of code-generating LLMs effectively for NAS is very novel. Defining architecture generation as a language/code modelling task is a new way of formulating the neural architecture search problem.
- The presentation of the paper is mostly clear except in some parts (refer to suggestions and questions in the weakness and questions section)
- Evaluation on the NB201 dataset and CIFAR10 dataset fairly exhaustive and well ablated.
- The authors release their code and additional details on the prompts used
Weaknesses
- Evaluation is limited: Currently NAS focuses a lot of transformer-based spaces [1], mobilenet-spaces [2] which are more practical and realistic compared to cell-based spaces. A lot of thee approaches release a surrogate predictor or the supernet itself, to save training times ie the architecture training part in algorithm 1. I recommend the authors evaluate the method on these search spaces too. This is also in my opinion very important to study how llmatic scales across larger architecture definitions (code) and datasets eg: ImageNet.
- Comparison to black box approaches and generative approaches: Since llmatic requires training a lot of architectures (or querying a benchmark multiple times), its search time is more comparable to black-box approaches instead of approaches like lambda-darts. It would be great to add other black-box methods (in addition to random search) to Table 1. Furthermore since the work very much falls in the line of generative NAS a comparison with DiffusionNAG [3] would also be great.
- Clarity: In the current version of the paper in figure 4, I couldn't see the green points referring to llmatic architectures, am I missing something?
- Minor : Page 4 2nd paragraph "In the first generation , ta simple neural network with one convolutional and one fully connected
layer initiates the evolution" -> "In the first generation , take simple neural network with one convolutional and one fully connected
layer initiates the evolution"
[1] Chen, M., Peng, H., Fu, J. and Ling, H., 2021. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 12270-12280).
[2] Cai, H., Gan, C., Wang, T., Zhang, Z. and Han, S., 2019. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791.
[3] An, S., Lee, H., Jo, J., Lee, S. and Hwang, S.J., 2023. DiffusionNAG: Task-guided Neural Architecture Generation with Diffusion Models. arXiv preprint arXiv:2305.16943.
Questions
- Evaluation: Check points in the weakness (evaluation)
- Clarity : Code generating llms are often imperfect in generating pytorch code. They often introduce bugs and require some human intervention to make a generated code snippet compilable. Did the authors face any such issues? In other works how does a given prompt ensures that an LLM would restrict itself to generate compilable architectures within the NABench 201 search space for eg? The prompt examples provided do not seem to indicate any instructions to the LLM to ensure this
- Could the authors comment on how their work compares to [4] which explores a very similar direction (but with prompt evolution)?
- Reproducibility: Reproducibility of approaches is a major challenge in NAS. Since the work relies on (open-source) LLMs could the authors comment on the reproducibility of their approach?
- Could you report search time comparison across all baselines in table 1?
[4] Chen, A., Dohan, D.M. and So, D.R., 2023. EvoPrompting: Language Models for Code-Level Neural Architecture Search. arXiv preprint arXiv:2302.14838.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.