Search for Efficient Large Language Models

Summary

This paper introduces a neural architecture search method for Large Language Models (LLMs) comprising three steps: inheriting the most salient weights from the original model to form the initial sub-network, using an evolutionary algorithm to search for the optimal sub-network, and reconstructing the original network’s output with calibration samples.

Strengths

Overall, I like the application of evolutionary algorithm to pruning problem. 1. The method overcomes the limitations of uniform sparsity across all layers—a common but sub-optimal constraint in previous structured pruning methods. 2. It avoids reliance on back-propagation, enhancing its applicability to larger models within a constrained memory budget.

Weaknesses

Here are some technical concerns: 1. The candidate evaluation is conducted based on perplexity on WikiText2. This is one metric on one task, then the searching algorithm might be biased towards the task used for candidate evaluation, resulting in a model favoring one task (e.g., language modeling) over others. How to ensure the performance of the searched model on other tasks? 2. Based on my experience, using perplexity as a metric for selecting pruning candidates can result in good performance on WikiText but sub-optimal performance on the MMLU dataset. It would be great if authors could validate their pruned (or searched) model on MMLU. 3. Table 5. The performances on QA tasks should also be presented. If the space is not enough, authors can put them in the Appendix and link it in the main text. 4. Figure 7. Tokens/s is not a very rigorous measurement for speed. The generated tokens per second can depend on many factors, e.g., floating point precision, KV Cache, flash attention, batch size, etc. I suggest authors use other metrics such as MACs or at least provide more specifications.

Questions

In addition, here are some points about writing: 5. The term "inheriting ratio" needs to be rigorously defined in an early section. In addition, authors would better add a sentence describing the difference between "inheriting ratio" and "sparsity" / "pruning ratio". In my understanding, "inheriting ratio" = 1 - "sparsity". 6. Figure 2 Left: it is better to add an annotation showing which axis is the layer index. I assume it is the horizontal axis. 7. Section 3.2, Lines 139-149: The writing of this part needs to be improved. It gives me a lot of confusion when I read this part. (i) Please define what $M$ and $P$ are in Line 140. (ii) Are the masks smooth or binary? If binary then $\{0, 1\}$ should be used instead of $\mathbb{R}$ in Line 140. (iii) Line 141, why do different layers of Attn / MLP share the same mask? If so, would the sparsity of each layer be the same? However, according to Figure 1, the sparsity should be different across layers. Also, in Line 146, authors "set the same inheriting ratios for the masks in all building blocks". So, these claims and figures are very chaotic. Additionally, what does "align the internal computations" mean? (iv) Line 144, Are Eq. (2) and (3) jointly optimized (e.g., take the sum) or separately? Are the masks of different layers jointly optimized or separately? 8. Section 3.3.1, Line 165: $\gamma$ should be rigorously defined.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

1. Section 5: The limitations provided by the authors are too trivial. It is obvious that larger models require more time for pruning/searching. Please add more in-depth discussions on limitations. Authors can refer to the points listed in the "Weaknesses" section for more substantial issues that need addressing.

Summary

This paper introduces an architecture search method based on mask mutation and candidate evaluation to find a subnet with better performance in the LLM. An evolution-based algorithm is applied to globally search the subset with a special initialization from the evaluation of parameter importance. After the search from LLM, a reformation algorithm is proposed to rectify the weights in the inherited LLM. Experimental results show that the compressed LLM can achieve better results compared to the baselines.

Strengths

1. A novel method for searching a subnet in the LLM. The search of subnet in LLM is not well-explored, and it would benefit the community. 2. The overall approach is reasonable and sufficient. The entire method’s pipeline is very clear, and the design for each component is reasonable and complete. 3. The experimental results are comprehensive and show its benefits compared to the baselines (SliceGPT and LLM-Pruner). A large and steady improvement is observed. 4. The paper is well-organized and easy to follow.

Weaknesses

1. My main concern is about the extra cost of searching the subnetworks. Since it still takes around 5 hours to search for the architecture, it introduces extra cost for pruning the model. What if we take these 5 hours to post-train the compressed model? Would the results be better or worse than the subnetwork search from the LLMs? 2. Since the training-free structured pruning is not a very useful setting, post-training the LLM with an acceptable amount of resources is more realistic. Thus, compared to just pruning and searching the mode, a more realistic setting is also to train the pruned model. And I’m not sure whether the searched sub-LLM would be better than the pruned one. Some previous work to prune CNNs suggest that with sufficient training, magnitude pruning would be a better choice than any manually designed metrics. Thus, it would be better to examine the performance of the compressed LLM with post-training.

Questions

1. For Mask Sharing: why does the same module in different modules share the same mask (Line 141)? In line 147 - 149, from my understanding, the mask is not the same for blocks in different layers. Using different masks for different layers seems to still have the aligned internal computations. I’m not sure if I understand correctly in this part. 2. What about the results if you post-train the compressed models? What are the results if compared with LLM-Pruner?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper introduces a new model architecture search method for LLMs that does not require additional training. The proposed architectures have structural sparsity and reach better performance than SOTA pruning baselines.

Strengths

- The proposed method does not require additional training. - The output model has structural sparsity, which is useful for hardware. - The paper is clear and well-written.

Weaknesses

- Why isn't there a comparison to SparseGPT?

Questions

Why isn't there a comparison to SparseGPT?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discuss limitations.

Summary

This paper proposes a technique for searching for efficient LLM subnets for fast inference, while still attaining strong performance. The proposed technique involve a training-free pruning stage based on a genetic algorithm, followed by a ``weight rectification'' stage that improves the resulting subnet. Empirically, the proposed method outperforms existing LLM pruning baselines.

Strengths

- Empirically, the proposed pruning technique appears to outperform a handful of existing pruning baselines, as shown in Figure 1 and Table 2. - The proposed technique appears to be general enough to work for a wide variety of existing LLMs, which appears to be a s hortcoming of existing LLM pruning techniques. - The proposed method is fairly simple in that it combines LLM pruning with a genetic algorithm for search.

Weaknesses

- From the abstract, it is unclear what the actual proposed method does beyond standard neural network pruning techniques. - The term ``reformation algorithm'' is used throughout the paper, and it is unclear what this means until much later in t he text. The intro and abstract should clarify this. - The main evaluation is done in terms of sparsity levels, which makes sense for this particular type of search space, how ever, the authors should also empirically validate the efficiency of the search process compared to existing pruning techn iques. E.g., how do these pruning techniques compare, subject to the same budget? Moreover, it would be interesting to see how the proposed method performs as a function of the search budget. - The authors should compare to other techniques beyond pruning, such as quantization which may achieve similar inference speed and memory improvements with better overall performance. In general, it seems that LLM pruning techniques take a sub stantial performance hit even at fairly modest sparsity levels (I'm mainly referring to Table 7).

Questions

- The authors mention in the limitations that the search cost scales with the model size. Can the authors characterize thi s a bit more? Do other pruning methods suffer from the same problem, and can the authors comment on what this scaling curv e actually looks like? - Why was the search budget set to 5 hours? Have the authors explored this choice empirically, and does this apply to all of the models and data settings that were evaluated?

Rating

4

Confidence

2

Soundness

2

Presentation

1

Contribution

2

Limitations

The authors adequately address limitations in the conclusion.

Side comment

Dear authors, Thank you for your response. Before we launch further discussions, I would like to ask an important question: Did you submit any global response, or any one-page pdf containng tables / figures? It is not visible to me at the moment. If yes, please contact the AC or PC to fix the systematic error. To be clear, this comment will not affect my ratings. I just want to make sure that I don't miss any further experiment results or responses from you.

Global rebuttal invisible to reviewers

Hi ACs, We have submitted one global rebuttal which includes important explanations and additional results following the instructions of one global rebuttal for the paper. However, we find that the global rebuttal is only visible to program chairs and authors. The reviewers are not able to read the global rebuttals. The reviewers can read the specific rebuttals corresponding to their reviews. It seems to be a technical issue for the visibility of the global rebuttal. Can you help us address this issue? Thanks Authors

response to rebuttal

Thank you for the response. I maintain my score for acceptance.