DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models

Summary

This paper concerns about the dimension dependence of hidden states across different layers. It have been argued that previous studies with dimension dependence would be inefficient due to additional parameters or architectural constraints. This work breaks the dimension dependence by introducing indexing operations to previous methods. Although simple, the proposed method is shown to be more effective than baselines due to permitted asymmetry across layers.

Strengths

1. It is for the first time highlighted that the dimension dependence may affect the performance of pruning. 2. The proposed method to alleviate the dimension dependence is rather simple and effective.

Weaknesses

1. Although the proposed method allows asymmetric dimensions in module weights across layers, it also requires the dimensions of hidden states from different layers maintain the same as that of the non-pruned model. This would restrict the use in scenarios where the dimension of the output hidden states is critically important (e.g., embedding for retrieval). 2. The introduced indexing operation could be very time-consuming. Unexpectedly, the latency introduced by the indexing may outweight the latency reduced by pruning. And it is not discussed whether the pruned model is faster than the non-pruned one. I would recommend including such experiments in proper positions.

Questions

N/A

Rating

5

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Summary

DISP-LLM introduces dimension-independent structural pruning for LLMs, breaking structural dependencies to allow flexible pruning across layers and dimensions. It uses a hypernetwork for pruning decisions and claims superior performance over existing methods on various LLMs without weight updates or additional parameters.

Strengths

- Novel approach to structural pruning in LLMs - Comprehensive evaluation across multiple LLM architectures - Theoretical foundation provided - Achieves good performance without weight updates or extra parameters

Weaknesses

- Limited novelty beyond combining existing concepts - Inconsistent performance gains across models/tasks - Insufficient ablation studies and hyperparameter sensitivity analysis - Lack of detailed computational efficiency comparisons - Inadequate error analysis and failure case exploration - Overstated claims based on limited experiments - Insufficient discussion of pruning ratio vs. performance trade-offs

Questions

- How sensitive is DISP-LLM to hypernetwork architecture and hyperparameters? - Can you provide detailed computational cost comparisons with baselines? - How does it perform on fine-tuning tasks? - Have you explored progressive pruning approaches?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

This paper proposes a novel structural pruning method for LLMs, primarily based on SliceGPT. It characterised in (i) removing structural dependence by facilitating each block to possess varying widths along its input and output dimensions (ii) no need for introducing addition paramaters like sliceGPT. The empirical studies show that it demonstrates superior performance compared to state-of-the-art structural pruning techniques for LLMs across a range of models, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.

Strengths

1. This paper is easy-to-read and topic is very important to efficient Machine Learning. 2. The proposed method balances the task performance and efficient, which are both beneficial to its practical and wide application. 3. The empirical results show its promising in model compressing across four models.

Weaknesses

1. More comprehensive literature review about model compressing, like sparse mechanism. 2. Any case study for qualitative analysis to the pruned model? It is possible to conduct some interpretability analysis showing where the difference between sliceGPT and the proposed method along with a specific input example.

Questions

1. When training with the pruning objectives all the layers are fine-tuned or only selected layers? 2. Any insights from Figure 6, different model width as layer changes. If the observations aligned with existing research results?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Pay attention to the generated sequence after model prune because the prediction changes are unpredictable.

Summary

Structural pruning is a method to prune the weights of large language models (LLMs) while keeping their original performance as much as possible it can. However, structural pruning has limitations caused by depending on the structure, like the residual connection of LLMs. In this work, the authors propose a new pruning method, dimension-independent structural pruning for LLMs (DISP-LLM), that can ignore structural dependence in pruning. This characteristic of DISP-LLM makes it possible to prune their weights more flexibly and provides new pruning ways using different feature maps between blocks and varying pruning dimensions of the input and output of the block. However, just pruning the dimensions and ignoring residual connections causes inconsistency between the input and output of a block. To solve this problem, the authors apply transformation matrices to each residual connection to absorb these discrepancies. Furthermore, pruned indices are optimized through training. Experimental results on the language modeling dataset WikiText-2 and the zero-shot task datasets PIQA, WinoGrande, HellaSwag, ARC-e, and ARC-c show that DISP-LLM tends to outperform conventional approaches in pruning LLMs, OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.

Strengths

- The proposed dimension-independent structural pruning for LLMs (DISP-LLM) can drastically prune dimensions due to its flexibility. - Adding transformation matrices to each residual connection to maintain input and output consistency in each transformer is novel - In DISP-LLM, pruned dimensions can be optimized through training. - Experiments are comprehensive and cover various kinds of LLMs. - DISP-LLM can achieve comparable or better perplexities to conventional pruning approaches that require updating the original weights by training. - In many cases, DISP-LLM outperforms conventional pruning approaches in zero-shot tasks even though some of them update the original weights by training. - The authors report the actual costs for pruning models and show the cost efficiency of DISP-LLM.

Weaknesses

- Even though DISP-LLM does not update the original weights of LLMs, training is required. - DISP-LLM requires hyper-tuning for $\lambda$.

Questions

- How did you decide to set $\lambda$ to 6? - How are the performances sensible to the changes of $\lambda$? - As you state in the paper, DISP-LLM is as efficient as LoRA. If so, can applying LoRA weights only to the kept dimensions be a baseline?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

DISP-LLM requires training and hyper-parameter tuning for $\lambda$ that controls mixing ratios of losses.

Regarding LoRA

Thank you for answering my questions! The response has resolved some of my concerns. Regarding the use of LoRA, one of the frequently used approaches is QLoRA [1], which uses LoRA on quantized frozen weights. Even though QLoRA does not prune the model weight, it accomplishes producing lightweight models with fine-tuned information. Therefore, considering the quantization in the comparison is along with the realistic situation. As seen in the general response, the author seems to notice the necessity of investigating the use of quantization in the proposed method. Currently, this direction has not been decided in the paper. However, if you can clarify the difference between the proposed method, quantization, and QLoRA in terms of their benefits and still claim the superiority of your approach to other methods, I can raise my score. [1] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." Advances in Neural Information Processing Systems 36 (2024).