AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

Summary

This paper introduces AlphaPruning, a novel framework for unstructured LLM pruning. The framework leverages HT-SR theory that utilizes the heavy-tailed shape of ESDs in layer-weight matrices to allocate layer-wise sparsity more effectively. By focusing on shape metrics rather than scale metrics, AlphaPruning demonstrates superior performance in maintaining model accuracy and reducing complexity. The method has been empirically validated across various architectures, showing significant improvements in performance and efficiency compared to existing methods, with strong generalizability to other compression techniques and architectures.

Strengths

+ This paper introduces a novel sparsity allocation method that leverages the heavy-tailed shape of ESDs in layer weight matrices, a concept previously unexplored in the literature. + The method is extensively validated with a range of LLM architectures, demonstrating significant improvements over state-of-the-art methods in terms of reducing perplexity, increasing accuracy, and achieving computational efficiency. + AlphaPruning exhibits remarkable adaptability, integrating well with various other model compression techniques and extending beyond LLMs to include large vision model architectures, proving its versatility and broad applicability.

Weaknesses

+ The novelty of the method for allocating sparsity based on layer quality (Section 3.2) is incremental, as a similar idea has previously been proposed in [1]. [1] Zhou, Yefan, et al. Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training.

Questions

+ Section 3.2 proposes a linear mapping to get sparsity from layer quality. Would other mappings, such as first compute the logarithmic of the m and then perform linear mapping, yield better outcomes? + I would like to see results on using AlphaPruning to determine layerwise sparsity for other structured pruning methods, such as OSSCAR [1]. [1] Meng, X., Ibrahim, S., Behdin, K., Hazimeh, et al. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization.

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors addressed the limitations.

Summary

This work presents AlphaPruning, which prunes weight matrices of LLM models with different layer-wise sparsity levels based on the Heavy-tailed self-regularization theory. Compared to pruning with uniform sparsity among layers, AlphaPruning alleviates performance degeneration when the sparsity level is high and fine-tuning is not applied. AlphaPruning is composable with existing pruning methods. Experiments with various LLM models, such as Llama and Llama2, demonstrate the effectiveness.

Strengths

- The presented method can be composed with other pruning methods. - The authors conducted experiments on various LLM architectures, such as Llama and Llama 2, with different baselines. In addition, they performed pruning of image classifiers, such as ConvNext and ViT. - Inference on CPU accelerates by the proposed pruning up to 3 times, depending on the sparsity.

Weaknesses

- Although they say that "AlphaPruning is theoretically driven" in L83, I could not find any theoretical justification of the method in the text. - While "Here, we provide a brief overview of HT-SR theory" (L100), it is not described, making it challenging to understand the proposed method. - Although not an expert of this domain, I think using stronger baselines would make this work better. For example, [Gale+19] demonstrated even simple magnitude pruning retains performance of ResNet-50 and Transformer even the sparsity level is set to 80%. [Gale+19] Trevor Gale, Erich Elsen, Sara Hooker "The State of Sparsity in Deep Neural Networks" arXiv 2019.

Questions

- Why is $\mu_{\mathbf{x}_i}$ is introduced in Equation (1)? It is merely used in the text except for L125. - What is the actual advantage of unstructured pruning as AlphaPruning over other acceleration techniques, such as quantization?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

- As written in Weaknesses, the baselines are quite limited.

Summary

This paper introduces Alpha Pruning, a novel approach for pruning large language models based on Heavy-Tailed Self-Regularization theory. Instead of applying a uniform pruning ratio across layers, Alpha Pruning utilizes PL_Alpha_Hill, derived from empirical spectral densities (ESDs), to assess how well-trained each layer is. It then assigns a lower pruning ratio to well-trained layers to preserve model performance. Alpha Pruning is evaluated across various LLM architectures and datasets, demonstrating superior performance compared to baseline uniform pruning and SOTA methods like OWL.

Strengths

- Alpha Pruning leverages HT-SR theory to provide a principled method for guiding layer-wise pruning decisions, contrasting with heuristic-based approaches - This pruning method has been evaluated on various large language models and exhibits robust performance, showcasing its effectiveness and generalizability. - By evaluating the importance of each layer and assigning non-uniform pruning ratios, Alpha Pruning can complement existing pruning techniques such as magnitude-based pruning and Wanda. It is also compatible with other model acceleration techniques, such as structured pruning and quantization. - Despite the challenges of unstructured pruning in achieving significant speedups compared to structured methods, Alpha Pruning still achieves noticeable efficiency gains with high pruning ratios

Weaknesses

- The paper’s explanation, particularly concerning HT-SR theory and terms used in the method section, may be challenging to grasp for readers unfamiliar with these concepts. Figure 1. a could benefit from clearer explanations. - While Alpha Pruning is compared against uniform pruning and OWL across multiple architectures, other layer-wise pruning methods prevalent in the computer vision community [1] could provide additional comparative insights. [1] Lee, Jaeho, et al. "Layer-adaptive sparsity for the magnitude-based pruning." arXiv preprint arXiv:2010.07611 (2020).

Questions

- How does the PL_Alpha_Hill metric evolve during fine-tuning? Does its value typically decrease as a block becomes better trained? - What might be the reason why a block is better trained than others? Is it possible to include PL_Alpha_Hill in the training process to train each block equally and accelerate the training? - What factors contribute to the non-linear relationship between sparsity and speedups is shown in the Table. 4? Is the maintenance of early layers a significant factor in this observation?

Rating

7

Confidence

3

Soundness

3

Presentation

2

Contribution

4

Limitations

The limitation of this work is well discussed in the Appendix.

Summary

The paper introduces AlphaPruning, a novel method for pruning large language models (LLMs) using Heavy-Tailed Self-Regularization (HT-SR) Theory. AlphaPruning uses ESDs of weight matrices to determine layerwise pruning ratios. The method demonstrates the ability to prune LLaMA-7B to 80% sparsity while maintaining reasonable perplexity.

Strengths

1. The proposed method, AlphaPruning, achieved SOTA performance across various tasks compared to OWL. 2. The experiments conducted were comprehensive and diverse.

Weaknesses

1. The motivation is not explicitly explained. It appears to apply a theoretical concept to the pruning area without providing quantitative or qualitative proof.

Questions

1. Can you explain why the property of $W^T*W$ can decide the sparsity of the layer and can be combined with different pruning metrics? 2. To my knowledge, large language models (LLMs) are generally well-trained. The article concludes that PL_Alpha_Hill can indicate whether a layer is well-trained. Are there other methods or indicators that demonstrate if these layers are not well-trained? 3. Have you compared the sparsity allocation of different layers with OWL? If so, is the sparsity allocation similar to that of OWL?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

This article includes a limitation.