ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models

Summary

The paper introduces ALPS, an optimization-based framework for one-shot pruning of large language models (LLMs). ALPS leverages an ADMM-based algorithm with operator splitting and preconditioned conjugate gradient methods to achieve improvements in sparsity and perplexity over state-of-the-art methods, particularly in high-sparsity regimes.

Strengths

**S1.** ALPS achieves substantial reductions in test perplexity and improved zero-shot benchmark performance for highly sparse models. **S2.** Provides theoretical convergence guarantees for $\ell_0$-constrained optimization problems with ADMM solver. **S3.** Implements efficient post-processing techniques via conjugate projected gradient, enhancing computational performance.

Weaknesses

**W1.** Important Reference Missing: The paper does not cite "Fast and optimal weight update for pruned large language models" by Boža, which addresses a similar problem using an ADMM-based optimization algorithm. This omission is significant as both papers share highly similar problem definitions and solutions. **W2.** Limited Novelty: ALPS closely resembles methods discussed in both "Fast and optimal pruning" and "Progressive weight pruning of deep neural networks using ADMM." The primary difference is ALPS's specific application to LLMs. However, this differentiation might not be substantial enough to establish ALPS as a novel contribution. Btw, the reference of "Progressive weight pruning of deep neural networks using ADMM." seems also missing in the paper. **W3.** Performance at High Sparsity: At very high sparsity levels, ALPS’s perplexity remains significantly higher than the dense model. This indicates that the pruned LLMs by ALPS may still perform poorly and being practically useless, although they are better than those pruned by comparison methods. **W4.** Unfair Comparison: The comparison with methods like Wanda and DSnoT, which do not involve retraining after pruning, is solely based on perplexity. This is unfair because it overlooks the overall running time per iteration/epoch. ALPS’s performance should be compared with these methods by considering both perplexity and computational efficiency to provide a more balanced evaluation. In addition, I believe if Wanda or DSnoT is combined with some re-training techniques, they can achieve a much lower perplexity as well. Reference: Boža, Vladimír. "Fast and optimal weight update for pruned large language models." arXiv preprint arXiv:2401.02938 (2024). Ye, Shaokai, et al. "Progressive weight pruning of deep neural networks using ADMM." arXiv preprint arXiv:1810.07378 (2018).

Questions

Please refer to my points listed in the weakness section. Besides those, 1. Could you provide a clearer explanation of the practical implications of your theoretical contributions? 2. I encourage the authors to report the overall running time per epoch for ALPS in comparison with Wanda and DSnoT. 3. Could you provide the perplexity of the dense models (without any pruning) in all the tables in the paper? 4. How does ALPS's speed of convergence compare with the referenced methods, and what practical benefits does the novel penalty parameter update scheme offer?

Rating

3

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have not thoroughly discussed the limitations and potential negative societal impacts. They should address the limited novelty compared to existing ADMM-based methods and discuss the risks and mitigation strategies for the misuse of more accessible powerful models.

Summary

This work presents an LLM pruning framework that formulates the problem as finding a sparse weight matrix to reconstruct the layer-wise activations. This work incorporates the operator splitting technique and preconditioned conjugate gradient methods to solve the pruning problem. Experiments demonstrate that the proposed method achieves better single-layer reconstruction error and improved performance on downstream tasks.

Strengths

This work is motivated by a clear theoretical rationale.

Weaknesses

Experiments could be more solid since many experiments are run on OPT, which is somewhat out-of-date. It would also be better to consider more challenging benchmarks, such as GSM8K or other questions that require generating a long answer.

Questions

It would be better to compare with other representative pruning methods beyond the one-shot unstructured pruning, and to explore how this method could be integrated with others. This work discusses engineering optimizations, such as incorporating PCG with GPU. Will you open-source the code or provide more efficient results from these engineering efforts?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

There is no section for limitation, but the author discusses some in the conclusion. One limitation is the setting of layer-wise activation reconstruction. Recent works, especially the activation-aware pruning methods, reported that activations are not equally important.

Summary

This paper introduces ALPS, a novel optimization-based framework for one-shot unstructured pruning of LLMs. The key contributions are: - Formulating LLM pruning as an l0-constrained optimization problem solved using operator splitting (ADMM). - A penalty parameter update scheme to accelerate convergence. - A post-processing step using preconditioned conjugate gradient (PCG) to refine weights. - Theoretical convergence guarantees for the proposed algorithm. The authors demonstrate that ALPS outperforms state-of-the-art pruning methods, especially at high sparsity levels, on various LLMs including OPT and LLaMA models.

Strengths

1. The proposed method directly addresses the pruning problem without relying on heuristics. 2. The ADMM algorithm introduced in the paper comes with theoretical convergence guarantees, adding reliability to the approach. 3. The paper includes comprehensive experiments on large-scale models (up to 30B parameters), demonstrating consistent improvements over existing methods.

Weaknesses

1. It would be beneficial to include evaluations on more recent models such as LLaMA-3, instruction-tuned models, and extremely large models (70B+ parameters) to demonstrate the method's applicability to the latest advancements in the field. 2. The current evaluation tasks are relatively limited. Expanding the evaluation to include knowledge-intensive tasks (e.g., MMLU) and reasoning-based tasks (e.g., GSM8K and HumanEval) would provide a more comprehensive assessment of the method's effectiveness. 3. The current focus is primarily on N:M sparsity patterns. It would be great to explore structured pruning, which could lead to more significant improvements in inference speed, making the approach more versatile and practical for real-world applications.

Questions

1. How does the runtime of ALPS compare to other methods, especially for very large models? 2. Can (how can) ALPS be extended to handle other types (structured sparsity) of structured sparsity patterns beyond N:M?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed some limitations of their work e.g. extending ALPS to incorporate structured pruning constraints and quantization.

Response to rebuttals

Thank you for your detailed responses. The paper is in good shape, but still have some concerns remain unsolved: 1. I understand the resource constraints, but some results on models like LLaMA-3 or a discussion on scaling to 70B+ parameters would be valuable. OPT-30B is relatively old, and to my knowledge, it's sometimes easier to prune them than more recent 'overtrained' models, e.g. Llama-3 and Qwen 1.5 / 2. 2. Including knowledge-intensive (e.g., MMLU) and reasoning-based tasks (e.g., GSM8K) would strengthen the paper. 3. Theoretical analysis of extension is promising, and empirical results or case studies on these structured sparsity patterns would provide practical insights.