ESPACE: Dimensionality Reduction of Activations for Model Compression

Summary

The proposed method applies a PCA-inspired method to the activations of LLMs for model compression.

Strengths

- While most papers focus on quantization, pruning, or weight decomposition, the proposed approach goes in an interesting direction. - The paper is easy to follow and the proposed method is simple. - The studied problem is relevant. - The limitations of the method are described. - The experiments contain multiple ablations to verify the effectiveness of the method.

Weaknesses

- Compression is never well-defined. Is it compression ratio w.r.t. W? What if you do not compress all weights of the model (line 112)? - Comparison with related baselines is missing. I understand your work is orthogonal to pruning and quantization methods, but it would make sense to compare with methods that use weight decomposition such as ASVD or the ones you mention in the paragraph starting on Line 41. - The practical benefit of the method can be better emphasized. - The quality of the figures can be improved, e.g., the text is grainy and Fig. 2 is too small to read. - Line 192, i==j -> i=j

Questions

- The paper could be made much stronger if you could show some practical benefits in terms of, e.g., inference time, VRAM requirement, or storage requirements. Right now you show compression ratios, but I believe more concrete numbers can make the method much more appealing.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are well addressed.

Summary

The paper introduces ESPACE (Eigen Static Principal Activation Component Estimation), a technique for compressing large language models (LLMs) by focusing on the dimensionality reduction of activation tensors rather than the traditional weight-centric tensor decomposition. The method involves projecting activation tensors onto a pre-calibrated set of principal components, which reduces the dimensionality of activations and results in weight compression during inference through matrix multiplication associativity.

Strengths

The paper introduces ESPACE for compressing LLMs through activation-centric dimensionality reduction. This differs from traditional weight-centric tensor decomposition techniques, which adds novelty to the research. The authors provide a solid theoretical foundation for their approach, including the derivation of optimal constructions for projection matrices to minimize mean squared error and forward propagated noise metrics. ESPACE can achieve up to 50% compression of models with a small increase in perplexity and a practical reduction in inference (GEMM) latency.

Weaknesses

To me ESPACE is simply PCA projection (with a reinvented name) of activation tensor. Can the authors defend a bit? The paper suggests that ESPACE can be used in conjunction with other compression techniques like pruning and quantization, as well as non-compressive acceleration methods such as speculative decoding. However, it does not provide empirical results for these combinations, which makes the claim unconvincing. Although the authors claim that ESPACE is orthogonal to other compression techniques, it would strengthen the paper to include comparisons with more baselines, such as pruning, to better demonstrate the significance of the proposed method. E.g., when compressed respectively & individually with ESPACE, pruning and quantization to the same model size, how do their perplexities compare to each other? You can demonstrate that on small models like llama-58M or GPT2-97MB. The necessity for a calibration phase that involves forward-passing multiple batches of data to estimate activation auto-correlation matrices might not scale well with larger datasets or more complex models, potentially limiting the method’s applicability in diverse settings.

Questions

How does the calibration set impact the final performance? Can you provide a sensitivity profile vs layers in the Transformer model? I.e., how would the local ranks of projection in individual layers affect the global performance? (with the belief that a deep compression in early layers can hamper the output even higher ranks are used for later layers.) A portfolio of layers’ importance would be crucial for the compression strategy. How does the projection influence the well-known outlier issue in activations and sometimes in the key values, which is observed in lots of LLM works?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The calibration set plays a crucial role in the quality of the principal components for projection, this needs to be analysed deeply to warrant good results.

Summary

The paper introduces a novel technique for compressing large language models (LLMs) by reducing the dimensionality of activation tensors. The ESPACE method differs from traditional weight-centric compression approaches by focusing on activation tensors instead. ESPACE projects these activations onto a pre-calibrated set of principal components, preserving expressivity during retraining while enabling weight compression at inference through associative matrix multiplication.

Strengths

1. The paper introduces a novel approach to compressing large language models by focusing on the dimensionality reduction of activation tensors rather than the weights themselves. This method, termed ESPACE (Eigen Static Principal Activation Component Estimation), represents a creative combination of principles from tensor decomposition and matrix multiplication associativity. 2. This paper is highly technical and presents a robust foundation for constructing optimal projection matrices that minimize the mean squared error and forward-propagated noise metrics. 3. The paper is well-written and organized logically. The authors effectively communicate complex ideas, making the novel approach accessible to readers.

Weaknesses

1. More direct comparisons with other state-of-the-art activation-based and tensor decomposition methods could provide a clearer benchmark of ESPACE's effectiveness. 2. The paper could benefit from a more thorough discussion of the trade-offs in selecting compression settings and projection matrices for different application scenarios. 3. Including a robust error analysis would help identify potential weaknesses or failure modes of ESPACE, enhancing understanding of its limitations. 4. The study focuses primarily on GPT3 and Llama models, potentially limiting the generalizability of the results to other architectures. 5. The paper does not extensively explore ESPACE's performance across various inference tasks that differ in length or complexity, which could provide a deeper understanding of its practical implications.

Questions

1. How does ESPACE perform on different types of inference workloads, particularly those requiring higher precision or longer inference times? Additional details on these scenarios would help assess the method’s practical applicability in diverse real-world settings. 2. Could you elaborate on the decision-making process for selecting specific compression settings and projection matrices? A detailed discussion on how to balance compression rate and model performance in various contexts would be highly valuable for practitioners. 3. Have you conducted any tests to evaluate ESPACE’s robustness under unusual conditions or edge cases? Sharing any findings or future plans for such analyses would provide a clearer understanding of the method’s limitations and potential areas for improvement.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. Include evaluations or discuss plans for testing ESPACE on various model architectures to better understand its generalizability. 2. Discuss potential challenges and solutions for deploying ESPACE in real-world systems, focusing on real-time performance and integration with existing workflows.

Summary

This paper introduces ESPACE, a method that reduces activation in models via tensor decomposition, thereby aiding in the reduction of model size and GEMM latency.

Strengths

- The paper is easy to follow; - The method is simple but efficient.

Weaknesses

- Wikitext-103 is a specific type of language dataset that focuses on knowledge-based content. However, models like GPT-3 and Llama2 are capable of handling a broader range of tasks, including math and question-answering. Therefore, it is necessary to evaluate perplexity (PPL) on a more diverse set of data. - ESPACE requires ongoing training. It is important to understand the duration required for ESPACE's continual training process. - In Table 1 and Table 2, since ESPACE is continually trained with more data than the original model, a direct comparison may not be entirely fair. It would be more convincing to compare the original model after it has been trained on the same amount of data as ESPACE. - For generative models such as GPT-3 and Llama2 discussed in this paper, there is a need for more comprehensive performance validation, including metrics like mmlu and math benchmarks.

Questions

Please see the weaknesses.

Rating

5

Confidence

4

Soundness

1

Presentation

2

Contribution

3

Limitations

The authors address the limitations.

Thanks for your reply!

I am happy with the authors' reply and the additional experimental results. I tend to keep my score.