Summary
This paper presents a new post-training quantization strategy called "CherryQ" that identifies "cherries" --- meaning parameters that disproportionately impact the model's loss --- stores those in fp16, and quantizes the rest of the model parameters using a standard symmetric min-max quantization (with groups). It shows that this approach yields better perplexity and downstream performance numbers than existing quantization methods, on several models (Llama-3 7b/13, Vicuna 1.5 7b/13b), several tasks (c4/wiki perplexity, Hellaswag, Winogrande, ARC, TruthfulQA, GSM8K, MMLU, chat win-rate), and at several compression rates (2 bits, 3 bits, 4 bits).
Some more details:
- To identify cherries, the paper proposes using E[g_i^2] as a measure of the impact of parameter w_i on the loss, where g_i is the gradient of the loss function with respect to that parameter. This is motivated by: (1) approximating the loss around each parameter w_i using a second order Taylor expansion, assuming E[g_i] = 0 (because model has converged), and assuming the second-order derivative of L w.r.t. w_i is equal to E[g_i^2] (based on results about Fisher information matrix equaling the Hessian from [16]).
- The paper shows that selecting cherries in this way (using the above approximation of the impact of each parameter on the loss) is better than using alternative metrics based on model weights or activations.
- Once cherries are identified, the non-cherries are quantized, and then the model is fine-tuned by training the cherries in a regular manner (non-quantized gradients and updates), while training the (quantized) non-cherries using the straight-through estimator.
Strengths
- The paper demonstrates that some parameters have *much* larger impact on loss than others (at least according to their way of approximating impact), a surprising result, that nicely motivates using mixed precision quantization.
- The proposed method perform well across numerous models, tasks, and compression rates relative to baselines.
- The proposed method is pretty simple and straightforward.
Weaknesses
- It would have been useful to **directly** confirm that the proposed way of approximating impact on loss is actually a good approximation of the impact of a parameter on the loss. This was only done in indirect ways (e.g., by showing that quantizing these parameters leads to better performance than quantizing using other metrics). It should be relatively straightforward to do this by selecting a random set of model parameters (both cherries and non-cherries), perturbing them slightly (one-by-one), and passing data through the perturbed and unperturbed model and measuring the loss difference. One could then show a scatter plot (and measure spearman rank correlation) to visualize how well the approximated impact on loss (E[g_i^2]) aligns with the actual impact on loss.
- QuIP (https://arxiv.org/abs/2307.13304) and QuIP# (https://arxiv.org/abs/2402.04396) are not compared to, and more recent state-of-the-art models are not benchmarked (e.g., Llama3).
- No inference speed results are shared. It would be great to see if this method can directly lead to faster inference, especially during decoding (which is memory-bound).
Questions
- See weaknesses. Are there any obstacles to running the experiment to validate the use of E[g_i^2] as the metric for measuring impact on loss?
- How "data-dependent" are the cherries? Can this be measured?
- How does CherryQ compare with QuIP/QuIP# on Llama-3?
Limitations
Limitations are discussed, though could be addressed better with some experiments (see questions above).