Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models

Summary

This paper proposes an application of the mixed precision quantization technique to the singular vectors of the delta weights, which is encountered when serving multiple aligned LLMs.

Strengths

Please see the “Questions” section.

Weaknesses

Please see the “Questions” section.

Questions

My review is as follows: 1) As a strength, this paper is well-written and makes it easy for the reader to follow the ideas. 2) The proposed method is a somewhat straightforward extension of mixed precision quantization to the singular vectors of the delta weights. The mixed precision quantization technique is well-known and commonly used by practitioners. I view the contribution of this paper as an extension of that. Because of that, I am not sure if the contributions of the paper could be considered novel. Having said that, it is good to see that the method performs well in multiple scenarios. 3) I understand that studying the compression of delta weights is important. In practice, the base model’s weights need to be quantized/compressed too for real-world situations. I don’t think the compression of base model weights and delta weights are completely orthogonal. I wonder what the authors think on the interdependency of the compression of base model weights and delta weights. In my opinion, it would improve the paper to include some results/discussion on this. 4) I’m wondering why the authors haven’t considered a more thorough search algorithm in Section 5.1 (or if they did, it would be great to include a discussion on it). Minor: 5) Typo in Table 1: “Aligend” 6) Typo in Table 6: “HuamnEval”

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes.

Summary

In the context of SVD compression of delta weight, the paper employs higher bitwidth for singular vectors corresponding to larger singular values. The available bitwidth 8, 3, 2 are empirically chosen. Once the bitwidths are assigned to the singular vectors, the vectors are group-wise quantized by GPTQ.

Strengths

The paper is generally well-written and the experiments are very thorough. The combination of delta-compression and mixed-precision quantization appears to be novel. The method is straightforward and well-motivated.

Weaknesses

Something is wrong with the latex encoding of this pdf. Sections are not detected properly by pdf readers, and there are odd reference errors such as the ones on line 26 and 30. There are three steps of optimization being done: the number of largest singular vectors chosen, the bitwidth assigned to the singular vectors, and the quantization algoritm performed on the singular vectors. In the paper, the three steps are being done sequentially, leading to arbitrary, empirically-driven and possibly suboptimal decisions.

Questions

Why is there an advantage to SVD the delta weights as opposed to SVD the aligned model weights directly?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper proposes an improved way to apply delta-compression for aligned language models (compact representations of the difference in weights between the pretrained and finetuned language models), which is just as effective w.r.t compression as the most extreme existing binary quanitzation strategy (BitDelta), but resolving many existing quality issues w.r.t. math and code tasks over previous methods. The key insight is that it is possible to combine the best of both worlds between low-rank methods (selecting the top singular values) and quantization methods (e.g. BitDelta), and represent more singular values by using decreasing precision for decreasing singular values: a mixed precision technique overall. The authors call this technique Delta-CoMe. When comparing low-rank, BitDelta, and Delta-CoMe at a fixed compression rate (equal to the binarization / BitDelta rate, which is quite aggressive), Delta-CoMe consistently outperforms all baselines and is close to matching performance with the full precision aligned model. Experiments take place over a diverse set of tasks (math, code, QA, safety, multimodal) as well as a diverse set of models at 7B and 13B scales and aligned versions (Llama 2, WizardMath, MagicCoders, Llama-2-Chat, LLAVA v1.5, OpenChat, Llama-3-8b-Instruct). The authors also provide quality comparisons between delta-compression methods and delta-tuning (i.e. LoRA), demonstrating that Delta-CoMe outperforms LoRA w.r.t quality.

Strengths

1. While neither low-rank or mixed precision quantization techniques are particularly novel, the combination of the two for delta-compression is novel and clever in the way that the framework smoothly models the trade-off between representing everything with low (binary) precision, and representing the most important features with full precision. Although this is not explored deeply in this paper (the authors recognize that this is a proof of concept of the effectiveness rather than an optimized solution), this tradeoff could be tuned to suit a variety of cases. This idea could also potentially be applied in other areas, such as PEFT or general mixed-precision work, as previous works used quite different heuristics to select which weights to have mixed precision on. 2. The results presented are significant. The approach significantly outperforms the baselines in all aspects (math, code, chat, multimodal), with the biggest improvements in math and code. The approach is the only one to come close to matching the non-compressed aligned model overall over all capabilites. This will be of interest to anyone interested in delta-compression literature or even PEFT techniques. It is the first work to apply mixed-precision to delta weights. 3. The experiments was done on a wide variety of models (math specialized tunings, code specialized tunings, chat tunings, multimodal tunings) and settings (7/8B, 13B, over base models like Llama2, Llama3 and Mistral) enough to demonstrate confidence in the ability of the technique. The authors also provide nice additional analysis of quantization error, and comparison in quality against vanilla LoRA as additional evidence for their approach. 4. The paper is overall well structured and well written, which makes the problem easy to understand and the author's intuition and investigation relatively easy to follow. There are some improvements that could be made (below), but it is nice overall.

Weaknesses

1. In Table 4, it does seem like on the larger 13B models WizardMath models, the performance recovery in GSM8k and MATH is not as strong as on 7 or 8B scale models. It does bring into question whether there are limitations to this approach not discussed in this work that may limit general adoptability. Whether that means not working as well on larger scales, or for the certain type of post-training happening in WizardMath. 1. Although I understand the author's motivation of using greedy decoding to encourage reproducibility and decrease noise in the evaluations, many models especially on chat tasks use sampling based-decoding, and having results here would improve the overall robustness of the experiments to verify that the techniques still work well. 1. Presentation a couple points in the paper could be improved : a. While the authors do note that the greedy search strategy to allocate mixed-precision to the singular values presented in Section 5.1 represents a proof of concept to show that even such an unoptimized approach can achieve good results, it would be valuable to understand how some of the design choices here were informed. For example, in the 3-precision setting, the range of singular vectors for the 2nd precision is set to be between 2 and 34, which feels arbitrary. What is the intuition behind this or were there ablations ran here? Being that Section 5.1 is the core description of the mixed-precision strategy, more detail and clarity here would be appreciated. b. (Minor) The introduction could be cleaned up a bit, with lines 21-36 being oddly detailed. The claim on line 44-46 to motivate the work is also not super well founded -- there are models on LMSys showing general ability with ~20B parameters, not too much bigger than those considered in this work.

Questions

1. What is the intuition behind choosing the singular vector bands for the different precisions the way that you do in Section 5.1? 1. Do you think the ideas here could apply to mixed-precision approaches more generally? 1. Why is the approach named Delta-CoMe? It appears to stand for delta compression method which seems very generic? And has nothing to do with the details of the approach. 1. Are there any practical limitations e.g. hardware with mixed-precision delta-compression not covered in this paper?

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors do clearly point out that a major limitation of the work is the lack of optimization for how the mixed precision is assigned. This makes the approach more of a proof of concept that even a naive solution works, rather than an optimized method. This is perfectly fine. It would be valuable to understand though whether theres actually limitations in deploying this type of approach on actual hardware with multi-tenant hosting. It would also be interesting to understand the boundaries to when this approach works or does not work, e.g. at what compression ratio would it break, whether theres a model scale too small or too large, etc.

Summary

This paper introduces Delta-CoMe, a delta compression method for LLM. It proposes to combine low-rank compression and low-bit compression together to achieve better performance. Specifically, it applies mixed-precision quantization for different singular vectors based on their singular values. The experimental results demonstrate good performance of the proposed method.

Strengths

1. The proposed method is easy and straight to implement for other LLMs, based on existing open-source tools. 2. The motivation and the proposed idea is clear and supported by the experiment results.

Weaknesses

1. The bit-width in mixed-precision seems to be a multi-objective optimization problem, and the authors adopt a greedy search in this paper. In Table 2, different settings vary a lot in performance. This suggests that it requires to decide the bit-width carefully, making the method not easy to generalize well to other tasks/models. Performance may not be guaranteed before doing a search. 2. The value of $r_{begin}$ and $r_{end}$ seem to be set intentionally and empirically without explanation or study. This would also impact the generalization of the method.

Questions

1. In equation 5, why the quantization of $V$ is $Quant_k(V, X)$ without $U$ and $\Sigma$ as in the quantization of $U$? 2. In Section 5.1, how are $r_{begin}$ and $r_{end}$ set to those values?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Performance on hardwares

To accelerate the inference of our proposed method, we implemented a computation kernel using Triton, which integrates the de-quantization process with matrix multiplication. This kernel supports the multiplication of matrices with different bit-widths. By using our customized Triton kernal, we can significantly improve the forward speed of Delta-CoMe: Ablation on batch, we set sequence length to 128, our method achieves 3x speedup than Pytorch. | bsz | Delta-CoMe w/ PyTorch (ms) | Our Customized Triton Kernal (ms)| |-----|-------------|----------| | 2 | 0.5995 | 0.1488 | | 4 | 0.6024 | 0.1995 | | 6 | 0.6136 | 0.2134 | | 8 | 0.6656 | 0.2561 | Ablation on hidden_size, we set batch 8. | hidden_size | Delta-CoMe w/ PyTorch (ms) | Our Customized Triton Kernal (ms) | |-------------|--------------|----------| | 1024 | 0.5369 | 0.1206 | | 2048 | 0.5586 | 0.1654 | | 3072 | 0.6011 | 0.2046 | | 4096 | 0.6656 | 0.2561 | | 5120 | 0.6897 | 0.2788 | Further, Han Guo et al. and Jianyu et al. have implemented more advanced kernels. We can draw on their methods to achieve higher acceleration ratios. To further demonstrate the hardware advantages of our method, in addition to inference speed, we have also provided memory usage in the table below. | Num. deployed models | Original Storage | Our Storage | |-----------------|------------------|-------------| | 2 | 26.67G | 15.39G | | 4 | 52.24G | 17.02G | | 8 | OOM | 20.28G | | 16 | OOM | 26.79G | | 32 | OOM | 39.84G | | 64 | OOM | 66.68G | Our method can save GPU memory significantly.