Library

decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points

Quantization emerges as one of the most promising compression technologies for deploying efficient large models for various real time application in recent years. Considering that the storage and IO of weights take up the vast majority of the overhead inside a large model, weight only quantization can lead to large gains. However, existing quantization schemes suffer from significant accuracy degradation at very low bits, or require some additional computational overhead when deployed, making it difficult to be applied to large-scale applications in industry. In this paper, we propose decoupleQ, achieving a substantial increase in model accuracy, especially at very low bits. decoupleQ abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, thus transforming the quantization problem into a traditional mathematical optimization problem with constraints, which is then solved alternatively by off-the-shelf optimization methods. Quantization via decoupleQ is linear and uniform, making it hardware-friendlier than non-uniform counterpart, and enabling the idea to be migrated to high-bit quantization to enhance its robustness. Our method has achieved well on-line accuracy near fp16/bf16 on the 2-bit quantization of large speech models in ByteDance. The code is available at https://github.com/bytedance/decoupleQ

cs.LG
View graphSource

Paper

Similar papers

FlipGuard: Defending Preference Alignment against Update Regression with Constrained OptimizationMingye Zhu, Yi Liu, Quan Wang et al.2024 · Conference on Empirical Methods in Natural Language Processing · 2 citations In Library
Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMsQitao Tan, Xiaoying Song, Arman Akbari et al.2026 · arXiv.org · 1 citation In Library
TamperBench: A Systematic Benchmark for Fine-Tuning Attack Resistance in Safety-Aligned Open-Weight Language ModelsHamda Aden2026 · Zenodo (CERN European Organization for Nuclear Research) In Library
TamperBench: A Systematic Benchmark for Fine-Tuning Attack Resistance in Safety-Aligned Open-Weight Language ModelsHamda Aden2026 · Zenodo (CERN European Organization for Nuclear Research) In Library
Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time AlignmentK. Krishna, J. Y. Cheng, Charlie Maalouf et al.2025 · arXiv.org · 4 citations In Library
Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal DirectionYuanbo Xie, Yingjie Zhang, Tianyun Liu et al.2025 · arXiv (Cornell University) · 6 citations In Library
Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMsWaiman Si, Mingjie Li, Michael Backes et al.2026 · Annual Meeting of the Association for Computational Linguistics · 0 citations In Library
GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-TuningZhouxiang Fang, Jiawei Zhou, Hanjie Chen2026 · arXiv.org · 0 citations In Library
FinHarmBench: Financial Jailbreak Benchmark and Unsupervised Safety Fine-Tuning via Refusal Steering DistillationYubin Choi, Yujin Yang, Subin Kim et al.2026 In Library
On the Safety of Graph Representation LearningXiaoguang Guo, Zehong Wang, Ziming Li et al.2026 · arXiv.org · 0 citations In Library

Peer review

RejectNeurIPS 2024 · via OpenReview
Reviewer G89P5/10 · confidence 4/52024-07-07

Summary

The paper introduces decoupleQ, a novel method that decouples model parameters into integer and floating-point parts. This approach transforms the quantization problem into a mathematical constrained optimization problem, avoiding the limitations of traditional heuristic quantization methods. DecoupleQ achieves a significant improvement over existing methods in LLM., especially at extreme low bits (2-bit) and also release the W2A16 CUDA kernel.

Strengths

1. DecoupleQ eliminates the need for ad-hoc techniques to handle outliers and sensitive channels, focusing solely on optimizing model accuracy under extreme low-bit quantization. 2. DecoupleQ achieves a notable advancement over existing methods in LLM, particularly at extremely low bit. And the W2A16 CUDA kernel has been released. 3. DecoupleQ approach can be readily extended to supervised fine-tuning (SFT) to enhance model accuracy, or adapted for downstream sub-tasks.

Weaknesses

1. Please correct me if I am wrong. It seems that decoupleQ combines several existing approaches. Specifically, it uses Adaround to get the integer part in ResNets and GPTQ to get the integer part in LLMs. Additionally, it integrates PTQ and QAT by applying PTQ to the integer part while using supervised training for the floating-point part. 2. Regarding your point from lines 58-61, I believe GPTQ clearly outlines how to calculate scale and zero point in their code. Moreover, GPTQ can be seen as a constrained optimization problem, where the constraints align with yours: each integer weight is confined within [$\alpha$, $\beta$], which is a default constraint in GPTQ. 3. Further experiments on LLMs are essential. For example, evaluating decoupleQ's performance in multi-task settings and within the LlaMa 3 family would provide valuable insights. 4. Could you provide more ablation studies in the second stage, such as experiments without training norm layers? 5. There is a typo in line 125. The first letter of 'decoupleQ' should be capitalized.

Questions

Please refer to the weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the weaknesses.

Authorsrebuttal2024-08-12

Thank you very much for your reply. We are sorry that we did not explain it clearly in our first reply. It is a pleasure to continue discussing with you. We have urgently conducted some ablation studies, and more experimental results are on the way. In this experiment, we choose to freeze or not freeze the layernom in LLama when training $(s,z)$ in the Block-wise minimization, and other settings are consistent with those in the original paper. Each element in the table represents (PPL of wiki-2, PPL of C4). | |1-7B |2-7B |1-13B | 1-30B | 1-65B | :---:|:---:|:---:|:---:|:---:|:---: | Train LN | (9.49, 11.41) |(9.74, 11.83) |(7.86, 9.86) | (6.37, 8.51) | (5.59, 7.63) | | Freeze LN | (9.69, 11.54)| (9.84, 11.97)| (7.90, 9.92) | (6.37, 8.52) | (5.48, 7.64) | When LN is frozen, PPL generally increases slightly. This may be due to the reduction of learnable parameters. Nevertheless, our results are still significantly better than previous shown in Table 3. Thanks for your comments, and we will add these results in the final version.   Now, please give us some time to compare decoupleQ with GPTQ and BRECQ in detail. Both decoupleQ and GPTQ contain a step to minimize the loss between pre- and post-quantization within a linear layer. But minimizing the loss between pre- and post-quantization within a linear layer is a very simple and easy idea[1,2,3,4,5]. The core contribution of these papers is not that they use the layer minimization, but how to achieve the layer minimization. For example, AWQ[1] achieves this goal via per-channel scaling that protects the salient weights; GPTQ[5] achieves this goal by updating the remaining elements while quantizing the previous elements, and adaround[2] by training the rounding up or down. While in decoupleQ, we achieve this goal by constrained mathematical optimization (6). We believe that formula (6) abstracts the essence of the problem, transforming a model quantization problem into a mathematical optimization problem, and it no longer needs to focus on some of the minutiae unique to quantization. $w$, $s$, and $z$ are now totally independent of the original weight $W_0$ in the optimization process, as long as the final output error is minimized. This is the essential difference between decoupleQ and previous methods. When solving formula (6), we provide two approximate solutions, (10) and (11). Both formulas are common unconstrained convex optimization problems. We used the the idea from GPTQ to solve (11), but this is not the only solution, because we can choose to solve (10) to obtain a more accurate result.   The comparison with BRECQ is similar. Block minimization is used in works[3,6,7]. In BRECQ, they train the rounding for GEMM weights, whereas in decoupleQ, we train $(s,z)$ and norm layers while keeping the GEMM weights frozen (because it has already been quantized to integers at this step). This is a very lightweight training compared with BRECQ, because GEMM weights occupy most of the model parameters, and $(s,z)$ only occupies a very small part of the model parameters.   In addition, if we only consider the first stage, i.e., minimizing the loss between pre-quantization and post-quantization within the linear layer, without considering the second stage, our results still outperform the others (especially GPTQ, the fairest comparison), as shown in Table 4 and Table 3.   [1] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration \ [2] Up or Down? Adaptive Rounding for Post-Training Quantization \ [3] AffineQuant: Affine Transformation Quantization for Large Language Models \ [4] QuIP: 2-Bit Quantization of Large Language Models With Guarantees \ [5] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers \ [6] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models \ [7] BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction   Thank you again for your valuable time and high-quality comments. We sincerely look forward to further discussions with you.

Reviewer jfxa3/10 · confidence 5/52024-07-07

Summary

This paper proposes a linear and uniform quantization method, decoupleQ, which abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, then transforming the quantization problem into integer and floating-point part. Experiments show decoupleQ achieves comparable acc as fp16/bf16 on 2-bit weight quantization setting.

Strengths

Experiments show decoupleQ achieves comparable acc as fp16/bf16 on 2-bit weight quantization setting.

Weaknesses

1. Experiments are based on W2A16, lower activation bitwidth(<=8bit) should be experimented. 2. The novelty is limited. The core idea of decoupleQ is similar to Normalization(Batch-Norm or Layer-Norm). The learnable floating part of decoupleQ equals to a learnable Normalization parameters. 3. More existing Quantization methods should be compared, such as, NWQ[1], PD-Quant[2] [1] Leveraging Inter-Layer Dependency for Post -Training Quantization [2] PD-Quant: Post-Training Quantization based on Prediction Difference Metric.

Questions

1. What's the gain for lower activation bitwidth(<=8bit) of decoupleQ? 2. Experiment comparison over more existing Quantization methods.

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

NA

Reviewer jfxa2024-08-09

Thank authors for the rebuttal. The rebuttal does not solve my concern. The novelty is limited: DecoupleQ is the same as Batch-Normalization or Group-Normalization if you make the BN or GN 's parameter learnable and merge them into normal quantization process. The effect between them from my point of view is the same. Further, comparison to current CNN quantization methods is not fully provided. Thus I still keep my score and reject this paper: it does not reach the bar of NeurIPS, in shortage of novelty proof and enough experiment comparison.

Authorsrebuttal2024-08-09

Clarification on the novelty

Thank you for your reply. The core of our work is quantization. To this end, we transform a quantization problem into a mathematical optimization problem. In decoupleQ, the quantization process includes not only the solution of the floating-point part $(s,z)$, but also the solution of the integer part $\widehat{W}$. However, Batch-Normalization or Group-Normalization is a skill for effective training. In some model structures, their parameters can indeed be merged into $(s,z)$, but this is not the case for all model structures especially in LLM. If conventional quantization cannot obtain a good solution for $\widehat{W}$, then no matter how we train the Normalization parameters, it will be difficult to obtain high model accuracy. decoupleQ is to solve the integer part $\widehat{W}$ and the floating-point part $(s,z)$ together, rather than just considering the floating-point part. &nbsp; Thank you again for your reply. If we have not clarified your concerns, we sincerely hope that you can raise them in time, and we will reply to you as soon as possible.

Reviewer 6SKU5/10 · confidence 4/52024-07-10

Summary

The paper presents decoupleQ, a post-training quantization method that improves the accuracy of quantized models, particularly at very low bit-widths (2-bit). It achieves this by separating model parameters into integer and floating-point components and formulating the quantization process as a constrained optimization problem. This approach eliminates the need for traditional quantization techniques like outlier handling and focuses on optimizing the core objective.

Strengths

1. The paper introduces a fresh perspective on quantization by abandoning traditional methods and reframing it as a constrained optimization problem. 2. decoupleQ demonstrates impressive results in 2-bit quantization, achieving accuracy comparable to higher precision formats like fp16/bf16 in large speech models. 3. The quantization process is linear and uniform, making it easier to implement in hardware compared to non-uniform methods.

Weaknesses

1. The paper's writing lacks cohesion and clarity regarding its ultimate goal. The paper also has several spelling mistakes. 2. The authors claim to separate the model parameters into integers and floating-point components. However, as far as I understand, this practice is not a novel contribution but rather a common approach in quantization. 3. They address a portion of the optimization problem using GPTQ and another portion similar to BRECQ. 4. The authors acknowledge that their solution may not be optimal. 5. The quantization process in decoupleQ can be more time-consuming than other methods.

Questions

1. The paper mentions achieving state-of-the-art accuracy in Llama-1/2. It would be helpful to see a more detailed comparison with other state-of-the-art quantization methods on this specific model. 2. The authors could elaborate on the potential impact of the weak correlation between PPL (perplexity) and loss in LLMs. How might this affect the practical application of decoupleQ for LLM quantization? 3. The paper briefly mentions the risk of overfitting. Providing more insights into mitigation strategies for this risk would be beneficial, especially when dealing with underdetermined matrices. 4. Given the longer runtime compared to other methods, it would be interesting to see a more comprehensive analysis of the trade-off between quantization time and model accuracy. 5. The authors could consider adding a section on future work, outlining potential directions for further research and improvement of the decoupleQ method.

Rating

5

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

Yes, however, it will be beneficial to divide the current Discussion section into separate Conclusion and Limitations sections.

Authorsrebuttal2024-08-04

responses to weaknesses

Thank you so much for taking the time to review our work and then give an accurate summary and outline our strengths. We will explain your concerns in detail. Due to page limitations, we put the responses to Questions at the "Rebuttal" box. # Weakness 1: The ultimate goal of our paper is to propose a novel quantization method to contribute to the industry and the community. Specifically, in the organization of the paper, we first formulate the quantization problem as the optimization objective(6) step by step in Sec. 3.1 and Sec. 3.2, which is the core of our paper. Once the optimization objective(6) is proposed, it is natural for us to think about how to solve it. So we propose a solution in Sec. 3.3 and Sec. 3.4. After solving the optimization objective(6) with the proposed method, a linear layer within a Transformer block is then quantized. In Sec.3.5, we further fine-tune $(s,z)$ at the block level to further improve the model accuracy. Thank you very much for your careful reading, and we will check for spelling mistakes carefully in the revised version. # Weakness 2: In the traditional quantization paradigm, the integer part and the floating point part are calculated using the following procedure (or its various variations): $$s=\frac{max(W_0)-min(W_0)}{2^N-1} \tag{a}$$ $$z=min(W_0)-\alpha s \tag{b}$$ $$\widehat{W} = \text{clip}(\lfloor \frac{W_0-z}{s} \rceil, \alpha, \beta) \tag{c}$$ where the meanings of the above notations are explained as in Sec.3.1. In this set of formulas, the solutions for the floating-point part and the integer part are interdependent. What we call "decouples the model parameters into integer and floating-point parts" aims to completely decouple the two and use the optimization formulation(6) to find the optimal solution. The feasible region of the $(s,z)$ and $w$ have no dependency and can be regarded as independent primal variables. Judging from the results, all common quantization schemes will result in integer and floating-point parts after the model is quantized. However, we focus on the solution process and the ideas and methodology embodied in this process. # Weakness 3: When solving formula (11), we used the solution of GPTQ. But this is only one optional step in our solution process. Our core contribution is not the solution to this step, but the construction of formula (6). Formula (6) is an optimization objective, and there should be many ways to solve it, and we just provide one. BRECQ aims to use block reconstruction to train the rounding of all the weights within the block, while in our work, we use block reconstruction to train the $(s,z)$ while freezing the integer part. In fact, we have given three levels of optimization solutions: 1. At the linear layer level, we should solve formula (6) to get the integer part and floating-point parts as the first step; 2. At block level, we freeze the integer part and only train the floating-point parts via block reconstruction to further improve the model accuracy; 3. At the model level, we can use labelled dataset to train all the floating-point parts in the model to further improve model accuracy, or adapt to the downstream sub-tasks. All of the three levels revolve around a common idea, which is to optimize the integer part and the floating-point part of the model separately. # Weakness 4: The value of our paper lies in treating the integer part and floating-point part of model quantization independently, and then transforming a quantization problem into a mathematical optimization problem (6). As for how to solve this formula (6) in isolation, it is not the core contribution of this paper. There are many solution methods in the field of mathematical optimization, and we only provide one. We admit that this solution may not be the best, but even a suboptimal solution can still achieve high quantization accuracy. Doesn’t this further illustrate the superiority of our method ( transforming a quantization problem into a mathematical optimization problem (6))? # Weakness 5: We unabashedly admitted in the paper that our quantization method is more time-consuming than other methods such as GPTQ. However, we also provide specific time costs as a consideration for engineers when choosing the most appropriate time-accuracy trade-off for application. In industry, it takes less than 40 hours (reported in the original paper, but can be reduced to 20 hours at the moment, via setting $J=1$, defined in line 208) to produce a quantized and high-accuracy large language model. We think it is of high industrial value. &nbsp; Thank you again for taking the time and effort to review our paper carefully and then asking these high-quality questions. We look forward to further discussions with you. We would be grateful if you could raise the rating after addressing your concerns. Many thanks again!

Reviewer 6SKU2024-08-11

Thank you for the detailed rebuttal. The authors have adequately addressed my concerns, and I am prepared to increase my rating accordingly. However, I believe the original manuscript could benefit from improved structural clarity. To enhance readability, I recommend the authors focus on improving the overall organization of the paper for the final submission.

Authorsrebuttal2024-08-11

Thank you for your recognition of our work and your valuable suggestions. We will carefully review the organization of the paper (especially Sec 1 and 3), introduce our ideas step by step, highlight the key points, and enhance the coherence of the text, . We will also check carefully to avoid spelling errors. Thank you again and wish you all the best.

Reviewer 4m2b6/10 · confidence 4/52024-07-12

Summary

This paper proposes a novel post-training quantization method to achieve 2-bit uniform quantization on large language and speech models. The proposed method decouples the quantized values into integer and floating-point parts, which are then optimized via a constrained optimization problem that can be solved with off-the-shelf solutions. The proposed method allows uniform quantization down to extreme bits.

Strengths

1. This paper proposes a novel optimization-based method to conduct PTQ on large models. The proposed method is solid and unique from previous methods. 2. The proposed method achieves good performance with only uniform quantization, without special procedure for outliers etc., providing direct benefit to the runtime of the quantized model on general hardware. 3. The limitations and future directions are clearly discussed in the paper.

Weaknesses

1. The distinction between the proposed decoupleQ and the traditional quantization methods are not clearly derived in Sec. 3.2. The statement that "(s,z) lost the traditional meaning" on line 138 is not clear. My understanding is that W, s, and z are now totally independent of the original weight w0 in the optimization process, as long as the final output error is minimized? I think adding a comparison with the optimization objective/procedure of the traditional quantization here will help. 2. The proposed method appears to be sensitive to the size of the calibration set, so that the calibration size reported in the experiments are much larger than that of the previous baselines. As it is understandable that the optimization process may require more data to avoid overfitting, it would be more fair if the baseline methods are also calibrated with the same dataset/training cost. 3. For the LLM experiments, only ppl is used as metric. However, the ppl has been shown to be an inaccurate metric to reflect the utility of the LLM after compression. More evaluations such as zero-shot performance on downstream tasks and the instruction following ability etc., as in SqueezeLLM and OmniQuant papers, would be helpful to see if the quantized model still retains the ability as the FP one.

Questions

Please see the weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations and potential social impacts are adequately addressed.

Reviewer 4m2b2024-08-08

Thanks for the rebuttal

I would like to thank the author for the rebuttal. The rebuttal resolves my concern on the calibration size. Generally speaking, as I agree with the other reviewers that more experiments can be done to show the effectiveness of the proposed method, I believe the existing evidence is enough to show the proposed method is appliable. Given the novelty and significance of the proposed method, I increase my score to weak accept.

Authorsrebuttal2024-08-09

Thanks for the reply

Thank you very much for your reply and for your recognition of our novelty and significance. Thanks again!

Authorsrebuttal2024-08-09

the value of the work

Dear all reviewers, &nbsp; We are very grateful to all reviewers for their valuable comments, and thankful for most of the reviewers in general appreciating that the work is novel, solid, fresh, and new, as well as for accurately summarizing our work and outlining its strengths. We sincerely admit that the lack of rich public experiments was indeed our shortcoming, although our PPL in Table 3 is lower than others by a large margin in W2A16. We will make our code public on GitHub (regardless of whether the paper is accepted or rejected finally), and continue to add more experimental results. The work is tangible and can be applied to industry. We do have launched 2-bit quantized large speech models in multiple consecutive releases of our company's core products, with the model accuracy near to bf16. After the reviewing period is over, the identity of our company and the products launched will be made public. Also, we have released the W2A16 CUDA kernel used in our products, which is under the merging review into NVIDIA tensorRT repo. We believe that the value of research lies in the novelty and its industrial application, rather than the number of experiments, especially in NeurIPS. And we believe that our work will make certain contributions to the industry and open-source community. &nbsp; We deeply appreciate all of reviewers’ time for reviewing and hope all the best! &nbsp; Thanks a lot, \ Authors

Reviewer G89P2024-08-11

Thank you for the detailed rebuttal. The author has addressed some of my concerns. However, conducting experiments without training norm layers is crucial, as other methods typically do not involve training norm layers. Additionally, I agree with other reviewers: They address a portion of the optimization problem using GPTQ and another portion similar to BRECQ. The novelty is limited, resembling more an aggregation of existing techniques rather than a novel contribution.

Program Chairsdecision2024-09-25

Decision

Reject

© 2026 NYSGPT2525 LLC