Summary
In this work, authors proposed a novel quantization method called REx, that leverages residual error expansion. The idea is to repeatedly apply quantization to the quantized residual error, which provides an increasingly better approximation of the original floating-point tensor as the number of steps grows. To combat the computational overhead from additional quantization steps, authors combine the residual expansion with the group sparsity. Two hyper-parameters of the method (a number of steps K, and the sparsity rate $\gamma$) are used to control the accuracy vs. speed trade-off. The method is agnostic to the quantization operator itself and can be combined with many existing methods.
Strengths
* Experiments on both CNNs (including some of the more difficult-to-quantize models like MobileNets and EfficientNets) and LLMs across various tasks + combination with several quantization operators from the prior work.
* Theoretical derivations on the upper bound of the quantization error + comparison with the empirical values.
* Some attention to efficient hardware implementation (Section 4.1), e.g. instead of using K separate CUDA kernels for the error expansion step, use a single one with concatenated output channels, etc.
Weaknesses
* Two hyper-parameters (K and $\gamma$) that have to be set. Even though $\gamma$ can be set to match a certain BOP target (as it is done in the paper), the user still has to select a value for K.
* I have some concerns related to the data-free nature of the method, please see and elaborate on the below question.
* I would like to see error bars / standard deviations in some of the results, e.g. in Table 2. I wonder if the improvement is statistically significant (it is well known that some of the datasets from GLUE are quite small which might lead to high standard deviations in the results).
Questions
* The method is advertised as data-free, however, I don't see how it is the case in general. As far as I understood, it's data-free if the quantization operator Q itself is data-free (which is the case of DFQ, for instance). However, in most experiments, the underlying Q is not data-free (e.g., AdaRound, BRECQ, or even naive quantization with static range estimation). Could you elaborate?
* Related to the previous comment, in the group sparsity, the importance of a neuron is defined by the norm of its weights (L151: assuming the data-free setting). Assuming we have access to activations and gradients, have you experimented with other importance metrics (e.g., gradient-based, FIT)?
* I Would like to know a bit more details on the outlier quantization in LLMs. How are outliers detected/defined? Will the suggested approach work with outliers that are both positive and negative? In Table 3, what gives the most improvement - the method itself or the special treat of the outliers (would be nice to have an ablation study and/or have some additional data on e.g. quantization error for tensors with outliers with W1 vs. W4).
* (Sorry if I missed it) Do you quantize embeddings, LayerNorm weights/biases, and LLM head, are there any specific assumptions (e.g., first and or last layer in 8-bit)?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Authors mentioned one limitation that the method does not adapt to the per-layer importance and runtime cost discrepancies.