Summary
The authors study the question of whether quantization (of weights and
activations in decoder-only language transformer architectures) fundamentally
reduces performance at large parameter counts, or whether this degradation of
performance, observed in prior works on this topic, is simply a consequence of
certain design choices in the optimizer. They consider post-training
quantization (PTQ) schemes, which simply quantize the weights and activations
of a pretrained (half-precision) model at inference time (to int8/int8) without any extra
quantization-aware retraining, and conduct experiments at the 6B parameter
scale (which entail measuring the accuracy drop after PTQ on 8 datasets)
varying the magnitude of weight decay, dropout rate, gradient clipping, and
half-precision type (fp16 vs. bf16). Their results suggest that larger weight
decay, gradient clipping, and bf16 data types improve PTQ performance; they
then train with appropriate settings at the 60B parameter level, and
demonstrate that they can significantly improve the average PTQ performance
relative to a comparable scale model (OPT-66B); Figure 1 highlights these
results. The authors then study the weights and activations in the OPT-66B
model and their own model in order to assess how their design choices have
affected various properties that played a role in prior narratives on why PTQ
performance might fundamentally be limited at large scales (specifically, the
role of "outlier dimensions", but also general quantization-type metrics such
as reconstruction RMSE).
Strengths
The authors' insights lead to a significant improvement over OPT-66B PTQ
accuracy drop with a comparable parameter count (the accuracy actually improves
on average). Their results also highlight useful general takeaways about how
regularization can help mitigate PTQ performance drop, including the utility of
bf16, weight decay, and gradient clipping, and they provide reasonable
explanations for why.
The authors have presented most methodological choices and details of their
experimental setup very clearly, which will aid reproducibility of their
results and the ability of researchers to build off the results of their study.
Weaknesses
It might be helpful to include additional quantitative comparisons to prior
work; currently the comparisons to prior art are substantial but mostly
qualitative. iT seems one of the cited works in the related work section,
FlexGen, achieves INT4 quantization of OPT-175B on Lambada and WikiText with
negligible performance (acc/ppl) penalty. It might also be nice to add brief
experimental studies demonstrating improved throughput/memory footprint of the
authors' PTQ approach, to substantiate the claims made at a few points in the
paper that the authors' approach will have such benefits over the
mixed-precision approach of Dettmers et al.
The description of the experimental protocol for section 4 is somewhat confusing -- lines
145-150 and 193-198 both suggest that training was done using early checkpoints
on 410M parameter models before "scaling" to 6B models, but no results seem to
be reported for 410M parameter models. It seems that this detail could be
omitted if no results of substance are reported to corroborate it. In addition,
it is not made completely clear in section 3.2 whether the 6B experiments
reported in section 4 are actually done via training-from-scratch or done from
early checkpoints; the text in lines 145-150 suggests that section 4 results
are not actually trained from scratch, but the actual protocol in section 3.2
does not clearly state this.
The description of the regularization study in the contributions section feels
slightly misstated -- the "controlled large-scale study" alluded to does not
actually seem to "train from random initialization" (uses early checkpoints),
and the hyperparameter search is actually only conducted at 6B parameter models
(from early checkpoints). This suggests an implicit hypothesis of this study is
that performance differences observed at "medium-scale" models are predictive
of those at larger scales; the fact that this is borne out up to 52B parameter
models seems to be an interesting consequence of the work, but then it does not
seem completely accurate to claim a systematic study up to this scale as a
contribution.
The takeaways from the weight/activation studies in section 5 do not seem to
lend themselves to straightforward conclusions or insights into why the
authors' design choices lead to superior PTQ robustness (see discussion at line
316). It might help the presentation if the authors selected a small subset of
the activation/weight metrics presented which lend themselves to a coherent
interpretation of why the authors' model and OPT/BLOOM have different PTQ
performance. One key takeaway that is mentioned both in the introduction and
the conclusion, that the authors' work refutes the hypothesis that outliers are
not inherent to scaling large language models, is not even presented in the
main body (it has been relegated to appendix B.3), and lines 323-324 suggest
the authors could not find correlations between outlier detection recipes and
PTQ performance. Discussing this study more in section 5 would seem necessary
given the way the authors have presented it as a contribution.
Questions
In line 268, performance differences on BLOOM are claimed; are these results
available or reported? From the appendix Table 4, it seems that these results
are referring to int4 quantization of weights only?
The authors do not test their design choices at the full 175B scale, and the
reason for this is not discussed. It would be helpful to know whether this is
just due to compute obstructions to training at this scale, or whether it is
because the authors' methods are not scaling properly to this scale.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Limitations are clearly specified throughout and in the conclusion; there is perhaps somewhat of a missed opportunity to discuss potential negative societal impact of LLM quantization.