Summary
The paper proposes two architectural modifications that mitigate the activation outlier problem that makes transformers challenging to quantize. Particularly, the authors conduct a comprehensive analysis of the underlying causes of outlier issues in various pre-trained Transformer models across different tasks and datasets. They identify specific architectural limitations that lead to large outliers during the pre-training phase. To tackle this challenge, the authors propose two solutions: 1/ clipped softmax and 2/ gated attention that can effectively address the outlier problem without sacrificing model performance when compared to vanilla Transformer models across various use cases. Furthermore, the suggested modifications result in smaller outlier values and enhance the quantization performance of the Transformer models, making them more robust to quantization.
Strengths
1. The paper presents strong motivations and novel approaches to address outlier issues that distinguish it from prior works. Unlike existing methods that focus on circumventing the impact through post-training techniques, this paper proposes a fundamental training-time solution.
2. The paper proposes a simple yet highly effective methodology, which directly addresses the fundamental issues and observations.
Weaknesses
1. Evaluating the proposed methodology by comparing it solely to the naive application of (vanilla) PTQ on the plain Transformer architecture might be somewhat unfair and not informative enough. Various outlier suppression methods have been proposed [] as post-training solutions, and thus it would be better to include a comparison with these methods. Given that the proposed methodology requires from-scratch training, its practical value will be justified if it outperforms post-training methods that require no additional training.
2. The paper introduces two distinct solutions; however, they are two orthogonal/competing methodologies that tackle the same problem. The paper lacks clarity regarding the specific circumstances under which each solution should be employed. That said, how can we decide which scheme to use when training a new model on a new dataset without having to try both and choose the better one?
3. As also stated in the paper, the evaluations have only been conducted on smaller-scale models (100M parameters). While the proposed architectural and train-time modifications show promising results that match/outperform the vanilla Transformer performance, it is unclear if this trend can persist on a larger scale (hundreds of millions to billions of parameters). It would be more informative if the authors can provide across different model scale regimes. Moreover, since the proposed solution is a pre-training methodology, it requires training from scratch and additional risk/cost related to extra hyperparameter tuning, which can potentially limit their practical application.
[1] Wei, Xiuying, et al. "Outlier suppression: Pushing the limit of low-bit transformer language models." Advances in Neural Information Processing Systems 35 (2022): 17402-17414.
[2] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." arXiv preprint arXiv:2211.10438 (2022).
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Please see the weakness section