Summary
In this paper, the authors study the quantization of orthogonal recurrent neural networks (ORNNs). They first investigate the impact of quantization on the orthogonality, including effects on the singular values of the quantized weights and deriving bounds for them. Then the authors propose and investigate two flavors of quantized orthogonal recurrent neural networks (QORNNs) which combine traditional ORNN training with quantization-aware training (QAT). In the experiments they show that QORNNs are competitive with SOTA ORNN, LSTM and FastRNN on a various benchmarks such as copying, flavors of MNIST and PTB.
Strengths
* The investigation on weight quantization effects on ORNNs is interesting and insightful (sec 3.4).
* Based on the experiments the proposed QRNNs (especially with STE-Bjorck) seem to perform well, even till 5-6 bits.
* The paper is clearly written and easy to follow (except a few minor points).
Weaknesses
* The biggest shortcoming of this paper is the limited novelty. Several points regarding this:
* The authors combine two off the shelf ORNN training algorithms with the most simple (and arguably outdates, see later) flavor of QAT. In other words, they add $q_k(W_i)$ to these algorithms (cf line 3, 4 in algorithms 1, 2, respectively) and assume STE.
* While I found the investigation in sec 3.4 interesting (see above), I would have expected that these insights come back in their algorithm design (or at least that the authors evaluate such metrics for their proposed approach, look at question whether these metrics in practice correlate with QORNN performance etc).
* On the quantization side, they seem to miss/ignore a lot of innovation that happened in the last 5ish years which could help to potentially have much better performance at low bit-widths. Most importantly:
* They argue keeping the scale fixed is common practice (line 241/242). Since the introduction of LSQ [1] this is long not common practice anymore.
* It is unclear to me why the authors not consider per-channel weights. As per-channel weights still work with full fixed-point/integer arithmetic [2], this is supported by almost any HW and commonly used by most quantization literature in the past years. This adds an additional degree of freedom that might be very helpful for ORNNs as by changing the scale ($\alpha$), as one could ensure that rows (or columns, depending on notation) still have a norm of 1 which seems important (cf sec 3.2).
* The proposed QORNNs are actually not orthogonal as the name or text suggest. Only the latent weights ($W_i$) are orthogonal, but the actual quantized weights used for inference ($q_k(W_i)$) are only approximately orthogonal. As the authors themselves show (cf figure 1, sec 3.4), this doesn’t give any guarantees and could be detrimental.
* As the paper positions itself more as ‘explorative’ (and QAT contribution is very limited), I would expect that they also more closely explore PTQ approaches. There are several degrees of freedom that are unexplored, e.g. setting the correct scale (alpha) or applying/adapting common PTQ algorithms such as AdaRound [3] or GPTQ [4].
* Minor:
* The experimental evaluation is limited to only ‘toy-sized’ datasets/tasks.
* While it is nice they obtain bounds for q_min/max, the established bounds are so loose that from a practical perspective such bounds are not useful (nor similar to the earlier study they are used to design or evaluate the algorithm).
* I do miss a comparison to challenges in quantizing transformers or SSMs. While it is arguable whether comparing to transformers it out of the scope of such a paper (as the authors claim), at least discussing/comparing whether the challenges in ORNNs are similar or different to transformers/SSMs would be helpful (e.g. do they suffer from similar outliers as observed in transformers, cf. [5,6]).
* Regarding SSMs, there is some recent work that would be interesting to compare to [7, 8].
**References:**
* [1] Esser et al., ICLR 2020, Learned Step Size Quantization.
* [2] Nagel et al., 2021, A White Paper on Neural Network Quantization.
* [3] Nagel et al. ICML 2020, Up or Down? Adaptive Rounding for Post-Training Quantization.
* [4] Frantar et al., ICLR 2023, Gptq: Accurate post-training quantization for generative pre-trained transformers.
* [5] Bondarenko et al., EMNLP 2021, Understanding and Overcoming the Challenges of Efficient Transformer Quantization.
* [6] Dettmers et al., NeurIPS 2022, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.
* [7] Pierre et al., ES-FoMo 2024, Mamba-PTQ: Outlier Channels in Recurrent Large Language Models
* [8] Abreu et al, NGSM 2024, Q-S5: Towards Quantized State Space Models
Questions
* It is unclear from the paper why you quantize U but not V. Could you please elaborate on that choice?
* It is unclear to me why the ProjUNN is much better in the copy task but worth on MNIST and PTB (both quantized and unquantized model). Do you have any idea or intuition why this is the case?
**Comments on presentation:**
* $\sigma$ is a bit overuse ($\sigma, \sigma_o$ but also $\sigma_{min}, \sigma_{max}$ for eigenvalues, …)
* The authors reuse $T$ in line 258 again as a power of the weight matrix, which seem based on the text has not any connection to the sequence length T. I suggest the authors either use a different symbol or explain why to the power of $T$ would be relevant here.
* I can see that this likely comes from the sequence length $T$, though given there is activation function $\sigma$ (and $Ux$) there, it is unclear why this would be important or relate to the final task loss. If this is the case, please elaborate.
* Many equations do not have numbers which makes it hard for readers to refer to them (even if authors do not refer themselves to some equations, a future reader might want to do so). Therefore I suggest to follow common best practices and number all equations that are not in-line.