MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates

This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 updates, called MKOR, that improves the training time and convergence properties of deep neural networks (DNNs). Second-order techniques, while enjoying higher convergence rates vs first-order counterparts, have cubic complexity with respect to either the model size and/or the training batch size. Hence they exhibit poor scalability and performance in transformer models, e.g. large language models (LLMs), because the batch sizes in these models scale by the attention mechanism sequence length, leading to large model size and batch sizes. MKOR's complexity is quadratic with respect to the model size, alleviating the computation bottlenecks in second-order methods. Because of their high computation complexity, state-of-the-art implementations of second-order methods can only afford to update the second order information infrequently, and thus do not fully exploit the promise of better convergence from these updates. By reducing the communication complexity of the second-order updates as well as achieving a linear communication complexity, MKOR increases the frequency of second order updates. We also propose a hybrid version of MKOR (called MKOR-H) that mid-training falls backs to a first order optimizer if the second order updates no longer accelerate convergence. Our experiments show that MKOR outperforms state -of-the-art first order methods, e.g. the LAMB optimizer, and best implementations of second-order methods, i.e. KAISA/KFAC, up to 2.57x and 1.85x respectively on BERT-Large-Uncased on 64 GPUs.

Paper

References (33)

Scroll for more · 21 remaining

Similar papers

Peer review

Reviewer dQpV5/10 · confidence 4/52023-06-30

Summary

This work introduces a new optimization algorithm for deep neural networks. Building upon the baseline Kronecker-factored curvature (K-FAC) algorithm, this new approach approximates each activation and gradient covariance matrix within K-FAC as a rank-one matrix. This approximation enables the efficient calculation of the inverse Hessian. By leveraging this enhanced efficiency, the algorithm allows for more frequent updates of the Hessian. Experimental results demonstrate great speed improvements compared to the state-of-the-art first and second order optimization algorithms.

Strengths

1. Efficient second-order optimization algorithm is important for the training of DNNs. 2. Experimental results show great speed-ups compared to the baseline algorithms. 3. The paper is in general well-written, and the main contribution is clear.

Weaknesses

One notable weakness of this research lies in the insufficient justification of the proposed technique, which involves applying rank-1 approximation to the covariance matrices used in K-FAC. While the work addresses the efficiency benefits of this method, it fails to thoroughly discuss the aspect of why employing rank-1 approximation does not impact the final accuracy or convergence of the algorithm. The following questions should be addressed to strengthen the research: What specific scenarios allow for accurate rank-1 approximation, and when might it be ineffective? Does the effectiveness of the approximation depend on the singular value decays of the covariance matrices? Are there any illustrative toy examples that can shed light on these considerations?

Questions

n/a

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

n/a

Reviewer zw2W7/10 · confidence 4/52023-07-06

Summary

MKOR introduces Kronecker rank-1 representation of covariance for second-order optimizers. Authors try to solve the complexity of large language models in second-order optimizers while the Fisher information approximation is computed using rank-1 Kronecker matrix factorization. They also propose a hybrid method to cover the first-order method. Their evaluation of large language models is fair and sound.

Strengths

The main idea behind this research is simple but solves the important problem of the applicability of second-order optimizers for large language models. The paper is well-motivated and experiments are sound and relevant.

Weaknesses

- Authors only argue why rank-1 is proposed from a practical standpoint but there is no theoretical justification to support why rank-1 is enough for Fisher information approximation.

Questions

- Rank1 approximation might be restrictive to approximate the Fisher information properly. I wonder in some cases an extension to higher ranks is necessary. I wonder if an extension towards SeKron [1] is natural. [1] SeKron: A Decomposition Method Supporting Many Factorization Structures arXiv:2210.06299 - Does adding a learning rate scalar in Equation (2) make the method more flexible to converge? - How well does the second-order MKOR method compare with the first-order with the same training budget in terms of time and memory?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The numerical stability requires SVD decomposition and matrix inversion which are expensive to compute.

Reviewer zw2W2023-08-18

Rating revised

I would like to thank the authors for their convincing rebuttal. I raise my rating accordingly.

Reviewer Ar2i7/10 · confidence 4/52023-07-07

Summary

The work introduces a second-order optimizer that uses rank 1 covariance activation and gradient statistics, and efficient inversion algorithms to accelerate an approximated estimate of 2nd order information. Another addition is the rescaling of gradients and norm-based stabilization, which help to stabilize the entire optimization procedure. This procedure yields a highly stable 2nd order optimizer that is both memory (communication) and computationally efficient and leads to improved convergence times compared to baselines.

Strengths

From the work its clear that the method depends on many independent factors. While some reviewers might see this as a weakness, I see this is a strength. Developing efficient 2nd order optimizer is a highly technical and difficult undertaking and this work provides multiple components that make a stable optimizer possible that is both memory and computationally efficient. The evaluation could have been extended to regular language modeling since BERT pretraining has become less academically relevant. Nevertheless, a BERT large pretraining is a rather convincing experimental setup. As such, the experimental robustness is also a strength of the paper.

Weaknesses

I think additional experiments for causal language modeling would have been beneficial since this would be the main advantage for more efficient training (BERT pretraining is fast enough for most researchers, but most researchers are unable to pretrain LLMs due to their computational cost). While this additional experiment would greatly improve the paper, the experimental methodology in this paper is already relatively robust.

Questions

No questions in particular.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are not discussed.

Reviewer KDBs5/10 · confidence 4/52023-07-10

Summary

The paper proposes a second-order optimizer named MKOR based on K-FAC. Compared with K-FAC, MKOR uses a rank-1 update to construct Kronecker factors enabling the inverse of Kronecker factors to be efficiently computed via the Sherman-Morrison-Woodbury (SMW) identity. In addition, the rank-1 update of factors in MKOR also helps reduce the communication traffic in distributed training with data parallel for aggregating factors. Experiments on the BERT-Large-Uncased model with the GLUE dataset show that MKOR outperforms first-order LAMB and second-order KAISA.

Strengths

1. An implicit inverse computation method with the SMW identity in computing the inverse preconditioner factors and a formula to control the balance of exploitation between the first-order and second-order information. 2. A hybrid method of MKOR with the first-order method, MKOR-H, which allows switchable from MKOR to the first-order optimizer in the later of training.

Weaknesses

1. An important baseline, Eva [A], which is extremely similar to MKOR, is missing. 2. Though MKOR-H seems to achieve some improvement, when switching MKOR to the first-order optimizer is unclear. 3. The comparison between MKOR and K-FAC seems to be not that fair (e.g., Figure 4(b)). [A] Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation, ICLR 2023.

Questions

1. The key idea of using rank-1 updates to replace the original Kronecker factors in K-FAC has been proposed in Eva [A]. In [A], the authors provide a better derivation of the preconditioner without storing the factor matrices so that they use Kronecker vectors (KVs) to calculate the inverse of KVs using the SMW formula, which is much better than MKOR as MKOR needs to store the matrix form of factors (i.e., $L_t^m$ and $R_t^m$). The similarity between MKOR and Eva and the advantage of Eva over MKOR make the originality of this work weak. It is suggested to clarify the differences between MKOR and Eva, and highlight some advantages of MKOR over Eva. 2. The main novel idea in MKOR is the balance between second-order and first-order information in updating $L_t^m$ and $R_t^m$ using Eq. (7) and (8) by controlling $\zeta$. However, how to choose $\zeta$ (or dynamically changing $\zeta$ to switch MKOR to SGD just like MKOR-H) is very important. The paper doesn’t provide such an analysis of when and how to use $\zeta$ though some results show that it can help improve convergence. 3. Similarly to the previous problem, MKOR-H is a combination between first-order optimizer and MKOR according to the loss during training. However, how to measure whether the loss is changed is unclear (e.g., results in Figure 2 haven’t such information either) and when to switch to the first-order optimizer is unclear. 4. What are the update frequencies of the second-order information for MKOR and K-FAC in Figure 2 and Table 3? And how do you choose such frequencies for comparison? 5. In the original paper of KAISA, the update frequency can be set to be very high (e.g., 100-500) for achieving good accuracy on ResNet models with CIFAR10 and ImageNet datasets, is it possible that MKOR could be better than KAISA in these kinds of scenarios? [A] Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation, ICLR 2023.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See Questions.

Reviewer KDBs2023-08-18

Rebuttal read

Acknowledgments to the authors for providing a detailed response and additional experiments. Most of my concerns are addressed so I raise my rating. Hope that these discussions would be included in the final version to make the paper clear.

Reviewer dQpV2023-08-19

Thank you for the response, it would be good to add the discussions above to the paper. I will raise my score.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC